The area of Carberry is first mentioned in the 11th century when David I granted lands here to the church. The first building where the current one now stands was a typical, plain tower house with additions made in the 16th, 18th and early 19th century. It was here on the estate, at Carberry Hill, in June 1567 that Mary Queen of Scots surrendered to an army raised by her lords and was led into captivity.
In 1961 Carberry Tower was given to the Church of Scotland by Lady Mary Elphinstone. It was used as conference centre and a chapel was built in the park but changed hands in 2004, 2011 and 2015. Carberry Tower is now a wedding, events venue and hotel and is open to the public offering fine afternoon teas etc.
The grounds of Carberry Tower contain many yews and areas of the parkland were planted in 1861 when the estate was redesigned by the 15th Lord Elphinstone. Most yews date from this period however there are other yews near to the house around the car park which have girth sizes suggesting their origin is over 200 years old.