It is known that there has been a house on the site from at least the 14th century, but nothing visible remains and there is no documentary record. The manor of Carlton was acquired by Nicholas Stapleton, eldest son of Miles de Stapleton, killed at the Battle of Bannockburn, who was Steward of the Household to King Edward II. The family originated at the manor of Stapleton-on-Tees near Darlington in North Yorkshire. Carlton passed to Nicholas's nephew Sir Bryan Stapleton, a Knight of the Garter and Warden of Calais, younger brother and heir of Sir Miles Stapleton, of Bedale, Yorkshire, a founder Knight of the Garter. The last in the male line of Stapleton of Carlton was Sir Miles Stapleton, 1st Baronet who died childless and left his estates to his nephew Nicholas Errington of Ponteland, Northumberland, who in accordance with the bequest adopted the name and arms of Stapleton. Nicholas's grandson, Thomas Stapleton, succeeded in 1750 and in 1765 improved the house and estate, commissioning Thomas White to landscape the park and Thomas Atkinson of York to add the long East Wing, the clock above which shows the date 1777. Thomas was a keen breeder and trainer of horses and with Sir Thomas Gascoigne, 8th Baronet won the St. Leger Stakes in 1778 with Hollandaise and in his own right the following year with Tommy. In 1795 he made an unsuccessful claim to the title Baron Beaumont, abeyant since 1507, which he was in remainder to through his great-grandmother Mrs Anne Errington, sister of Sir Miles Stapleton, 1st Baronet, both descendants via a distant female line of John Beaumont, 1st Viscount Beaumont, 6th Baron Beaumont, himself a descendant in the male line of John de Brienne, King of Jerusalem, by his third wife, Berengaria of Leon, and thus a second cousin of King Edward II of England. His son Miles died without issue, when the estate passed to his first-cousin Thomas Stapleton, whose son Miles Stapleton, 8th Baron Beaumont in 1840 successfully obtained the abeyant title Baron Beaumont. The house was re-built to its present form in 1873-5 by his son Henry Stapleton, 9th Baron Beaumont. His heir was his younger brother Miles Stapleton, 10th Baron Beaumont, an officer in the British Army, who married Mary Ethel Tempest, daughter and heiress of Sir Charles Henry Tempest, 1st Baronet of Heaton, Bolton le Moors, Lancashire. He was killed in a shooting accident only three years later and was succeeded by his infant daughter, Mona Fitzalan-Howard, suo jure 11th Baroness Beaumont, who married Bernard Fitzalan-Howard, 3rd Baron Howard of Glossop, heir to the Dukedom of Norfolk. Lady Beaumont owned Carlton until her death in 1971. During the Second World War the house was used as an auxiliary military hospital but was afterwards restored to its original condition. Her eldest son, Miles Francis Stapleton Fitzalan-Howard, 17th Duke of Norfolk, 12th Baron Beaumont, inherited both the Beaumont and Howard of Glossop baronies and in 1975 also succeeded his cousin as 17th Duke of Norfolk and Earl Marshal of England, with several other peerage titles and also inherited the Duke's estates including his principal seat of Arundel Castle in Sussex, which he made his home. After many years with no residents, Carlton Towers was re-occupied in 1990 by his second son Lord Gerald Fitzalan-Howard and his wife Lady Gerald Fitzalan-Howard.
In popular culture
The house was used as the location of "Hetton Abbey" for the 1988 film version of Evelyn Waugh's novel A Handful of Dust. It was featured on The Guest Wing, a television programme shown on Sky Atlantic in Great Britain in April and May 2012. The South Australian Film Corporation also used the house in the film Like Minds, starring Toni Collette. The TV series Micawber starring David Jason used the house for filming. The TV show, The Darling Buds of May used the house in their show,. The character played by Bernard Cribbins in the 1967 film Casino Royale was named "Carlton Towers of the F.O.", playing off the name of an earlier British comedy, Carlton-Browne of the F.O. Carlton Towers was also the name of several blocks of highrise council flats in the Little London area of Leeds. The flats were demolished in February 2010 as part of a major housing regeneration project.