In ancient times, the manor of Caverswall was held by the eponymous Caverswall family, who in 1275 were granted licence to crenellate their manor house. The resulting medieval moated castle was approximately rectangular in plan with four angle towers and a keep within the curtain walls. In the 15th century the castle, which was owned by the Caverswall family, became the seat of the Montgomery family, three of whom served as High Sheriff of Staffordshire. During the English Civil War it was garrisoned by parliamentary forces and was much decayed and neglected by the end of the 16th century. The neglected castle was purchased in 1615 by Matthew Cradock of Stafford, a local wool merchant, first mayor of Stafford in 1614 and Member of Parliament for Stafford in 1621. He built the present mansion house retaining the old castle walls, to a design, it is said, of Robert Smythson or John Smythson. The three storey house has five bays each with stone mullioned and transomed windows. There is a castellated parapet and an entrance porch When the Cradock male line failed the estate was sold in 1655 to William Joliffe but the eventual heir William Vane, 2nd Viscount Vane, was forced to sell it. Thereafter the castle had several owners. In 1811 it was occupied as a nunnery by a Benedictine order who sold it in 1853 to Sir Perceval Radcliffe, when they then relocated to Oulton Abbey. In the 1880s it was rented by the Wedgwood family. In 1891 it was purchased by W.E. Bowers who carried out extensive renovations and much improved the property. W.A. Bowers then sold it in 1933 to the Sisters of the Holy Ghost, who in turn sold it in 1965 to another convent, the Daughters of the House of Mary. When they left in 1977 the Castle was sold in various lots. More recently the castle was bought in 2006 by property tycoon Robin MacDonald for £1.7m who spent £1 million renovating it. Since the late 1970s, numerous planning applications for various commercial uses have been submitted, but they have all been refused. The owner was prosecuted for allowing some holiday letting use in 2013 and fined £17,000 and ordered to pay £100,000 costs. The castle has several enforcement notices preventing various uses including "events, activity days, parties, photography & romantic breaks" In February 2015 the property was offered for sale with Sotheby's for £3 million. In 2017 the owner talked about the issues he had faced from locals and the planning department: The castle has been for sale since September 2019.