The De Wauncys were amongst the earliest possessors of this manor since the Conquest. From them, it descended to the Brookes. On 24 April 1471, Thomas Lovett II acquired Astwell in exchange for his hereditary estates by a family arrangement with his mother's cousin, Dowse Billing and her husband William Brooke. The house was begun by Lovett. George Shirley, Esq., Lovett's grandson, was the next possessor. He partially or totally rebuilt the manor house. From him this manor descended to Washington Shirley, 5th Earl Ferrers by whom it was sold in 1763 to Richard Grenville-Temple, 2nd Earl Temple. He was succeeded by his nephew, George Nugent-Temple-Grenville, 1st Marquess of Buckingham, and whose son, Richard Temple-Nugent-Brydges-Chandos-Grenville, 1st Duke of Buckingham and Chandos was also associated with the manor house. Biddlesden Abbey had possessions in the manor. By 1874, the manor house had been largely dismantled, some of the "inferior offices" becoming a farmhouse, and a "broad embattled tower" remaining. The castle has been Grade II Listed since 1953 but the report states that "in 1958 the building was restored for Mrs. Joan Wake". News items in 2019 reported that the mansion "retains fine features with mullions, window seats and leaded lights, fireplaces and an oak staircase." The building, with gatehouse tower that includes a spiral staircase leading to rooms and to the castellated roof, offered four bedrooms and three reception rooms; it was said to be habitable but additional restoration was desirable.
Architecture and fittings
The embattled gatehouse tower with the lower attachment to the left is of Lovett's time. To the west, a large courtyard house with over forty rooms accumulated under later owners in the Lovett and Shirley families in the 16th and 17th centuries, of which only a fragment survives. It still has mullioned windows with arched lights, suggesting 16th rather than 17th century. Inside is a fireplace with a broad frieze of simple geometrical motifs above the four-centered arch. Early 18th century drawings show ranges added against both the west and south sides of the house which were built up to or over the moat. These may be related to infilling the moat and setting out a garden with formal terraces on the south and east sides of the house. The south range of the building was altered in the 19th century and again in 1957 when the house was remodelled and the roof timbers replaced.
Tower
In 1918, Evans described the tower. At the time, Sir George Shirley's arms appear on its entrance. It was the gatehouse tower of an earlier house. The original doorway was built up, and another one was opened beside it. There is a basement with porter's lodge on the right of the entrance, and two upper storeys. The floors were well preserved, but the interior had become a pigeon-house. The room on the first floor has shields of arms under the whitewash, and there is a blocked-up doorway at this level, which possibly opened into an external gallery. Both this room and the one above have garderobes. From the leads, there is a good view of the site.
Grounds
A deer park was created in 1547. On 30 November 1943, a B-17 Flying Fortress bomber, No.42-3048 from USAAF station 109 Podington of the 327th bomb squadron, 92nd bomb group, 8th bomber command crashed near the farm buildings. It had taken off on its way with the rest of the squadron on a mission to bomb the industrial complex at Solingen, Germany. All 10 crew members were killed. On 9 November 2008, Lt Col Terry Hayes, Deputy Commander of the American 422nd Air Base Group at RAF Croughton laid a special wreath and read out the names of the American airmen. The men were included in the 2008 Remembrance Service in at Helmdon parish church.