The 19th-century Handbook for Travellers in Yorkshire and for Residents in the County recounts that the ancient castle was built by Wada, ruler of Hälsingland. Leland in his Itineraries, circa 1545, refers to several local legends supposing Wada to have been a giant who built many castles and roads in Yorkshire.
Second castle
A second castle, which occupied the entire width of the ridge, seems to have been Norman, presumably constructed by Nigel Fossard, who obtained the property after the Norman Conquest. Fossard is recorded in the Domesday Book of 1086 as a tenant of 114 manors, all in Yorkshire, including under Robert, Count of Mortain of "Grif", identified as Mulgrave in the hundred of Langbaurgh. He became himself a tenant-in-chief of the king in 1088, and a holder of the large feudal barony whose caput was at Mulgrave, hence known as the barony of Mulgrave, which according to the Cartae Baronum return made in 1166 comprised 33 1/2 knight's fees. The main approach was located on the west, with two stone towers overlooking the entrance. Moats prohibited approach from the east and ensured that western approach was by means of a drawbridge. Differing levels of land surrounding the containing walls caused the wall to bulge outwards, which required buttressing. Some of the bricks used in the structure are clearly Roman. in his "Display of Heraldry", 1610 Nigel Fossard's son Robert died c.1135, Robert's son William I d. c. 1170, leaving a son William II who died in 1195 leaving an heiress Joan, who brought the barony and castle to her husband Robert de Turnham. Their only surviving child and heiress was Isabel de Turnham who brought the barony and castle to Peter de Mauley to whom she had been granted in marriage by King John on the escheatment of the barony. De Mauley was a native of Poitou, whose marriage to this wealthy heiress is said to have been his reward for having murdered in 1203 Prince Arthur, the son of John's elder brother who threatened his succession to the throne. He was governor of Corfe Castle in Dorset where he acted as jailer of Eleanor, Arthur's sister. Peter I's heir was Peter II de Mauley, who married Joan de Brus, one of five sisters of Peter III de Brus, feudal baron of Skelton, Yorkshire, who was his brother-in-law, having married Hilary de Mauley, Peter II's sister. In the time of Peter II the barony was held by knight service of supplying two knights in time of war in the king's presence for 40 days per annum. Peter II's heir was Peter III de Mauley, who married Nicole de Ghent, sister and in her issue co-heir to Gilbert V de Ghent, feudal baron of Folkingham, Lincolnshire. Peter III was summoned to parliament by writ dated 23 June 1296, creating him Baron Mauley. Peter III's seal can be seen as one of 72 appended to the Barons' Letter, 1301 to the Pope" sealed at the Parliament of Lincoln in January 1301, and shows him on the reverse in the usual pose for early seals holding sword and shield astride his galloping war-horse, with the tails of his surcoat swept back by the wind. His arms within a heater-shaped escutcheon show a bend with a field diapered with scroll-work, which are blazoned as borne by him on the Falkirk Roll as: Or, a bend sable. His heir was Peter IV de Mauley. Camden states that the first Peter was succeeded by 7 others bearing his name. The castle passed to Sir John Bigot of Settrington, Yorks., on his marriage to Constance de Mauley, eldest daughter & co-heiress of Peter VII de Mauley, whose son Peter VIII had died in 1415 without issue, when the barony of Mauley by writ became extinct. Bigot was 5th in descent from Hugh Bigod, 3rd Earl of Norfolk. The property passed to the Radcliffes on the marriage of Dorothy Bigot into that family, before settling in 1625 on Lord Sheffield of Butterwick, later to be titled Earl of Mulgrave by Charles I. There is evidence that the initial construction by the Fossards was updated and altered by subsequent residents. The castle was garrisoned by royalists during the English Civil War. Subsequently, it was dismantled by Parliamentary order in 1647; the lie of bricks from the destruction suggest that gunpowder might have been employed for this purpose.
Present manor
The third construction, a castellated mansion, was ordered by Lady Catherine Darnley, Duchess of Buckingham, illegitimate daughter of James II of England and then wife of John Sheffield, 1st Duke of Buckingham and Normanby. The property passed to the Phipps family in 1718, when the Duchess's daughter Lady Catherine Annesley married William Phipps. A summer house was also built on the grounds, according to legend on the spot of a hermitage which William de Percy established in 1150. In 1858 Duleep Singh, the last Maharajah of the Punjab, took a lease on Mulgrave Castle. As of 2010, the property is held by Constantine Phipps, 5th Marquess of Normanby. In 2003, supermodel Elle Macpherson took out a lease on the Mulgrave estate, considered one of England's finest shooting estates, including the right to live in the Phipps family's ancestral home, Mulgrave Castle, during the four-month shooting season.