The site of the castle, in the west of the town is likely to have been a Roman encampment, as it has an irregular and quadrilateral layout.
Fifteenth century origins
Penrith Castle was built between 1399 and 1470 as a defence against Scottish raids; it has been said that, unlike so many of its counterparts in the north, 'the building exhibits no indication of very ancient date.' The lordship of Penrith was created in 1397 as a grant to the newly created earl of Westmorland, but the castle was first mentioned in a grant of 1437. Previously believed to have been first built by William Strickland who later become Bishop of Carlisle, the lateness of this reference, it has been suggested, indicates that the most likely builder was Richard Neville, Earl of Salisbury. However, it is not known whether this was ‘new work on a fresh site, or whether he simply utilized Strickland’s works as the core of his building.’ It is now considered more likely that Strickland built Hutton Hall, near the church in Penrith, instead, and furthermore that a 'tenurial reconstruction shows that he did not hold the castle site.' There is a possible reference to a construction in the site of the castle as early as 1412, and Ralph Neville is known to have granted the building of a tower there, from his comital seat at Raby, the next year. Either way, it seems most unlikely that 'it was built not by a baron, but by the inhabitants themselves, for their own defence,' as was once stated. The liberty and castle of Penrith became the most important offices held by the Neville family in the fifteenth century, and their most important source of patronage locally. By 1441, Salisbury was sub-letting the lordship of Penrith to Lumley, bishop of Carlisle, until 1444, who was then Warden of the West March, and Professor A.J. Pollard has estimated that 'the Nevilles’ Penrith estates were worth approximately £350.’ Following Salisbury's death in 1460, Richard, Earl of Warwick, the 'Kingmaker,' inherited the Castle and Lordship, but was himself slain at the Battle of Barnet without leaving a male heir, so they reverted to the crown. They were granted in 1471 to Richard, Duke of Gloucester by King Edward IV, who used Penrith as a base whilst 'taking effectual measures' against the Scots, and also 'enjoyed the revenues of the estates' of the Forest of Cumberland. It was at the same time that the duke was appointed sheriff of Cumberland five consecutive years, being described as 'of Penrith Castle' in 1478.