Born in Bristol, Milsom began his career with his hometown club Bristol City. He made his professional debut for the club during the 1993–94 season but was released in March 1995 having made two further appearances. He joined Cardiff City, making two appearances as a substitute before starting in a 1–0 defeat to Bristol Rovers before being released at the end of the season. He subsequently signed for Oxford United, making one appearance for the club in the Football League Trophy, before dropping out of professional football. He signed for Gloucester City in 1995, scoring on his debut against Chelmsford City. In two seasons at Meadow Park, he scored 13 goals and 60 appearances. After spells with Trowbridge Town and Clevedon Town, Milsom signed for Bath City on a two-year contract in July 2001 for a fee between £2000 and £3000, rejecting an offer from Worcester City. A first team regular for two seasons, he made over 60 appearances in all competitions for Bath, before joining Tiverton Town in 2003. He made his debut for the club on 20 September 2003 in a 3–3 draw with Chelmsford City and scored his first goal two months later in a 1–0 victory over Dover Athletic. At Tiverton, Milsom also began playing as a central defender. He made over 100 appearances for the club before leaving in 2006. Milsom spent one season with Chippenham Town before joining Mangotsfield United in 2007. He was appointed manager of Mangotsfield during the 2007–08 season and helped the club avoid relegation after winning five consecutive matches. The following season, he was fired in January 2009 as the club struggled with Milsom blaming financial issues at the club, stating his budget had been "cut, and cut, and cut again." In March 2011, Milsom was appointed joint-manager of Paulton Rovers along with Mark Harrington. Eventually, Harrington took over sole managerial responsibility with Milsom remaining as his assistant. Harrington was eventually replaced by Nick Bunyard with Milsom remaining as his assistant. The pair led the club to its highest ever league position before stepping down in 2015. The pair took charge of Frome Townin September 2015. However, Milsom left his role after less than a year.
