Downer's previous clubs include Leyton Orient, Aldershot Town, Weymouth, Grays Athletic, Wivenhoe Town, Rushden & Diamonds and Sutton United. Having played for the West Ham youth academy, Downer, who at the time was playing for Leyton Orient, went on trial with Newcastle United. However, Bobby Robson decided against signing the 19-year-old Downer, as he had a recurring problem with his left knee that needed an operation. This has hampered his career and at one point he retired from football. In between his 'retirement', Downer worked as a bricklayer. On 26 May 2009, Downer signed a one-year contract to remain at Rushden & Diamonds for the duration of the 2009–10 season. During the close-season after, he decided not to renew his contract offered and signed with Sutton United, with whom he won the Isthmian League Premier Division at the end of the 2010–11 season. He scored his first Sutton league goal in a 4–3 home victory over Wealdstone on 27 November 2010. He won the Players' Player of the Year award for the 2011–12 season. Downer rejoined Sutton in 2015 for a third spell after spending the previous campaign at Maidenhead. He made his 200th career appearance for Sutton in September 2015 and was part of the team which won the National League South title. On 4 December 2016, Downer appeared in The Non-League Paper's team of the day alongside teammate Roarie Deacon and manager Paul Doswell following an impressive performance in Sutton's 2–1 victory over League Two side Cheltenham Town in the 2nd round of the FA Cup on 3 December. Downer appeared in every round of Sutton's historic run to the 5th round of the FA Cup for the first time ever, including a 3–1 victory over League One team and local rivals AFC Wimbledon on 17 January 2017, a 1–0 victory over Championship team Leeds United on 29 January and a 0–2 defeat to Premier LeagueArsenal on 20 February. Downer appeared again in the NLP team of the day after a 3–2 away victory against Torquay United on 25 February, when he was forced to step in as goalkeeper following an injury to Ross Worner. He made his 250th appearance for Sutton United in February 2017.
