Robert Duffy, who is older brother to footballer Richard Duffy, played his junior football in Swansea for Garden Village. He came through the youth system at Charlton Athletic, but was never offered a professional contract. Duffy followed Charlton's youth team manager Terry Westley to Rushden & Diamonds in 2001, where he made over 30 appearances, most as a substitute. He fractured his kneecap in a training ground accident in August 2003 and did not play again until January 2004, when he played most of a reserve game. He struggled to regain form, was loaned to non-league club Stamford and had short spells with Cambridge United and Kettering Town before joining Gainsborough Trinity in January 2006. Duffy ended the 2005–06 season at Stevenage Borough.
Oxford United
Duffy was recommended to Oxford United by former youth development officer, Malcolm Elias, and joined the club on a one-month trial in July 2006. He was signed by Oxford at the beginning of the 2006–07 season on a two-year contract, and was Oxford United's top scorer in that season, including a hat-trick against Forest Green Rovers in October 2006. In September 2007, Duffy broke his left arm after falling awkwardly during a match, which sidelined him for two months. In November, he returned to action as a late substitute in the 78th minute against Ebbsfleet United, but was sent off just 20 seconds later after an off-the-ball incident with Ebbsfleet's Mark Ricketts as he went past him. Duffy was placed on the transfer list in January 2008 as manager Darren Patterson tried to free up funds to bring in new players, and joined Wrexham on loan later that month. He made only six substitute appearances for Wrexham and in April 2008, was released and returned to Oxford.
Newport County
He was released by Oxford at the end of the season and, after an unsuccessful trial at Mansfield Town, he signed for Conference South side Newport County in July 2008.
Mansfield Town
On 7 January 2009 Duffy then signed permanently for Mansfield Town, managed by David Holdsworth, twin brother of his manager at Newport County, Dean Holdsworth. Despite his lack of goals at Newport, David Holdsworth's faith in Duffy was repaid as he scored the winning goal on his Mansfield debut, going on to score eight further times as Mansfield pulled away from the Conference relegation zone to finish in mid-table.
Grimsby Town
On Monday 24 January 2011 it was stated on BBC Radio Humberside that Duffy had failed to agree personal terms with Grimsby Town after a fee was agreed between the two clubs. After further negotiation, it was announced on 28 January that Duffy had signed a two-year deal with the Mariners. Duffy scored 28 seconds into his Grimsby debut in their 3–2 win over Eastbourne Borough on 29 January 2011 and three days later he scored again as Town drew 2–2 away at Southport. Following the conclusion of the 10–11 season and despite being under contract, Duffy was told that he no longer had a future at the club and he was free to find himself a new one. On 16 May 2011 Town chairman John Fenty mentioned that the club had made a mistake in signing Duffy and teammates Charles Ademeno and Serge Makofo. Duffy however stayed for nearly the entire 11–12 campaign, and after playing a cameo role all season he was released before the final game of the season.
Lincoln City
On Monday 21 May 2012 Duffy signed a 1-year contract with Lincoln City. On 9 April 2013 Duffy was released by the club after an injury hit season had restricted him to only 3 games during the 2012–13 season.
