Born in Hackney, London, Olufemi came through the ranks at Colchester United's Academy, primarily as a winger who could also play at full-back. He began to feature in the clubs' reserve team in 2011, where he scored in a 6–0 win against Stevenage Reserves in their first game of the season on 7 September, helping the side win the last-ever Football Combination North Division title that season.
Alongside fellow Academy products Bradley Hamilton, Drey Wright, Ryan Melaugh and Shaun Phillips, Olufemi signed a two-year professional development contract with the club in May 2012. He then featured a number of times for the first-team in their 2012–13 pre-season campaign, where he scored twice in a 6–0 victory over Heybridge Swifts in the opening fixture on 14 July. He scored again in pre-season, this time for a young Colchester United XI against Needham Market in a 3–1 away defeat on 31 July. Olufemi was selected for the matchday squad for the first time in Colchester's trip to Bournemouth on 15 December, although he was an unused substitute in the U's 1–0 defeat. He made his first-team debut in a League One match away at Bury on 16 March 2013. He came on as a 70th-minute substitute for goalscorer Clinton Morrison in the 2–1 victory, impressing manager Joe Dunne with his performance. His appearance at Bury was to be his only game of the season.
Olufemi once again appeared alongside the first-team for their opening 2013–14 pre-season fixture, which again saw them come up against Heybridge Swifts. He played in the second half of the 2–1 win on 10 July 2013. With captainBrian Wilson missing much of pre-season through injury, Olufemi was handed his chance to impress at full-back, where he featured in a friendly 2–0 win at Bishop's Stortford on 13 July, as well as providing the assist for Colchester's goal in their 2–1 home reverse to Premier LeagueWest Ham United on 16 July, sending in a cross for Jabo Ibehre to head home a consolation goal. He also faced further Premier League opposition when he started against Tottenham Hotspur on 19 July in a 0–0 draw. Before making any first-team appearances in the 2013–14 season, Olufemi was offered a three-year contract extension to see him through to the summer of 2016 with the club on 20 August 2013. Following this, he was called into action to replace Ryan Dickson at left-back who was suffering from illness. He subsequently made his first start for the club in a 1–1 home draw with Carlisle United on 23 August, playing the full 90 minutes and hitting the bar with a fine strike in the second half. With Dickson returning for the U's next match on 31 August, Olufemi was required to cover Brian Wilson at right-back, who was also suffering from illness. Colchester went on to lose the game 2–1 to league leaders Leyton Orient. Olufemi then made his third consecutive start in a third different position, this time playing as a winger alongside Clinton Morrison and Gavin Massey in a 4–1 Football League Trophy defeat to Dagenham & Redbridge. He continued his run in the first-team squad in an injury depleted side during a 2–0 defeat to Coventry City on 8 September. On the back of his excellent start to the season, manager Dunne described Olufemi as a "great example for all the younger players coming through ", praising his "determination" and his "attitude and drive". A thigh injury then ruled him out of contention until 30 October, when he returned to action in the Essex Senior Cup, scoring in the 4–0 win over Great Wakering Rovers. He made his return to first-team action as a second-half substitute for Gavin Massey in a 1–1 draw with Preston North End on 23 November. Olufemi ended the campaign with 14 appearances to his name.
After beginning to establish himself in the first-team in the latter stages of 2013–14 season, Olufemi featured regularly in pre-season for the 2014–15 season, featuring in the second-half team selection in Colchester's 8–0 win against Heybridge on 15 July 2014 and starting in a 2–1 friendly defeat at Southend United on 22 July, However, he suffered a ruptured Achilles tendon injury during a friendly with Bishop's Stortford on 30 July, leaving the player out of action for at least six months. In October 2014, it was reported that he was making good progress in his recovery after the removal of a wedge from his boot cast. At this stage, he was targeting a comeback ready for the Christmas period. He returned to action with the development squad in their 2014–15 Professional U21 Development League match against Charlton Athletic on 9 February 2015, coming on as a second-half substitute in the 0–0 draw.
2015–16 season
Olufemi finally made his Colchester United first-team comeback on 26 September 2015 when he replaced the injured Kane Vincent-Young in the starting line-up of his side's 2–1 win against Swindon Town at the County Ground, playing the full 90-minutes. It was his first start since April 2014. After making eleven appearances during the season, the club announced that Olufemi would be leaving Colchester after seven years following the expiry of his contract.
