Southwarth was born in Blackburn to Robert and Martha Southwarth. He played alongside his younger brother Jack Southworth for most of the early part of his career, joining him at Chester and Blackburn Olympic. At Chester he missed the 1885 Boxing Day fixture as he was appearing in pantomime. In his role as a conductor, at the Royalty Theatre in Chester, he would have to be back in the city after away matches in time for 7.30pm performances, affecting the number of appearances he made for the club. On the field, an early squad summary by the Chester Chronicle said he was captain and a "dashing and safe back". In 1888, he moved with his brother from Blackburn Olympic to their neighbours and fierce rivals Blackburn Rovers. The first season of the Football League began, for Blackburn Rovers, on 15 September 1888 at Leamington Road, then home of Blackburn Rovers. Southworth played at left-back in this match and Rovers drew with Accrington 5-5. James Southworth played in 19 of the 22 League matches played by Blackburn Rovers in season 1888–89. James Southworth played in the two FA Cup semi-final matches against Wolverhampton Wanderers. Blackburn Rovers lost the replay 3–1. As a full-back he played in a defence that achieved three clean sheets and kept the opposition to one-League-goal-in-a-match on three occasions. Blackburn Rovers ended the inaugural League season in fourth place. He missed most of the 1889-90 league season but was selected for the FA Cup final against Sheffield Wednesday at The Oval on 29 March1890. Rovers were the favourites to win in view both of their record of three victories in the previous six seasons and their superior league placing. Blackburn fielded a teamconsisting of nine England or Scotland internationals. Rovers lived up to expectations as they comfortably won the Cup defeating their Yorkshire opponents 6–1 with goals by Billy Townley, Nat Walton, Jack Southworth and Joe Lofthouse. As Philip Gibbons wrote in his book Association Football in Victorian England: "The Blackburn side had given one of the finest exhibitions of attacking football in an FA Cup Final, with England internationals, Walton, Townley, Lofthouse and John Southworth at the peak of their form." The F.A. Cup final was James Southworth's final appearance for Rovers. He made 21 league and 6 cup appearances.
