Recruited from the Alphington Football Club, Chanter signed with Fitzroy in 1946, avoiding being bound to Collingwood under new zoning laws which were soon to be adopted. His father Fred Chanter had played one senior game for Fitzroy, in the round seven match against Essendon, at the Brunswick Street Oval on 12 June 1920. Chanter made his debut for Fitzroy, aged 25, on 25 May 1946, against South Melbourne at the Brunswick Street Oval; South Melbourne won the match, 10.12 to 7.21. He played 5 senior games in 1946, and 13 in 1947. When Fred Hughson left Fitzroy at the end of the 1947 season, Chanter took over as fullback. Chanter developed a reputation as a tough and physical player, who was reviled by fans of opposing teams. After a match in August 1949 at Footscray where he was booed throughout the game, a journalist for the Sporting Globe wrote that "Chanter is a vigorous player. As I see him, he can be over-vigorous at times, but that does not excuse the behavior of some crowds towards him. There was another demonstration against him at Footscray on Saturday and in the opinion of Frank Walsh it was most unfair and entirely undeserved." Chanter stated later that it was nothing new, saying "At Geelong, I was hit with a stone that I saw deliberately thrown at me. At Collingwood, they threw bottles and blue metal at me. And in the much-discussed match at Essendon, after which I had to appear before the Tribunal, spectators spat in my face. I play the game hard and I don't squeal when the game is played hard against me. But surely there is a limit." In 1951, he was full-back in the Victorian State team that played against South Australia at the MCG on Saturday, 26 May 1951. He was one of the best players on the ground in Victoria's unconvincing eight point win, 10.11 to 9.9. In 1951 he became the first Fitzroy fullback to win the club's Best and Fairest award. In 1952, he was vice-captain; and, on Saturday, 28 June 1952, in round ten of the 1952 season, at a very, very muddy Brunswick Street Oval, in a tough, rugged match where Fitzroy 13.12 beat Essendon 5.8, Chanter played at full back against Essendon champion full-forward, John Coleman. Coleman, who would finish the season with 103 goals, did not score a goal in the match; and this was the first time that Coleman was held goal-less in his entire 98 game career. He had less than half a dozen kicks for the entire match, and was only able to score two behinds, one of which was scored in the last scoring kick of the match. He played his last game for Fitzroy in the 1952 Preliminary Final against Collingwood, which Collingwood won, 11.15 to 9.8.
