Spencer was born in Cleveland County, Oklahoma and attended Capitol Hill High School in Oklahoma City. After graduating, he enrolled at Oklahoma State University and played on the school's football team starting in 1942. After his first season, however, Spencer left Oklahoma to serve in the U.S. Army during World War II. He was in the Army's 2nd Armored Division and was the player-coach of its football team, which won the European service championship in 1945. After the war, Spencer returned to Oklahoma State for the 1946 and 1947 seasons. In 1947, he was chosen to play on the West team in the annual East–West Shrine Game, a college all-star game. Spencer had been selected by the Philadelphia Eagles of the National Football League in the 1945 draft, but he signed with the Brooklyn Dodgers of the All-America Football Conference after graduating in 1948.
Professional career
Spencer played as a tackle both on offense and defense for the Dodgers in 1948, when the team finished with a 2–12 win–loss record. As the Dodgers struggled on the field, the team also struggled financially and was forced to merge with the New York Yankees after the season. That left a number of former Brooklyn players available for other AAFC teams, and the Cleveland Browns signed Spencer. Spencer played for the Browns during the 1949 season, when the team finished with a 9–1–2 win–loss–tie record and won the AAFC championship. He was then traded to the Green Bay Packers of the NFL, where he played for two seasons and was the defensive captain before retiring due to an injury.
Coaching career
After retiring from football, Spencer launched a coaching career. His first job was as an assistant coach at Austin College in Texas starting in 1953. He was promoted to head football coach in 1955, and held that position through the 1960 season. Spencer was hired as an assistant coach for the Houston Oilers in the American Football League in 1961. The Oilers, who were coached by fellow former Browns tackle Lou Rymkus when he joined, won the AFL championship that year. The team finished with an 11–3 record the following season, but lost the championship game to the Dallas Texans. Spencer stayed with the Oilers until 1965. Spencer moved to a coaching job with the Edmonton Eskimos of the Canadian Football League in 1966 and 1967. He was then hired in 1968 as the offensive line coach for the AFL's New York Jets, who were led by Weeb Ewbank, one of Spencer's coaches when he played in Cleveland. Led by quarterbackJoe Namath, the Jets finished the 1968 regular season with an 11–3 record and beat the heavily favored Baltimore Colts to win Super Bowl III. Spencer stayed with the Jets through the 1970 season. Spencer became the offensive line coach for the NFL's St. Louis Cardinals in 1971 before taking a similar role in 1972 at the University of Kansas. He coached at Kansas under Don Fambrough through the 1973 season. Spencer's next stop was as an assistant for the World Football League's Chicago Fire in 1974. The Fire disbanded after the season, however, and Spencer got a job as an assistant with the Kansas City Chiefs, where he coached for six years. He then took a job as an offensive line coach for the New Orleans Saints in 1981. The Saints hired Carl Mauck in 1982 to work with Spencer coaching the linemen, and the following year Spencer was moved to quality control coach. He stayed in that role through the 1985 season.
Spencer and his wife, Betty Jo, had two children. He died of cancer in 1996. In 1997, Austin College established the Coach Joe Spencer Award for Meritorious Service and Lifetime Achievement in Coaching in his honor. The award is given to Austin College alumni and staff who have distinguished athletic coaching careers. Spencer was also inducted into the Austin College Athletic Hall of Honor in 1969.