Miller was born in Detroit, Michigan in 1886. His father, James G. Miller, was the secretary of the Union Depot Railroad in Detroit. Miller attended Detroit Central High School where he won varsity letters in football, baseball and basketball. The Detroit Free Press called him "one of the best known athletes Detroit ever has turned out." Miller was considered one of Detroit's best basketball players on defense and played end for Detroit Central's football team. The Free Press in 1907 described his high school football career as follows:
"Joy Miller was one of the best ends that ever played on the Central High school; valuable on both attack and defense, a sure tackler and powerful in the interference. Plays sent around his end usually went to pieces for little or no gain, and opposing backs have reason to remember the vigor and certainty of his tackling."
In addition to playing for Detroit Central High School, Miller also played football and basketball for the Detroit Athletic Club teams. Miller enrolled at the University of Michigan in 1906 but was not eligible to play varsity football until his sophomore year in 1907. He played in several games for the 1907 Michigan Wolverines football team, including the rivalry game against the University of Pennsylvania. He started three games for the 1907 team, two at the left end position and one at the left halfback position. Despite having played a substantial part in several games, team captainPaul Magoffin did award Miller a varsity "M" in 1907. Miller elected not to play football in 1908, but he returned to the football team in 1909. In 1909, Miller started all seven games for the Wolverines, six at right end and one at quarterback. He played the first seven games at the end position and was switched to quarterback without notice to quarterback for the season's final game against the University of Minnesota. According to the Detroit Free Press, "Miller won a high place in the admiration of his teammates through his stellar work at quarterback in the Minnesota game." At the end of the 1909 season, Miller was elected by his teammate to be captain of the 1910 team. However, in December 1910, Miller became embroiled in a scandal that attracted national newspaper coverage when it was discovered that he had played the 1909 football season without registering for or attending classes during the fall semester. The Board in Control of Athletics stripped him of his selection as team captain and his varsity letter in December 1909. Following an investigation and resolution of the University of MichiganStudent Council, the University of Michigan Engineering Faculty expelled Joy from the university in January 1910. Joy disappeared for several months during and after the investigation and expulsion and was discovered in March 1910 wandering in western Canada. Joy claimed to have lost all memory of his identity and past events. In a draft registration card completed by Miller in September 1918, he indicated that he was a resident of Highland Park, Michigan, working in the moving and storage business. At the time of the 1920 Census, James J. Miller, age 33, was a resident of Highland Park, Michigan, along with his wife Edith, son Henry L., and daughter Janet. He listed his occupation as vice president of a storage company. At the time of the 1930 Census, James J. Miller, age 43, was a resident of Grosse Ile, Michigan, along with his wife Edith, son Leonard, and daughter Janet. He listed his occupation as vice president of a warehouse.
