Schwartz was born in 1906 and grew up in Chehalis, Washington. He enrolled at Washington State College, where he played college football from 1928 to 1930. During the 1928 season, he played at the guard position and was named to several all-coast teams. In 1929, Washington State coach Babe Hollingbery moved Schwartz to the fullback position. Schwartz excelled in his first year at the fullback position, scoring 13 touchdowns in the first eight games of the season. According to the Associated Press, his total of 78 points in the first eight-game ranked him as the third highest scorer in college football behind Alton Marsters of Dartmouth and Gene McEver of Tennessee. During the 1930 season, Schwartz was the captain of the Washington State football team that won a Pacific conference championship and a berth in the 1931 Rose Bowl. He was described by George Kirksey of the United Press as "a star at crashing the line on offense and backing up the line on defense." The Cougars lost to Alabama in the Rose Bowl by a score of 24 to 0. At the end of the 1930 season, Schwartz was selected as the third-team fullback on the Associated Press' All-American team. He graduated from Washington State in June 1931. After his death in 1949, former coach Hollingbery said, "Elmer was one of the all-time football greats at Washington State."
Professional football
In August 1931, Schwartz announced that he had agreed to play professional football for the Portsmouth Spartans of the NationalFootball League. Schwartz appeared in 12 games and scored two touchdowns for the 1931 Spartans team that finished second in the NFL with an 11–3 record. In 1932, Schwartz joined the Chicago Cardinals. Schwartz was injured in the third game of the 1932 season against the Chicago Bears. Schwartz appeared in only three games for the Cardinals. In July 1933, Jap Douds, the head coach of the Pittsburgh Pirates, announced that Schwartz had been signed to play fullback for the 1933 season. Schwartz appeared in 10 games for the Pirates, gaining 94 yards on 38 carries.
Later years and death
Schwartz died in March 1949 from a cerebral hemorrhage after being struck by a construction company security guard. The guard told police he struck Schwartz after Schwartz "refused to leave the property where was guard." After a preliminary hearing, the guard was ordered to stand trial for murder. Witnesses testified that the guard "beat Schwartz to the ground twice in three separate attacks after ordering Schwartz from the grounds of the T. E. Connelly company construction camp near Pulga."