Frazer was born in 1827 in Hardin County, Tennessee. He was a brother of Charles W. Frazer, who also would serve as a Confederate general. In 1845 Frazer entered the United States Military Academy at West Point, and graduated four years later standing 34th out of 43 cadets. He was appointed a brevetsecond lieutenant in the 2nd U.S. Infantry Regiment on July 1, 1849. Frazer was promoted to second lieutenant on June 30 the following year, and to first lieutenant in the 9th U.S. Infantry on March 3, 1855. Two years later he was promoted to the rank of captain on May 1. Frazer had spent his U.S. Army career on "routine garrison duty at various points" across the United States, until resigning his commission, which was accepted on March 15, 1861.
Civil War service
When the American Civil War began, Frazer chose to follow the Confederate cause. At the time he was likely living in Mississippi, the state that had appointed Frazer to West Point and to the Confederate Army in 1861. On March 16 Frazer was commissioned a captain in the Confederacy's regular army infantry. On April 5 was ordered to Louisiana to supervise the regular army recruiting for Baton Rouge and for New Orleans, Frazer's headquarters. On May 11 he was ordered to end his recruiting duties and proceed to Baton Rouge and directly command the city's barracks. On June 17 he was appointed to the provisional army in the 8th Alabama Infantry with the rank of lieutenant colonel. Frazer chose to resign from the Confederate Army in March 1862, but re-entered the service as colonel and commander of the 28th Alabama Infantry on November 2.
Cumberland Gap
Following the Kentucky Campaign, Frazer again resigned from the Confederate Army in late 1862. His most notable military service occurred after he was appointed a brigadier general on May 19, 1863, and given command of the 5th Brigade of the Army of Tennessee that July. His force consisted of about 2,300 men in three regiments and a battery of artillery, and was ordered to defend the Cumberland Gap "at all hazards" by area commander Maj. Gen. Simon B. Buckner, who was himself ordered to rejoin the Army of Tennessee just prior to the Battle of Chickamauga. Frazer began strengthening the fortifications already there to try to block any Federal advance into East Tennessee. That fall, forces under Union Maj. Gen. Ambrose Burnside moved toward the Cumberland Gap. Burnside sent a brigade under Col. John F. DeCourcy to deal with Frazer's men, while the rest of the Union soldiers quickly marched on and captured Knoxville, Tennessee, on September 2. DeCourcy threatened the Gap from the north, but Frazer did not think this force sufficient to remove him and held his position. On September 7, Burnside sent another brigade led by Brig. Gen. James M. Shackelford, who approached Frazer's men from the south and demanded surrender of the Confederates, which Frazer again rejected. On September 8, both DeCourcy and Shackelford sent messages requesting Frazer's surrender, and those too were rejected. The next day, Burnside himself along with yet another brigade closed on Frazer's position. At about 10:00 a.m., Burnside demanded a surrender, saying he now had enough soldiers to assault and take the Confederate garrison. This was finally enough to convince Frazer to give up the Gap, which he did at about 3 p.m. on September 9. No blood was shed in the three-day Battle of the Cumberland Gap. in relation to the Wilderness Road from Virginia into Kentucky Burnside had succeeded in deceiving Frazer, leading him to believe that the opposing Federals were much stronger than they appeared. Frazer thought his Confederates to be surrounded, outnumbered, and lacking enough provisions needed for a prolonged siege, and he unconditionally surrendered his garrison. The Confederate defeats at Vicksburg and Gettysburg earlier in July may also have affected Frazer's decision. As they lined up, the Confederates were shocked to see the size of the force to which they had surrendered. Although around 100–300 men did escape through DeCourcy's lines just after the surrender, Frazer, his men and their weapons, and 14 guns were turned over to Burnside. This also gave the Union control of the Cumberland Gap, which it retained until the war's end.
Confinement
Frazer was sent to join other captured Confederate officers at Fort Warren in Boston Harbor, where he spent the rest of the war. During this time the Confederate Congress took up Frazer's nomination to brigadier general. Due to public and political criticism of his actions at the Cumberland Gap, Frazer's appointment was promptly rejected on February 16, 1864. After the conflict ended he was finally released and paroled from Fort Warren on July 24, 1865.
Postbellum
After the Civil War Frazer relocated to Arkansas, where he began operating a plantation. Several years later he moved to New York City, where he engaged successfully in business. Frazer died there from an accident in the spring of 1906, and was buried in Clifton Springs.
