Returning to Virginia, Barbour moved to the state's northwest corner. Monongalia County voters once elected him as one of their two representatives in the Virginia House of Delegates, where he served in 1857-58. In January 1859, he was appointed as Superintendent at the federal armory in Harpers Ferry, Virginia. He served there until the AmericanCivil War began in 1861. In October 1859, abolitionist John Brown raided the arsenal. Brown's raiders captured the entire armory and town, which Brown knew to be minimally guarded by civilians, although ultimately he failed and was captured because he remained in town too long. Recent research questions whether Brown really attempted to steal the weapons to support a slave rebellion, considering that explanation Virginia slaveholder propaganda. Barbour wrote that he was visiting the federal armory at Springfield, Massachusetts. "Had I been here, I could have done no good. Old Brown would have taken Gen. Scott if he had been here. A military man could have done nothing more than a civilian, unless there had been a corps of soldiers under him.... It is ridiculous to talk about it, as if the presence of a military man would have awed Old Brown." Despite the fiasco, voters from Jefferson County elected Barbour and fellow former delegate Logan Osburn to represent them in the Virginia Secession Convention of 1861. While Osburn resolutely voted against secession on both April 4 and April 17, Barbour switched his vote, voting on April 17 to secede, as did his brother James Barbour, who was a delegate representing their native Culpeper County.
American Civil War
During the American Civil War, Barbour served in the Confederate States Army as a quartermaster, on the staff of Joseph E. Johnston as well as Leonidas Polk. By December 1861, he had been promoted to quartermaster of all the Confederate armies, but complaints soon arose that he failed to settle accounts, gambled extensively and managed loosely. Major Barbour served in Meridian, Mississippi and by 1864 was demoted to assistant quartermaster in Montgomery, Alabama, where he remained after the war ended, although paroled in Greensboro. Jubal Anderson Early disliked Barbour, who termed him "not energetic or efficient."
Death
Alfred Madison Barbour died on April 4, 1866, in Montgomery, Alabama. His remains were returned to Culpeper, Culpeper County, Virginia and interred at Fairview Cemetery.