Carroll was promoted to the rank of captain in the 10th U.S. Infantry shortly after the Civil War erupted. In December 1861, he was appointed as colonel of the 8th Ohio Infantry, a three-years' regiment that exclusively saw duty in the Eastern Theater. During the 1862 Valley Campaign, Carroll commanded the 4th Brigade in James Shields' division. Carroll commanded the vanguard of the Union army at the Battle of Cross Keys. He was commended for his performance at the Battle of Cedar Mountain. Transferred with this brigade to the Northern Virginia area, he was severely wounded in the chest in a fight near the Rapidan River. He recovered in time to resume his field command in III Corps before the Battle of Fredericksburg. In 1863, Carroll commanded the 1st Brigade, 3rd Division of the II Corps at the Battle of Chancellorsville. During the second day at Gettysburg, his 8th Ohio was involved in skirmishing along the Emmitsburg Road, while the other three regiments of the Gibraltar Brigade, 14th Indiana, Col. John Coons 4th Ohio, Lieut. Col. Leonard W. Carpenter and 7th West Virginia, Lieut. Col. Jonathan H. Lockwood (counter-attacked oncoming Confederates from North Carolina and the Louisiana Tigers from Jubal Early's division on the slopes of Cemetery Hill, driving them back in the growing darkness. Carroll later led his depleted brigade in some small engagements during the Mine Run Campaign. Carroll was promoted to brigadier general on May 12, 1864. He was twice wounded during the Overland Campaign, once at the Wilderness and again at Spotsylvania Court House. His left arm was amputated. After his recovery, he commanded the Department of West Virginia briefly and later led a division in the Army of the Shenandoah.
Post-war career
When the war ended, Carroll stayed in the postbellumRegular Army, serving in the inspector general's department. Partially invalided by his war-time injuries, he retired from the army in 1869 with the brevet rank of major general. In August 1886, his wife divorced him.
Death and legacy
In late January 1893, Carroll contracted pneumonia and died six days later at the age of 60 at his country residence, "Belleview", near Takoma Park, Maryland. His death was hastened by his never having fully recovered from his wounds. His funeral was held in St. John's Church in Washington, D.C.. He was buried with full military honors in Oak Hill Cemetery in Georgetown, Washington, D.C.. The main street in Takoma Park is named Carroll Avenue in his memory.