Late in 1776, Virginia's Revolutionary government constructed the fort of timber and rammed earth. It was built for a garrison of 150 men near previous entrenchments from 1774, with the work supervised by Benedict Arnold, who was with the patriot forces at the time. In 1779, a British fleet commanded by Admiral Sir George Collier confiscated its artillery and supplies and destroyed most of the parapet; the garrison evacuated the fort before it was captured. During 1781, both Benedict Arnold and Lord Cornwallis occupied the fort. A 1781 British map of fortifications erected by them on the Elizabeth River indicates that they rebuilt Fort Nelson.
The fort was reconstructed in 1794 as an earthworkbastion fort under what was later termed the first system of US fortifications. Fort Norfolk was built at the same time to create a crossfire with Fort Nelson on enemy ships in the Elizabeth River. In 1795 Captain Richard S. Blackburn's company was being formed in the Norfolk area, and by 1799 garrisoned both Fort Norfolk and Fort Nelson. In 1802–1804 the fort was again rebuilt of earth lined with brick, to a design by architect B. Henry Latrobe as part of what was later termed the second system of US fortifications. The work was initially supervised by Major Decius Wadsworth, and the rebuilt fort had a capacity of 37 guns with barracks for one company. In 1807 the Chesapeake–Leopard affair off Hampton Roads caused a heightened state of alert at Fort Nelson, then commanded by Captain John Saunders. Heavy mortars were ordered transferred from Newport, Rhode Island to the fort. The Army's report on fortifications in December 1808 stated that Fort Nelson had been strengthened and materials for a new battery at Hospital Point had been procured. By the end of 1809 the fort was "supported by half bastions" and mounted 33 guns. Although Fort Nelson itself never saw conflict, it was garrisoned during the War of 1812. An elongated chain was stretched from Fort Norfolk to Fort Nelson in order to prevent the British fleet from attacking Gosport Navy Yard, Norfolk, and/or Portsmouth. In the main action of the war in the Norfolk area, the British were repulsed in the Battle of Craney Island and did not enter Norfolk. However, they proceeded up Chesapeake Bay to burn Washington, D.C. and unsuccessfully attack Baltimore, as there were no forts guarding the mouth of the bay at the time. This led to the building of Fort Monroe beginning in the 1820s, to close the bay to enemy vessels. This made the Elizabeth River forts obsolete. Fort Nelson was abandoned in 1824 and torn down in 1827 to make room for the naval hospital; reportedly much of the fort's brick was re-used in building the hospital.
During the Civil War, the Confederate government rebuilt Fort Nelson as an 18-gun battery. However, Fort Nelson came under federal control on 10 May 1862, when the Union army, under the leadership of Major General John E. Wool, occupied the Norfolk area following a Confederate withdrawal. It is unclear when the fort was abandoned and demolished, probably in 1865 after the Civil War ended.
Present
No trace of the fort remains today. In 2006 Fort Nelson Park was opened on the site of a Civil War-era gasworks, south of the hospital property and some distance from Hospital Point, which is in the northeastern part of the hospital property. The park commemorates the fort in a publicly accessible area. The park has several naval guns and other nautical items. The Fort Nelson Chapter of the Daughters of the American Revolution is named for the fort.
