In the early 19th century, sheep raising became popular and multiple woolen mills sprang up along the branches of the Saxtons River. Soapstone was quarried on nearby Bear Mountain. The town became a notable stagecoach hub for traffic across the Green Mountains into Albany, New York. One inn from that era, "the Old Tavern," was founded in 1801. It remains one of the oldest continually operating hotels in the United States. Grafton had a population of almost 1,500 just before the American Civil War. The town suffered severe losses during the Civil War. Local cemeteries in the village hold many tombstones of casualties from the Battle of Gettysburg. After the war, the community declined in population. The soapstone quarry was depleted and closed late in the 19th century. Around the time of the Great Depression, the town's population was less than 400. In the 1960s the Windham Foundation was established to help restore the village. It purchased the Old Tavern and many residences in the area. The foundation also established an artisanal cheese business, the Grafton Village Cheese Company, and built a world-class cross-country skiing center at Grafton Ponds that offers a popular mountain biking program for children in the summer months. The restoration efforts attracted new residents from metropolitan New York and Boston.
As of the census of 2000, there were 649 people, 291 households, and 190 families residing in the town. The population density was 16.9 people per square mile. There were 434 housing units at an average density of 11.3 per square mile. The racial makeup of the town was 99.23% White, 0.31% Asian, and 0.46% from two or more races. Hispanic or Latino of any race were 0.77% of the population. There were 291 households, out of which 21.3% had children under the age of 18 living with them, 56.4% were married couplesliving together, 4.8% had a female householder with no husband present, and 34.4% were non-families. 28.5% of all households were made up of individuals, and 10.7% had someone living alone who was 65 years of age or older. The average household size was 2.23 and the average family size was 2.73. In the town, the population was spread out, with 18.2% under the age of 18, 5.5% from 18 to 24, 24.7% from 25 to 44, 33.4% from 45 to 64, and 18.2% who were 65 years of age or older. The median age was 46 years. For every 100 females, there were 114.2 males. For every 100 females age 18 and over, there were 113.3 males. The median income for a household in the town was $42,313, and the median income for a family was $48,250. Males had a median income of $31,346 versus $21,406 for females. The per capita income for the town was $23,617. About 6.3% of families and 10.6% of the population were below the poverty line, including 11.0% of those under age 18 and 12.0% of those age 65 or over.
