The Gunditjmara aboriginals lived in the Port Fairy area prior to European arrival. In the early 19th centurywhalers and seal hunters used the coast in this region. The bay is reported to have been named by the crew of the sealing cutter The Fairy in 1828. Whatever its origins, the name Port Fairy was in general use by 1835. John Griffiths established a whaling station in 1835 and a store was opened in 1839. In 1843, James Atkinson, a Sydney solicitor, purchased land in the town by special survey. He drained the swamps, subdivided and leased the land, and built a harbour on the Moyne River. He named the town "Belfast" after his hometown in Ireland. In the 1840s, significant conflict between pastoral squatters and aboriginals occurred. In 1842, 27 squatters from the Port Fairy neighbourhood signed a letter to Superintendent Charles Latrobe reporting many 'outrages' committed by the 'natives' and requesting the government provide security. These clashes later known as the Eumeralla Wars, formed part of the battle over land use and resources between traditional owners and Europeans across the Victoria's western district. The post office opened on 1 July 1843 Agriculture developed in the region, and Belfast became an important transport hub. By 1857 the town had a population of 2,190. In the mid-to-late 19th century, Belfast was one of Australia's largest ports, catering to the whaling industry. In 1887 the town was renamed Port Fairy as a result of an Act of Parliament. A railway was extended to the town in 1890, but closed in 1977. It has been converted into the Port Fairy to Warrnambool Rail Trail. The Port Fairy Magistrates' Court closed on 1 January 1990.
At the, Port Fairy had a population of 3,340. Its main industries are tourism and fishing, and it is the home port for one of Victoria's largest fishing fleets. A pharmaceutical factory owned by Sun Pharma is located on the outskirts of the town. Port Fairy is home to two Primary school education facilities, The Port Fairy Consolidated School and St. Patricks Parish Primary School. Port Fairy was voted as one of the world's most livable cities with a population under 20,000 after winning the 2012 International LivCom award. Port Fairy has a rich history and 50 buildings are protected by the National Trust of Australia. Griffiths Island nearby holds a breeding colony of the short-tailed shearwater or Australian muttonbird. The Port Fairy Folk Festival is held during the Labour Day long weekend in March each year. The festival has run continuously since 1977. In 2016, Port Fairy will celebrate the 40th edition of the Folk Festival from 11 to 14 March. Over the 40 festivals there have been around 3,500 acts including over 500 international acts and over 12,000 musicians to an audience of beyond 240,000 ticket holders and 1,000,000 attendees. The other major celebration in the town is the , held over summer from Christmas Eve to 26 January. Port Fairy also has a highly rated 18-hole links golf course, the Port Fairy Golf Club, situated on Woodbine Road. The town has an Australian rules football team, the Port Fairy Seagulls, which plays in the Hampden Football League. Port Fairy has many surfing spots for all skill levels including the Old Passage, a right hand rocky break at the old entrance to the Moyne river; the Lighthouse, a right-hand point break off Griffiths island; and the East Beach with many right and left beach breaks.
