Fraser was born on 21 August 1832 in Big Brook, a small rural township in Pictou County, Nova Scotia, Canada. He was the youngest son of Jane and William Fraser. His parents shared the same surname, although no familial connection has been noted between the two. Fraser's father was born in Beauly, Inverness-shire, Scotland, and claimed descent from Clan Fraser of Lovat. He arrived in Nova Scotia in 1801, as a small child. According to Fraser, his mother spoke fluent Scottish Gaelic and one of his grandfathers spoke "very little English". In 1906, he would become the inaugural patron of the Gaelic Society of Victoria, an organisation devoted to keeping the Gaelic language and customs alive in Australia. In an address to the society, he said that he could speak only a few sentences of Gaelic, but that he could still understand most conversations. In 1839, when Fraser was seven years old, his father was killed in a sawmilling accident. He began working on the family farm at a young age, and received limited formal schooling. Attracted by the Victorian gold rush, he immigrated to Australia at the age of 21, arriving in Melbourne in 1853. He immediately went to the goldfields at Bendigo, where he turned a profit selling supplies to miners.
Business career
In 1855, Fraser moved to Melbourne and opened a store on Elizabeth Street. One of his more notable contracts was to supply ballast to the Deniliquin and Moama Railway Company, a privately owned railway which connected Moama on the Murray River to Deniliquin in southern New South Wales. Instead of supplying blue metal, Fraser supplied quartz from the slag heaps of Bendigo gold mines. It met the specifications of the contract, but was not what was expected by the owners of the railway. Fraser later bought extensive estates in the Western District of Victoria and became a leader of the wealthy wool-growing class known as the Squattocracy.
Fraser married Margaret Bolger in 1862 and had two daughters. Following her death in 1880, he married Anne Collins in 1885 and had three sons with her. His brother-in-law was Robert Martin Collins, a member of the Queensland Parliament. One of these, Neville Fraser, inherited Simon Fraser's property at Balpool in the Riverina district of New South Wales, where the future Australian Prime MinisterMalcolm Fraser grew up. Another son, Simon, was a champion footballer and rowed for Australia in the 1912 Stockholm Olympics. Simon died just two months before his father, aged 32. Fraser died of bronchitis on 30 July 1919, aged 86, in Melbourne. He was survived by his wife, Anne, a daughter and two sons.