The Hermitage was originally built in 1830 by the Reverend George Sheed. The property is about two miles west of Ancaster, in the Dundas Valley. Otto Ives was the third land owner. He was an English officer who had fought in the Greek War of Independence against the Ottoman Empire. He met Magdalene Diamanti, a daughter of the Governor of an Aegean island. They married in Corfu in 1824, and had arrived in Ancaster by 1833. They brought her sister or niece with them to act as a companion for Mrs Ives. It was here that Otto Ives purchased the Hermitage from the heirs of the Rev. George Sheed. Ives had hired a coachman by the name of William Black. Although it is said that Black was also a tutor in the English language, evidence exists that this post was filled by Mary Rosebeury. The ladies of the household spoke only Greek, and it is said that Black fell in love with the sister or niece. Black went to Otto Ives and asked for his niece's hand in marriage. Ives was very upset by the thought, and rejected the proposal. The next morning, Ives and his wife were to go out for the day, but the coachman was not at the front door with the carriage as planned. Ives went out to the barn to see why the coachman had not appeared, and discovered Black's body dangling from the rafters in the barn near the first Hermitage. Although this story has become legend in Ancaster, and although Otto Ives owned the Hermitage in the 1830s, there is little other information from the time to substantiate the legend of the coachman and the niece's love affair. There is also a note appended to deeds of sale, mentioning that a family friend had hanged himself from a tree in the bush, because of love for the niece. In 1853, the Hermitage was purchased by George Gordon Browne Leith and his wife Eleanor Ferrier. Over the next several years, a large stone house, attendant outbuildings, a farmhouse, barns, and entrance lodge were constructed. After Mrs Leith's death, the property was purchased by her youngest daughter, Eleanor Alma Dick Lauder. She lived here until the house was destroyed by fire in 1934, and afterwards, in a small house constructed inside the ruins.