It is now uninhabited, but at one time a Norse hall stood there, and it was the scene of the killing of Earl Erlend Haraldsson by Earls Rögnvald Kali Kolsson and Harald Maddadsson in 1154. Erlend celebrated after returning to the islet for Christmas and retired to his ship the worse for drink on the night of 21 December. Despite being warned of an attack by Sweyn Asleifsson and the presence of a full moon his men were taken by surprise by the attack by his co-rulers and Erlend was killed. Later a small nunnery was built on the islet leading to a legend that no frogs or toads could live there. It is also said that unmarried woman who became pregnant would go there to pray at an abandoned shrine to St Mary. Jo Ben's 1529 Descriptions of Orkney says of Damsay:
Here there are no hills, and it is the most pleasant of all, and is called Tempe. The church in this island is dedicated to the Virgin Mary, to which many pregnant women make visits in style. No frogs, toads, or other noxious terrestrial animals whatever are ever found here. The women here are sterile, and if they do become pregnant never bring forth with life. It is related that sometimes the haughty are carried away for the space of one hour, but truly afterwards restored. The distance of this island from Kirkwall is two miles.
Archaeology
Archaeologist Caroline Wickham-Jones has discovered a number of submerged structures off Damsay, which appear to be of Neolithic origin. She said: "We have certainly got a lot of stonework. There are some quite interesting things. You can see voids or entrances... The really interesting thing about this bay is the stories relating to things under the sea and sea-level change. Our ancestors were dealing with similar problems to ourselves and we'd like to see how they coped with it." She described a feature like a "stone table" made up of "a large slab about a metre and a half long and it's sitting up on four pillars or walls" and said that the "quality and condition of some of the stonework is remarkable. Nothing like this has ever been found on the seabed around the UK." A fieldwork report published in 2010 stated that "local ethno-archaeological information suggests the presence at one time of a causeway to the west, across a stretch of shallow water and existing skerries" that connected Damsay to the mainland.