Armytage was born at Ticknall, Derbyshire, England in 1795, and was educated at schools in Yorkshire. He was the son of George Armytage, who died in Australia in 1853, having emigrated at the age of eighty-seven. Armytage junior subsequently studied engineering in London until his twentieth year, when, on 28 February 1815, he sailed for Australia in the Hebe.
In the following year he landed in Van Diemen's Land, where he was allotted a few acres of land at Bagdad, which were increased to 500 acres in 1817. Armytage received sizable grants of land at Bagdad, and later obtained larger areas in Western Victoria as a pastoral squatter. As a result of much of this land being forcibly taken from Aboriginals by the Armytages and their colonial property managers, they were involved in several instances of frontier conflict. For example, when taking ownership of an area near Geelong in 1836, Charles Franks was speared to death by Wathaurung. A subsequent punitive expedition against local Aboriginals apparently served as "a warning to the natives not in the future to commit wanton excesses" against the British occupiers. The advantages of obtaining vast areas of land virtually for free were soon realised and the Armytages became exceedingly wealthy, owning famous mansions such as The Hermitage in Geelong, and Como House in Toorak. They expanded their enterprise into properties in New South Wales and Queensland, and also into the lucrative frozen meat industry. In 1818 he married Miss Elizabeth Peters. In 1826 he received a further grant of, and built upon it the first watermill in Tasmania. During this period Armytage was part of the Bagdad division of the Tasmanian police who were heavily involved in the Black War that resulted in the deaths of a possible 900 Indigenous Australians.
In 1835 Port Phillip District commenced to attract settlers; and in 1836 his eldest son Thomas visited the district, and camped on the Werribee River. During 1836 an investigation into the murder of local indigenous people took place in regards to a Charles Franks who had been employed by George Armytage to secure selection of desired parcels of land. Franks was found to have been using lead as a poison. Franks stated that he was using lead to poison "Crows" which at the time was a slang term for Indigenous Australians and possibly a reference to Jim Crow. In 1847 Mr. Armytage proceeded to Victoria, and settled upon his son George Armytage's station at Ingleby, where his eldest son had died of typhus fever on 12 September 1842. In 1851 he settled at Geelong, and built "The Hermitage".
Death
Armytage died of erysipelas in 1862, his widow surviving him until 1873.