After nine years' service, his wife's health failing, Brown returned to the United States and became a pastor at Sand Beach Church and teacher of boys at Owasco Outlet, near Auburn. He worked for the formation of a college for women, which was situated first in Auburn and then in Elmira, New York and now known as Elmira College. Brown was responsible for sponsoring Yung Wing ; the first Chinese student to graduate from a U.S. university, graduating from Yale College in 1854.
Japan
When by the Harris Treaty of 1858, Kanagawa and Nagasaki in Japan were opened to trade and residence, Brown sailed for the former, arriving on November 3, 1859. On arrival, Brown shared residential accommodation with the family of the Presbyterian medical missionary Dr. James Curtis Hepburn, then residing at Jobutsuji in Kanagawa, a dilapidated temple formerly occupied by the Dutch consulate. Brown and Hepburn, both benefiting from the experience of living and working in China, were noted pioneers in the study of the Japanese language. In collaboration with Dr. Hepburn and others, Brown made substantial contributions to the translation of the New Testament into Japanese. Brown was also a gifted teacher, Ernest Satow, then a student interpreter at the British legation, who many years later became British Consul to Japan, described the Japanese language lessons received from Brown to be, "of the greatest value." Brown began presiding at Christian ecumenical religious services held at the Jobutsuji in Kanagawa from the second Sunday after his arrival in November 1859. In July 1860, at the request of English-speaking merchants in Yokohama, Brown begun to preach regularly at Sunday morning service that attracted 30 to 40 congregants each week. In 1861 Brown also contributed to drawing up the plans and specifications for the British Anglican Garrison Church built on Lot 105 in the foreign settlement. The Garrison Church, also known as Christ Church, was the forerunner of Christ Church, Yokohama, rebuilt in 1901 on a prominent position on the Bluff overlooking the Port of Yokohama. On Lot 167 in the heart of the Kannaicommercial district, Brown was also able to establish a Reformed Church, later named in 1872 as Union Church, Yokohama. At Yokohama, Brown also opened a school in which hundreds of young men, afterwards leaders in various walks of life, were educated. Brown acted as honorary chaplain to the United States legation, teaching and preaching for over 20 years. He was one of the founders of the Asiatic Society of Japan and a prominent contributor to early Meiji Period higher education. Following a fire that destroyed much of his home, personal library, manuscripts, and notes, Brown returned to the United States for a two-year furlough in May 1867. In June of the same year he was awarded an honorary Doctorate in Divinity by New York University. Brown returned to Japan in 1869, arriving at Yokohama on August 26, to take up a new position as principal of a government funded school in Niigata. The Niigata sojourn was only brief; desiring to be close to his fellow New Testament translators, Brown accepted a new teaching post and relocated back to Yokohama in 1870. Brown, suffering from ill health, left Japan for the United States in the Autumn of 1879.
