Anecdotal stories claim that in 1858 Samuel Brannan paid $1,500 for lumber salvaged from a ship that foundered on the rocky shore's basalt cliffs near Seal Rocks and built the first Cliff House. While Brannan may have constructed a building there, no historical evidence of this building exists and its role in the origin of the Cliff House remains apocryphal. According to the National Park Conservancy, the Cliff House was built by Senator John Buckley and C. C. Butler, opened in 1863 and leased to Captain Junius G. Foster. It was a long trek on foot from the city and the restaurant hosted mostly horseback riders, small game hunters or picnickers on day outings. With the opening of the privately built Point Lobos toll road a year later, the Cliff House became a Sunday destination among the Carriage trade. Later the builders of the toll road constructed a two-mile speedway adjacent to it where well-to-do San Franciscans raced their horses along the way. On weekends, there was little room at the Cliff House hitching racks for tethering the horses for the thousands of rigs. Soon, omnibus, railways and streetcar lines made it to near Lone Mountain where passengers transferred to stagecoach lines to the beach. The growth of Golden Gate Park attracted beach travellers, in search of meals and a look at the sea lions sunning themselves on Seal Rocks just off the cliffs, to visit the area. In 1877, the toll road, now Geary Street, was purchased by the city for around $25,000. In 1883, after a few years of downturn, the Cliff House was bought by Adolph Sutro, who had made a fortune in silver by solving the problems of ventilating and draining the mines of Nevada's Comstock Lode. After a few years of quiet management by J.M. Wilkens, the Cliff House was severely damaged by a dynamite explosion when the schooner, Parallel, ran aground on January 16, 1887. The blast was heard a hundred miles away and demolished the entire north wing of the tavern. The building was repaired, but was later completely destroyed by fire on Christmas night 1894 due to a defective flue. Wilkens was unable to save the guest register, which included the signatures of three Presidents and dozens of illustrious world-famous visitors. This incarnation of the Cliff House, with its various extensions, had lasted for 31 years.
1896–1907
In 1896, Adolph Sutro rebuilt the Cliff House from the ground as a seven-story Victorian Chateau, called by some "the Gingerbread Palace", below his estate on the bluffs of Sutro Heights. This was the same year work began on the famous Sutro Baths in a small cove immediately north of the Restaurant. The baths included six of the largest indoor swimming pools, a museum, a skating rink and other pleasure grounds. Great throngs of San Franciscans arrived on steam trains, bicycles, carts and horse wagons on Sunday excursions. Sutro purchased some of the collection of stuffed animals, artwork, and historic items from Woodward's Gardens to display at both the Cliff House and Sutro Baths. The 1896 Cliff House survived the 1906 earthquake with little damage, but burned to the ground on the evening of September 7, 1907.
1909–1937
After the fire, Dr. Emma Merritt, Sutro's daughter, commissioned Reid & Reid to rebuild the restaurant in a neo-classical style. It was completed within two years and is the basis of the structure seen today. In 1914, the guidebook Bohemian San Francsco described it as " one of the great Bohemian restaurants of San Francisco.... while you have thought you had good breakfasts before this, you know that now you are having the best of them all."
1937–2003
In 1937, George and Leo Whitney purchased the Cliff House, to complement their Playland-at-the-Beach attraction nearby, and extensively remodelled it into an American roadhouse. From 1955 until 1961, a sky tram operated across the Sutro Baths basin, taking up to 25 visitors at a time from Point Lobos, enhanced by an artificial waterfall, to the outer balcony of the Cliff House. In the 1960s, upon the closing of Playland, the Musée Mécanique, a museum of 20th-century penny arcade games, was moved into the basement of the Cliff House. In the early 1970s the land-side exterior of the building was decorated with an expansive mural painting depicting crashing waves, painted by artist-musicians Michael Cotten and Prairie Prince. The building was acquired by the National Park Service in 1977 and became part of the Golden Gate National Recreation Area.
2003–present
In 2003, as part of an extensive renovation, many of Whitney's additions were removed and the building was restored to its 1909 appearance. A new two-story wing was constructed overlooking what were by then the ruins of the Sutro Baths.. During the site restoration, the Musée Mécanique was moved to Fisherman's Wharf.
Today
When the Cliff House became part of the Golden Gate National Recreation Area in 1977, the National Park Service contracted with Dan and Mary Hountalas as official concessionaires of the property. The NPS renewed its contract with the Hountalas family, under the name Peanut Wagon, in 1998. Peanut Wagon continues to manage Cliff House operations and worked with the Park Service during the extensive site restoration that was completed in 2004. The Cliff House features two restaurants, the casual dining Bistro Restaurant and the more formal Sutro's. Additionally, the Terrace Room serves a Sunday Brunch buffet. There is a gift shop in the building, and the historic camera obscura is on a deck overlooking the ocean. During the 2013 government shutdown, the US Park Service ordered the restaurant closed. The owners defied the order, but were forced to close again. They reopened with permission on October 12, 2013.
The area immediately around Cliff House is part of the setting of Jack London's novel The Scarlet Plague. Jack London also sets the meeting of Maud Sangster and Pat Glendon Jr. here in The Abysmal Brute. An image of the third Cliff House was used on the cover of the album Imaginos by Blue Öyster Cult. The Cliff House is featured in the Ubisoft video game, Watch Dogs 2.