During the Sengoku period, the Sasaki family was appointed constabulary authority over Ōmi and other nearby provinces by the Ashikaga shogunate in Kyoto. A castle was needed as an administrative center and military rallying point. The ridged peak of Kinugasa Mountain, where Kannonsho-ji was already, was selected as the new site. A castle from that lofty position would command a view of the plains below and would serve as a symbol of power in the region. Construction was completed in 1468; it was originally named Sasaki Castle. Sasaki Castle was intended to defend Ōmi Province and was the base-castle of the Rokkaku clan. It was attacked and taken not long after construction during the Ōnin War. Renovations were done during the Kōji Era, and the stone ramparts were enlarged to accommodate small cannons. In 1568, Oda Nobunaga was escorting Ashikaga Yoshiaki to Kyoto to install him as shōgun. As the expedition crossed through the area, Rokkaku forces, loyal to the Ashikaga shōgun, left their stronghold to disrupt or arrest their advance. Nobunaga forces led by Niwa Nagahide defeated the Rokkaku on the battlefield and entered Kannonji Castle unopposed and in triumph, before resuming his march to Kyoto. The Rokkaku clan continued to use Kannonji Castle as a base castle until their worst defeat in 1570 by Shibata Katsuie. It is thought that the castle was abandoned sometime afterward. In 1582 Kannonji Castle was attacked and burned, along with Azuchi Castle, possibly at the hands of looters from among the local peasantry. In 1969 and 1970, artifacts were discovered during archaeological excavations in the main courtyard.
If hiking, the trail head that leads up the west side of Kinugasa Mountain is about a 25-minute walk from Azuchi Station, or about 10 minutes from the nearby Azuchi Castle Archaeology Museum and the Nobunaga Hall. The trail leads up the west side of the mountain, first to the ancient temple Kuwanomi-dera, then through a forest of Japanese cedars, and up to Kannonji Castle site. From the main courtyard, there is a trail leading on to Kannonsho-ji, a temple with an impressive panoramic view. If driving, there are two roads that lead up the mountain. One follows a spur on the southwest slope and is accessed near the Nakasendō Highway in Azuchi. The other leads up the eastern slope, and can be found near Route 202 off the Nakasendō Highway in the neighboring town of Gokashō. Both roads lead to Kannonsho-ji, a few minutes' walk from Kannonji Castle ruins.
