In April 1864, shortly before the end of the Caucasian War, Colonel Ivan Diomidovich Popko of the Kuban Cossack Army was appointed commander of the newly formed Psekupskiy Regiment in the Klyuchevaya stanitsa. Despite numerous duties, he found time to establish a museum of mountaineers. In July 1864, Popko sent a special letter to the heads of the stanitsas, emphasizing the importance of preserving the traces left by the Abadzehs and Bzhedugs in the regiment's area.

Popko noted that although these monuments were modest, they represented valuable material for studying the culture of the people. Exhibits for the museum began arriving at the headquarters of the Psekupskiy Regiment in Klyuchevaya (part of Goryachy Klyuch since 1965), and soon the need for their storage arose.

A solution was found: a mountain saklya was installed in the regimental yard, where the museum was located. The exhibits included items from craftsmen of that time, as well as artifacts from the era of the ancient Greeks and Genoese, household items, and agricultural tools.

In 1867, during a visit by Grand Duke Mikhail, the viceroy of the Tsar in the Caucasus, he was presented with the museum's exhibits, divided into three sections: paleontological, mineralogical, and ethnographic.

This was the first museum in the Caucasus. Even in Tiflis, a similar institution opened only in January 1867, despite the presence of the Russian viceroyalty in the Caucasus there. In Yekaterinodar, the museum appeared only in 1879.

Today, the city historical museum of Goryachy Klyuch considers itself the successor of the first museum in the Caucasus, founded in 1864. It is located on the central street of the resort, in a building opened in May 2000. The exhibits are presented in the halls of archaeology and paleontology, Kuban life, military history, and an exhibition hall.