The construction of the governor's house in the Nizhny Novgorod Kremlin is linked to significant urban planning changes initiated in Nizhny Novgorod by the decree of Emperor Nicholas I. These changes concerned the territory of the Kremlin, where a new ensemble of the city's administrative and spiritual center was planned, as well as the placement of an imperial residence for the emperor's visits. The initial project for the governor's house was developed in early 1835 by P.D. Gotman, a well-known builder and supervisor of works in Nizhny Novgorod. However, Nicholas I rejected the project due to "excesses in buildings and decorations" and ordered a reduction in construction costs. The final project in the style of late classicism was approved on November 21, 1836, and developed by architect I.I. Sharleman. Construction began in 1837 and continued for several years. Initially, the supervision of the works was carried out by architect A.L. Leer, who was later replaced by assistant architect A.I. Sheffer and city architect A.E. Turmyshev. During the construction process, minor changes were made to the project. The main facades of the house were oriented: the northern one facing the Volga, and the southern one towards the Spaso-Preobrazhensky Cathedral. On the first floor were the living quarters of the governor's family, on the second, the imperial apartments, a living room, a dining room, an office, bedrooms, and other rooms. The third floor housed the home church of the Holy Spirit, consecrated on September 18, 1845, where Bishop John of Nizhny Novgorod and Arzamas held the first liturgy.

On the sides of the building were two wings: the eastern one housed the gubernatorial office and apartments for officials, while the western one contained the garrison guardhouse. A wrought iron fence on a stone base was installed between the house and the wing. All construction work was completed in the summer of 1841.

Five years later, due to landslides on the slope, the building, located above the cliff of the Kremlin hill, was damaged and needed repairs. In 1846, engineer A.I. Delvig, known for constructing the first Nizhny Novgorod water supply and fountain, applied concrete for the first time in the city's history to seal cracks in the foundations and basements.

After the construction of the governor's palace, the project by architects I.E. Efimov and P.D. Gotman from 1834 was implemented to create the Kremlin garden on the site of the former Holy Spirit Monastery. In 1852, a winter garden with southern trees, a grotto, and a fountain was laid out on the western side of the house. The governor's garden descended down the slope from the house to the northern side of the Kremlin, from the Georgievskaya Tower to the Ivanov Gates. The stone church of the Holy Spirit Monastery was transferred to the Old Believer community of Nizhny Novgorod, and in 1840 it became the house church of the governor's house. In 1858, the house hosted a meeting between French writer Alexandre Dumas père and Decembrist Ivan Annenkov and his wife Polina Göbel, the prototypes for his novel "The Fencing Master." In the second half of the 19th century, the building underwent several repairs, particularly in 1885, 1894, and 1896. At that time, a metal canopy over the main entrance on cast-iron pillars and oval ramps for the driveways were installed according to the project of architect N.I. Ivanov. By spring 1906, cracks appeared in the walls of the building, threatening its integrity. The question of demolition or partial reconstruction was considered, but the ministry decided to carry out a major repair with foundation reinforcement. Junior engineer E.M. Michurin was appointed responsible for the repairs. After the 1917 revolution, the house was renamed the Palace of Freedom. From October 1917, the church in the house ceased to operate, and in 1918 it was liquidated, with all its historical and artistic items disappearing, and presumably the bell tower was demolished at that time. Later, new gubernatorial authorities were housed in the building. Since 1992, the former governor's house has housed an exhibition of the Nizhny Novgorod State Art Museum.