In 1796, the architect of Count Alexei Grigorievich Orlov created an 'English garden' and built a small pavilion on the edge of a high hill above the river, which later became known as the Summer House. One of its facades completed the upper part of the park, while from the balcony of another, there was a view of the southern expanses of Moscow, from the Kremlin to the Sparrow Hills. In front of the house, the park sharply descended, where a pond was dug and the 'Bathing' pavilion, the romantic 'Mysterious' grotto, and other park attractions were built. Not all of them have survived. The most famous was the 'Air Theater,' created in 1830 and lasting until 1834. It was a covered gallery in the shape of a semicircle, where the stage was arranged so that the scenery was replaced by trees and bushes.

The Summer House was wooden and built in a classical style, with a simple symmetrical plan. Each of the large facades was adorned with six Corinthian columns, while the small facades featured one-story gazebos with lattice shutters.

The Englishwoman Catherine Wilmot, who visited the garden in 1807, noted: 'It is cut by countless paths, hills, lawns, and cliffs; the variety of gazebos and bathing houses delights the eye. Birch bark is used in the park architecture, which looks wonderful, as if the summer houses and gazebos were created by nature itself.'

Since 1831, the Neskuchny Garden passed into the ownership of Nicholas I, although initially only partially. The Neskuchnoye estate was acquired by the Palace Department from the Shakhovskys in November 1826, while the Golitsyn estate could only be purchased by the tsar in 1842. N. P. Golitsyna, known as 'The Queen of Spades,' refused to sell the plot and stated in her will that it could only be sold five years after her death. Only then were all three parts united into the present Neskuchny Garden. During the imperial family's stay in Moscow, their residence became the neighboring Alexandrinsky Palace, while the Summer House was used for outdoor tea parties by the imperial family until 1917.

After the revolution, the House was used in various ways, with the longest occupancy being as a creative house for the Union of Soviet Writers. Before the transformation of Gorky Park, there was a library-reading room there. Now, a photo studio is located in this place.