The three buildings of Vasily Panyushov's income house were located on two plots between Arbat and Sivtsev VRAZHK, which historically belonged to one owner. In the mid-19th century, these properties belonged to actual state councilor N.M. Vasilychikov, who sold all the buildings and two gardens, one facing Arbat and the other facing Sivtsev VRAZHK, to the wife of a sub-lieutenant, Sofia Evlampievna Sukhodolskaya. As a result, the plot passed to merchant V. Panyushov.

In 1912, three income house buildings were constructed here according to the design of architect Vasily Kazakov, which were popularly nicknamed the "ant hill." This nickname referred more to the courtyard buildings of house 51 (buildings 2 and 3), which were not visible from Arbat. A. Rybakov wrote in his "Novel-Memoir": "the buildings stand closely one behind the other, low arched passages connect two deep dark courtyards, covered by squares of gray Moscow sky."

After the revolution, the house underwent changes: after 1917, workers and military personnel were settled in the spacious sunny apartments of the first building, displacing the previous owners. The luxurious apartments turned into communal ones. The house finally became an "ant hill."

The third building faced Sivtsev VRAZHK and was located deep in the park that had been preserved from the previous owners. A. Rybakov, who lived in the second building in apartment 87, wrote that the courtyard buildings had identical "small apartments, cramped and dark." They were not even crowded, as there was nowhere to go.

The building always remained residential.