The building, erected in 1848, belonged to the famous Taganrog landlord Nikolai Dmitrievich Alferaki. The project was developed by architect Andrei Ivanovich Shtakenshneider, a professor at the St. Petersburg Academy of Arts. In its time, Alferaki Palace was the center of cultural life in Taganrog, hosting grand balls for the elite.

Next to the palace, where Nikolai Dmitrievich Alferaki lived, there was a wing of the corridor system. His sons Nikolai, Achilles, Sergey, and Mikhail lived there along with teachers and governesses.

In the late 1870s, the entire block, including the palace, passed to merchant Negropont, who sold the property in parts. The palace with the garden was acquired by a merchant society for the "Commercial Assembly" club, where concerts were held. One of these featured Professor L. S. Auer from the St. Petersburg Conservatory and composer Taneyev, attended by A. P. Chekhov. The club was also visited by P. I. Tchaikovsky during his visit to Taganrog. The atmosphere of the palace and its club, as well as walks in the garden, were reflected in Chekhov's stories "Ionich", "Mask", and "My Life".

By 1840, N. D. Alferaki owned 2,000 desyatins of land and 125 souls of peasants near Taganrog; he owned 8 stone houses and 25 stone shops at the Taganrog market. Due to work commitments, he often traveled while his family lived in Kharkov. In 1845 and 1848, Alferaki worked at the Kharkov provincial post office and simultaneously managed the affairs of the Kharkov theater, effectively financing it.