During the reign of Catherine II, the lands around the city of Pochep, as well as the city itself, were part of the vast estates of the former Little Russian hetman Kirill Razumovsky and his son Alexei. A small manor house, referred to as the "hunting castle," served as a place of rest for Count Razumovsky during hunting trips. The exact circumstances of its construction and the name of the architect remain unknown.

Alexei Razumovsky left the Krasnoryg volost to his children, born out of wedlock and bearing the surname Perovsky, Lev and Vasily. In 1825, their brother Alexei, a well-known writer who published under the pseudonym "Antoniy Pogorelskiy," settled in the estates of Krasny Rog and Pogoreltsy. After separating from her husband, his sister, Countess Anna Tolstaya, came here to raise her young son. After her brother's death in 1837, she became the rightful owner of Krasny Rog.

Anna Alexeyevna's son, Alexei Konstantinovich Tolstoy, continued the literary traditions of his uncle, who wrote the fantastic tale "The Double, or My Evenings in Little Russia" here. In 1841, he published the story "Upyr" under the pseudonym Krasnorygskiy, describing vampires that settled in a noble estate.

Due to complicated relations with his mother, Tolstoy settled not far from St. Petersburg in an estate called "Pustynka." After his mother's death and his retirement in 1861, he moved to Krasny Rog, where he worked on poems and dramas on historical themes.

In 1875, Alexei Konstantinovich Tolstoy, suffering from severe headaches, died in Krasny Rog from a morphine overdose and was buried in the local church. His widow, Sofia Andreyevna, returned to the capital, and two years later, in June, the philosopher V.S. Solovyov visited the empty estate.

Only the wing and the household building of the estate have survived. Other buildings, such as the gardener's house and the watchman's lodge, were erected by the heirs after Tolstoy's death. The "hunting castle" itself burned down on the night of January 1, 1942.

The Literary-Memorial Museum of A.K. Tolstoy was opened on September 3, 1967, in the surviving wing of the estate, and in 1972, a bust of the poet was installed nearby. The museum's exhibition includes personal belongings of Tolstoy, household items from his time, copies of portraits of his acquaintances, and antique books. Since its opening, the museum has held the regional poetry day "Silver Lyre" in Krasny Rog on the last Sunday of August every year.

The project to restore the main house of the estate was developed by Bryansk architect V.N. Gorodkov in the 1960s but was only implemented from 1988 to 1993 with the support of writer N.M. Gribachev.

In 2014, the Ministry of Culture allocated 40 million rubles for the recreation of the original interior of the museum-estate. Some items from the 19th century will be purchased from private collections, and the most expensive acquisition may be a wardrobe costing 2 million rubles. Director Elena Lovyago noted that the exact furnishings of the writer's house are unknown, so the museum is oriented towards a typical appearance of a noble estate of that time.

In 2017, all the furniture from the estate will be sent for restoration in anticipation of the anniversary of A.K. Tolstoy.