The first building of the monastery was the "small cell", where its founder Cyprian lived. Later, a wooden house was built for the abbot in this location, situated to the north of the churches of Vladimir and Prepolovenie. In 1764, due to its dilapidation, the house was demolished by Archimandrite Jacob.

By the blessing of Archbishop Alexander of Veliky Ustyug, on August 10, 1695, a small single-domed church in honor of All Saints was built next to the abbot's house, along with a one-story stone building that served as a hospital. The church suffered from fires multiple times and was rebuilt in 1720.

In 1734-1735, under Archimandrite Pachomius, the building was reconstructed, adding a second floor. The lower floor of the Church of All Saints housed the hospital, while on the opposite side, there was one cell and a vestibule. The second floor contained six cells and a vestibule.

In 1763, a wooden terrace was built from the abbot's house to the northern porch of the Archangel Cathedral. In 1892, by the order of Bishop Varsonofy, the terrace was converted into a warm corridor with an iron roof. It existed until 1918, when it was demolished due to dilapidation.

In 1771, Archimandrite Misail added two cells on the western side. After a fire in May 1772, the city governor Prince Ivan Alexeyevich Yengalychev moved into the abbot's house with his family, adding a wooden balcony with two staircases on the southern side.

In 1786, the Church of All Saints was expanded, converted into a house church, and connected with the cells. Choirs appeared on the western side, and the iconostasis was painted on canvas.

In 1841, under Archimandrite Augustine, the cells on the upper floor were thoroughly renovated, and a stone balcony with four columns and a pediment was built to replace the old wooden porch, which housed the staircases inside.

In 1888 and 1889, the abbot's house was renovated and adapted for the residence of the Veliky Ustyug archbishop. Two rooms were added to the lower floor, the floors were replaced, and new tiled stoves were installed. The roof and outer walls were painted. At that time, the All Saints Church was converted into a living space for the monastery's brotherhood.

From 1941 to 1945, the building was used by the Pukhovich Military School. Since 1945, it has served as an auxiliary administrative space for the automotive technical school.