By decree of June 13, 1839, the construction of a cemetery church in honor of the Holy Apostles Peter and Paul was permitted. The Sumy merchant Fyodor Sapozhkov initially donated 500 thousand rubles for the construction of the church. In May 1843, the composition of the trustees for the construction of the Peter and Paul Church was determined, which included merchants Fyodor Sapozhkov, Ivan Kurnosov, Andrey Kopylov, and Fyodor Stozhkov. The construction was overseen by Archpriest Peter Maretsky. The church was consecrated in October 1851 by Archbishop Filaret Gumilevsky of Kharkiv and Akhtyrka. There was a chapel in the church dedicated to St. John the Baptist, located in the building of the local almshouse, built by I. G. Kharitonenkо.

Nearby are the graves of the Kharitonenkо family members. Beautiful marble sculptures, created by French sculptor Aristide Croisy, adorn these burials. Croisy's works can also be seen in the Paris City Hall, the gardens of the Palais-Royal, and the Louvre. They were brought by P. I. Kharitonenkо in 1900 from Paris after the World Exposition. One of the sculptures is placed on the grave of his daughter Zinaida, who died in childhood, and another on the grave of his father, Ivan Gerasimovich. Here lies Pavel Ivanovich Kharitonenkо himself.

On the altar side of the church is the grave of another well-known benefactor of the city, merchant Dmitry Ivanovich Sukhanov, who was a trustee and long-time elder of the Spaso-Preobrazhensky Cathedral and did much for its improvement. It is believed that the monument on his grave was created by the famous sculptor P. Antokolsky. Nearby are the graves of the first ruling archbishops of the Sumy diocese, established in 1945, Ilarion Prokhorovna and Yevstratiy Podolsky.