In 1896, a church was built in honor of St. Prince Alexander Nevsky at the military cemetery. The church was small, with an area of only 112 sq.m, not including the altar and narthex. Funeral services and requiems were held here for military and civilian workers of the railway brigade, which included three battalions.

From April 1921 to October 1931, this church, like the later Protection Church, remained the only Orthodox church in the city. This was due to a fire that destroyed the parish church of the Smolensk Icon of the Mother of God, which was located where the modern Holy Protection Church stands.

After World War II, due to the development of the city center, it was proposed to move the church outside the city, to the site where Telman Street is now located. However, the parishioners refused, as at that time it was a wasteland. As a result, in 1949, the church and the rectory were dismantled. The community of St. Alexander Nevsky merged with the Protection community, and from the materials obtained after the dismantling, rectory houses for the Holy Protection parish were built. Currently, a taxi stand is located where the former cemetery church stood in Lenin Square.

Interestingly, a church in honor of St. Alexander Nevsky did appear on Telman Street, and near the site of the old church, the Church of the Holy Myrrh-Bearing Women now stands.