According to one of the modern versions, the first stone tower of the new kremlin, laid in 1500, was the Tverskaya. This version is preferred because the tower protected an important area of the city with a trading settlement and docks. There is also a legend about the siege of the city in 1505 by the united forces of the Kazan Khan Muhammad-Emin and his brother-in-law from the Nogai Horde. According to tradition, the siege was lifted after the Nogai murza was killed by a shot from the Ivanovskaya Tower, fired by a captured Lithuanian cannonner.

Of all the surviving towers of the kremlin, the Ivanovskaya is in the worst condition due to constant water exposure flowing from the Clock Mountain and along the roofless walls. Therefore, its walls were reinforced from the very beginning with buttresses on wooden piles. Over time, only a pile of stone and brick rubble remained behind the buttresses. The tower suffered from fires, and in 1531, there was an explosion of gunpowder inside, leading to significant reconstruction. After the explosion, the outer walls above the circular gallery were not restored, and battlements appeared on the northeast wall of the gallery, formed from the piers between the combat windows. The other sides were adorned with small decorative battlements. The side rooms of the tower were used as storage.

The southern part of the extension next to the tower served as a prison. In the 17th century, there was a provincial house here where interrogations and investigations were conducted. After the kremlin lost its military significance by 1697, repairs and reconstructions continued. The upper extension was carelessly rebuilt after the fire of 1715 and took on the appearance of an ordinary one-story house with windows, a low roof, and a chimney. In the same century, due to significant destruction, the tower stood empty. In 1786-1788, it was repaired and fully adapted for a prison.

Until 1887, the Ivanovskaya Tower housed the Ivanovo police station and fire department. In 1815, breakthrough gates with a flat ceiling appeared in the wall next to the tower, similar to the outer gates of the tower. In 1819, the last fire occurred in the tower, after which the battlements of the surrounding gallery began to deteriorate. The house on the tower became two stories, accessed by a staircase in a wooden extension on the southwestern side of the tower, decorated with wooden battlements. Other temporary wooden structures were built around the tower. During repairs, internal ceilings were created in new places from vaults on rails and I-beams. From 1887 to 1926, the upper extension of the tower housed the archive of the Nizhny Novgorod Scientific Archive Commission with a reading room. In 1934, the first tier of the tower was adapted for a chemical warehouse, installing a concrete ceiling. The tower was also used as a fuel storage. Despite repairs in the 18th, 19th centuries, and the 20s of the 20th century, the tower remained in an emergency state. In 1946, the masonry of the inner facade and the vault of the passage began to collapse.

During the restoration of the tower as part of the Kremlin's restoration under the guidance of architect S.L. Agafonov in the 1950s, a version was chosen to recreate the presumed original appearance of the tower. Although the option with an open bypass gallery after the explosion of 1531 was also considered. The project was successfully implemented, but some minor details were never completed. The wooden ceilings between the tiers, the shutters of the loopholes, and the barriers of the large combat windows between the battlements, as well as the double-leaf gates, were not restored. The mechanism of only one drawbridge remained non-functional. And most importantly, the mechanism of the drawbridge and the bridge itself were not recreated, which meant that the tower never became passable. Access to the room behind the always lowered grilles was closed to tourists. The extension with the city staircase also could not be accessed for a long time. Nevertheless, the tower acquired a quite medieval appearance, only disrupted by the massive commemorative sign on the facade to the left of the gates.