Limestone is notoriously soft and wears easily. For ex, the Pyramids at Giza were covered in Limestone. While some of it was "reused", most of it deteriorated. It's not a long lasting substance.