The answer, like every actual historical answer, is "it's complicated." When you read a dozen credible books about events in Rome during the 2nd and 3rd century, when it supposedly "declined" (even though calling something a "decline" is dubious just because it stopped expanding), then come back.