Yeah that’s my main concern with the non-military victories - you need to still have a significant military presence to keep them off you while you try to pursue what you’re going for, and it’s a lot easier to cripple another civ by taking a few key cities than economic/diplomatic pressure