Taking CON damage or CHA damage doesn't effect the current fight. So you don't get the feedback loop until you are forced to pick strength or dex. See?