Maybe by the time your N is back up to the size of your regression sample. It still seems like a stretch to apply population statistics about variance to pairwise differences like this.