The celibacy was probably because he thought Jesus would return in his lifetime (an idea probably shared by most Christians based on the “this generation will not pass” saying in the synoptic gospels). But Paul was still fine with marriage saying that Jesus’ brothers were married and still preached.