"Yeah, David Cameron actually offered me a cabinet job, but the commute wasn't convenient for me"