There is no reason whatsoever for his character to be buff, he was a medical Doctor Who became a sorcerer, none of those require any serious amount of physical strength the way playing Thor or Cap does