You're probably right - this is after all entirely a personal reaction on my part. One of the problems for me is the Master popping up once or twice a year. After he's succeeded in killing the Doctor in Logopolis, how do you stop further appearances from feeling anti-climactic?
To be fair, the 5 Doctors actually has something interesting for him to do - be sent to rescue the Doctor and react badly when the 3rd Doctor tells him to sod off. But stories like Timeflight and Mark of the Rani feel very much "The Master's doing this because he's the Master and it's what he does."