Anyway James - to answer your question - I would have to say Colin Baker, but many of the problems weren't his fault.
For a start, they initially made him unlikeable. His first adventure - The Twin Dilemma - saw him attack his assistant as his regeneration didn't work properly. His erratic behaviour was the reason why his costume was so hideous - but they never sought to change it.
His tenure as the Doctor was beset with issues off screen - there were complaints that some of the stories were too violent, and the controller of the BBC at the time didn't want to see re-hashed Dalek/Cybermen/Master stories so insisted new enemies were written. Colin Baker's first series saw the old characters but I'd argue that Attack of the Cybermen is the worst Cyberman story and Revelation of the Daleks is the worst Dalek story!
So they wrote new characters like Sabalom Glitz who were frankly shit (probably why you've never heard of him!). As a result, audience figures plummeted and the show was axed for the first time in it's 25 year history.
They brought it back for Trial of a Time Lord which had it's moments but wasn't that great, and after that Colin Baker was sacked.
Colin Baker wasn't exactly a bad actor, but a shit costume, shit storylines and an annoying habit of shouting a lot when he really didn't need to means he's my least favourite.