There are several phrases that drive me batty. Usually it's because they're overused or, in many cases redundant.
For example, the phrase "the fact of the matter" is one I hear regularly. I'm guessing that because most of the people I hear saying this phrase are lawyers, there is probably a time (in court?) when using this phrase is appropriate. Unfortunately, it's everywhere it's not supposed to be. If I said to you, "The fact of the matter is the recipe calls for two eggs, so you should put in two eggs unless you're doubling the recipe, in which case you put in four eggs," it's pretty clear that what I'm saying is a fact. I don't need you to preface your factual statement in order to understand that it's a fact. This phrase is especially annoying, however, when it's used to preface an opinion. Grr. I would explain further, but that would take us into politics (since that is another area I hear this phrase used regularly), and I try to avoid politics like the plague- or at least like the formerly-known-as-Swine Flu.
Along similar lines of redundancy are the phrases "Truthfully," "The God's honest truth," and "Honestly." If you need to throw these phrases in front of what you're saying, does that not imply that everything else you've said has been lies?
I am sure we all have phrases that we use (or misuse) that drive people batty. I'm equally sure we all have pet peeve phrases. What's your pet peeve phrase?