You're not homophobic if you don't like gay people.
No, you're just a bigoted a**hole.
To be fair, if one believes that homosexuality is a learned behavior rather than an innate tendency shaped by a variety of inputs, and if one believes in a religion which has specific or general strictures against homosexual behavior, it is possible to truly believe that those who have gay sex are doing something wrong, something they should be fighting to resist, just like sex outside of whatever cultural structure you believe in might be equally wrong. In that case, it is quite possible to view gay people as those indulging a vice and thus not people you want to associate with. I think this view is mistaken, but it is certainly possible to dislike "gayness" without being a bigot per se.
Comparisons to racism break down if one side doesn't believe that homosexuality is an actual expression of human sexuality, but is instead a behavior, a
choice. Society often (and quite rightly) disapproves of individual choices when they are harmful to people (even to those who practice them; suicide is often outlawed and regularly sees intervention, for instance; to the eyes of the sort of person I'm describing, homosexuality seems an aberrant psychological problem akin to the severe depression that drives someone to injure themselves, and equally worthy both of disapproval and laws meant to restrain it.)
Really, one ought to talk more about racist/sexist/whateverist behaviors rather than people. It's possible to do something offensive without meaning it or even realizing it, or even when trying to HELP. Consider, for instance, the rather sad situation of ex-gay ministry, which tries to "cure" its patients of being gay. Many members are formerly-practicing homosexuals who truly believe they've found a better, purer, and cleaner way to live. While I would classify their actions as inappropriate and intrusive (just like I'd disapprove of a group that tried to, say, "cure" monogamy in happily-married individuals), I have a hard time calling them "bigoted assholes."