No, it's not unfair. It depends on how innocuous the subject matter is. If someone were to tell me they liked Harry Potter I would beg to differ and may even think about how many other things we may not have in common. I would not, however, dislike them. If someone were to tell me they thought the Holocaust was a great thing though, I would dislike them.
Opinions represent core values. They are a perfectly valid reason for disliking people. In fact, they are the main reason we choose to dislike some individuals.