Fantasy may have the most rabid and obsessive fans, but it also has the staunchest detractors of any mainstream genre. We all know people who simply refuse to watch fantasy films or read fantasy books of their own volition. They may have sat through The Fellowship of the Ring grudgingly, but didn't bother with the rest of the series. They probably associate the genre with asocial nerds, fan conventions, and Dungeons & Dragons. They can only shrug at the exuberance of the devotees. Fantasy is "not their thing."
Why are fantasy movies (and the genre in general) so polarizing? I've long thought it has something to do with viewers' relative affinity for cinematic worlds. Some people go to the movies to see something that directly relates to their own lives, something that takes place in the universe they live in and know. Others – myself among them, if you haven't figured it out – flip for new, self-contained worlds that could exist independently of the movie; wonderful and strange places we feel like it's possible to actually inhabit. This might explain why those who like good fantasy also tend to enjoy good science-fiction.