I came to SF through means other than literature. Films such as the Star Wars trilogy, Blade Runner (I watched this at far too young an age) etc, TV (yes, Doctor Who played a big part), comics (for example, 2000 AD), RPGs. I read from the library when I was wee, and SF formed the bulk of my selections- probably because it looked similar to my non-reading interests. I don't remember much of it, but I did have a taste for Doctor who novelisations when i was young. And Asimov's robot anthologies.
What made me an avid reader of Sf novels was Ian M Banks Culture books. These mark the transition between picking up a couple of interesting library books and actually seeking out good-quality SF. Why Banks? He's the top Scottish genre author, so his books were prominently displayed in the SF section of the big bookshop in town.
As for fantasy, my dad persuaded me to read The Hobbit, and LOTR. They were the epitome of fantasy in my mind, until I discovered the Earthsea books. And Discworld. I also read Narnia- because I loved the BBC TV series (laughable special effects and all). I wonder if many people are first discovering SF or fantasy through print these days. Perhaps getting into print SF through films, TV etc is the norm now.
And, no doubt, there are those whose first SF story was delivered to them via a podcatcher.