They're both "things that aren't so" (although some sci-fi is "things that aren't so now but could be possible in future.")
Ummm... at the risk of hijacking the thread, that got me to thinking (yeah, a dangerous thing, I know).
Fantasy has it's roots in a belief system that was deemed to describe how things
actually were. People really believed (or still believe) in elves, ghosts, and such, even if they seemed extraordinary compared to, say, cows or trees. Heck, I know a fairly acculturated Inuit guy who told me in all seriousness about the
Ishigaq or "little people" that he and his sister saw when they were kids.
Many Science Fiction premises are plausible because in our own lifetimes (okay, at least for some of the more senior of us) we see things around us that were once in the realm of SF, even if in a slightly different form. (eg. Computers got much more powerful, but smaller and cheaper, and not bigger and more expensive.)
Either way, maybe the "things that aren't so" quality is less important than "it
might be so".