I've put a lot of thought into this, too, as my stories, for the most part, can't decide which camp they'd like to be in. I adore literary fiction (I'm an English teacher), but I recognize that much of it is pretty stuffy and defined by an ever-changing set of rules established by people I've never heard of and don't care to know (and never will--I live in the South). A lot of the time, these rules seem to require that a novel or story has no satisfying ending; judging by the pieces I've read recently in The New Yorker, this is a trend that is, sadly, gaining steam instead of dying quietly and with dignity under the house like it's supposed to.
Much of literary style is characterized by an author's willingness to "experiment," although often this experimentation seems less novel and more incestuous, since literary authors tend to "experiment" in exactly the same ways. Usually, there is a focus on character over plot and an irritating propensity to use words learned in grad school that may or may not be the best choice, but are always the most esoteric. Done well, this can be beautiful and intriguing. Done poorly (and it often is), it's insufferable.
The great thing about speculative fiction is that, for the most part, nobody's following a trend. Sci-fi, fantasy, and horror editors are looking for great writing and great stories, and they're not paying attention to rules set by two or three insular publications. There's fantastic writing in genre fiction, but it's fantastic writing because, well, it's fantastic. It's not (usually) pretentious, and it's not a chore to read.
To me, it's like New Orleans jazz vs. experimental jazz. Louie Armstrong didn't torture his audience, but he definitely tortured his art form: he messed with it, did things that other people didn't, took chances. Experimental jazz is technically good, I guess, but it's hell to listen to. And all of it pretty much sounds the same.
Like I said, I do like literary fiction, but I have my limits. I'm a huge fan of Donna Tartt's--I loved The Secret History and The Goldfinch--but The Little Friend, her second novel...well, I almost burned it in the yard. (At the risk of spoiling it for you, it's a mystery--and it's never solved. That's not "the point," apparently. Still angry, and it's been months.)
That said, there have been several literary sci-fi and horror works I've truly enjoyed. Joyce Carol Oates's writing, I believe, falls into that category, and it's excellent. And one of the best books I've read in the past several years is Gold Fame Citrus, by Claire Vaye Watkins, the current darling of the literary scene. (For good reason; she's fantastic.) Gold Fame Citrus is post-apocalyptic, firmly in the spec fic camp, but its rambling, often philosophical narrative keeps it within the realm of literary fiction, too.