OK. I have developed a Theory (which is mine) about Shaggy Dogs.
The Theory (which is mine) Part I:
A shaggy dog rests on a punchline. A punchline is like a twist ending.
A twist ending is successful when, and only when, it evolves from the framework of the piece. It must feel, in the immortal words of someone-er-other, "inevitable yet surprising."
With a punchline, we have the surprising down, oh yes we do! But how to acheive the inevitable?
The Theory (which is mine) Part I(B):
Tonal consistency: This has been brought up several times. If we're reading a story that appears to be moving in one direction, and then it takes a hard reverse, this is almost (though not quite) impossible to pull off. This is particularly true if the story has been successful at engaging a particular emotion from the reader, e.g. sadness. If the reader is truly invested in the plight of poor little Rudy, and then poor little Rudy is sacrificed at the alter of a pun, the reader is likely to shake her fist in condemnation! Poor little Rudy! How could you throw him under the bus like that?
Framework: This is what I think we have not discussed yet. Any story operates within a frame of readerly expectation. Tonal consistency is part of, but not all of, that frame. A horror story, for instance, operates within the framework of a horror story. The reader will be upset if it suddenly transforms into a mystery. Likewise, a story that begins with detailed character development will feel amiss if it ends up short-changing the characters -- even if it maintains a consistent tone. Nancy Kress discusses this in _Beginnings, Middles, and Ends_. The beginning of the story makes promises to the reader. Those promises must be delivered. Leading to:
The Theory (which is mine) Part II -- PUNS as Punchlines:
A pun is a metafictional device.
99% of stories operate via the conceit that the reader should, rather than paying attention to the language qua language, pay attention to the images, characters, and actions which are created by the words. A pun ending -- unlike any other kind of twist -- takes those images, characters, and actions and brings them back to the level of letters formed into words. The illusion of story is shattered.
Therefore, a pun is almost always a break in the frame of the story. This allows it to be surprising, but shatters inevitability. This, I believe, is the source of the problem: a shaggy dog pun feels like a cheat because it tricks the reader into reading the story as a story, only to break the frame and deliver a metafictional ending which shatters the illusion.
The Theory (which is mine) Part II(B):
Verbally rendered jokes, however, rely on these very devices.
While a groan-worthy pun in a story makes readers angry, in a pun-based joke, the pain of the groan is the highest acheivement possible.
What's the difference? Why do verbal jokes function when most shaggy dog stories fail?
My answer: Jokes are based on different conceits and reader expectations. The verbal cue of the joke allows the reader to understand the joke as a metafictional frame. We do not listen to jokes in the way we read stories, with suspension of disbelief. It therefore does not feel like a cheat when the suspension of disbelief is undercut for a pun.
The Theory (which is mine) Part III:
Therefore, in order to keep promises to your reader (a la Kress) and to maintain a sense that the story comes to an inevitable yet surprising conclusion, one must find a way to bring the reader into a metafictional frame.
I have brainstormed several ways of doing this. Doubtless there are more.
A frame within a frame - If a character tells a shaggy dog story in order to accomplish a goal, the reader is cued in much the same way the listener to a joke is cued. The reader may feel free to listen to the joke and be amused with the joke-teller character.
An observant character - A character with a wry sense of humor can observe a pun that sounds out of place in narration. This works best if the character understands the pun (as opposed to the shaggy dog example I placed in the "Humphrey" thread) or if another character understands the pun. Then, within the context of the story, the pun is considered to be a pun. This maintains the frame of suspension of disbelief rather than breaking it.
A narrator with a personality - If the narrator of a story is a character in and of hirself (rather than the objective, transparent or disembodied narrator which is common in contemporary third person fiction), then that narrator can deliver the pun in the same way a character can deliver the pun. It is within the frame, rather than without.
An early presented metafictional cue - It should be possible to cue the reader of a story that the story is looking at language as language, rather than purely a tool for relaying information. The Xanth books, for instance, are constantly calling attention to words as words via puns. A shaggy dog punchline (which Anthony uses many of) is therefore consistent with the readers frame. Other metafictional tools I can think of which might lay this kind of groundwork include: a title which makes it clear the story is a shaggy dog, or placing the story in a joke section, things like that.
I think all these things could work -- if properly rendered -- to allow the reader entrance into the frame of the shaggy dog story, so that the reader will not feel cheated.
Appendix to The Theory (which is mine) -- SUBVERTING reader expectations:
Whenever a story is begun, certain contracts with the reader are assumed. Narrative conventions dictate certain ways in which a story must flow.
Some writers, for currently mysterious reasons, are bloody fabulous at subverting reader expectations without coming across as cheats. Via careful plotting, Connie Willis is able to acheive this in several powerful novels.
Given the fact that subversion is possible, it must be true that a shaggy dog story can theoretically be created without cueing the reader into the meta-framework. However, I've yet to see it successfully done.
Appendix to The Theory (which is mine) Part B:
For those who are curious and/or annoyed, the (which is mine) gag is a reference to a Monty Python routine.