All the characters are making fun of Prospero because he is looking at things in a mystic way and doesn't care about the technical details and physics. Changing that would change the story so far that it would be about something completely else.
i'm not sure how familiar you are with fantasy but this is actually dealt with quite often. the two camps are usually represented by magic users (for physics) and clerics (for religion, well, clerics). in this case Prospero would probably be best represented as a bard tied to religion in a society so magic saturated that almost nobody worships the old deities.
The same is true with the main reason for the Dive. This is fundamentally a story about SCIENCE. They are doing this for SCIENCE to understand the physics of the world in a deeper level and if you change that it wouldn't be the same story.
this right here is an example of why almost all of the is/isn't sf debates happen. in terms of the story, as a plot device, the science isn't essential. it can be tough to see, especially cases like this where so much page space is given over to scientific explanation. the plot doesn't really get going until the fourth section. before that all we really have is the introduction of a couple characters and a lot of exposition about technologies and the physics of a black hole.
from a purely narrative point of view the black hole is nothing more than a source of motivation for the characters. if you haven't looked into narrative structures (
character archetypes,
macguffins, etc.) this can be counter intuitive but when someone says that any sf story can be translated into another genre this what they mean. replacing the black hole with, say, an astral plane from which no one can return results in little to no change in the character's actions.
and this is where the debates are coming from. when some people say 'story' they mean the characters' actions and the events that occur, other people's definition includes the technical exposition and physics lessons the author has put in.
what you've said about the theme of the story changing in a fantasy setting is a good point but i think it would still be workable. weakened probably, but workable. the magic users are interested in exploring and understanding the world (and bending it to their will), the bard takes the traditional view of interpreting everything according to doctrine.
Also the special type of immortality that they have as intelligent software and the fact that they are cloning themselves for the Dive makes it additionally hard to change into fantasy imho.
there are common fantasy devices that simulate these. astral projection (sort of a freed soul) and mirror spells for example.
