In my mind, a sport must have 3 qualities:
1. Athletic ability (exercise, not just practice, must improve play)
2. Personal competition (you're playing against another person, not the world record, chance, or a computer)
3. Quantitative scoring (a score must be apparent with very little judgment or personal taste involved)
An activity which fulfills only . . .
#1 is a athletic experience (e.g. hiking, yoga, sex)
#2 is a contest (e.g. Iron Chef, art competition, politics)
#3 is a test (e.g. stock market, most single-player computer games, solitaire)
#1 and #2 is a athletic competition (eg gymnastics, diving, competitive dancing)
#1 and #3 is a athletic performance (eg breaking a world record by yourself, climbing a mountain first)
#2 and #3 is a game (eg chess, darts, poker)
Some activities are borderline. For example, golf only sometimes meets requirements #1 and #2. You could play golf by yourself driving a cart, or you could compete with others and walk the 18 holes. Foil fencing does not always meet requirement #3, since judges have to determine who has "right of way" which can sometimes be a judgment call. Auto racing is also borderline, since I've heard some claim that it requires athletic ability since there isn't power steering, but I see that as dubious.