Aggression
Despite their formidable intelligence and playful tendencies, dolphins are large, powerful and sometimes very aggressive animals. The bottlenose dolphin (the archetypal "Flipper," clown of the sea) has been observed roughly killing porpoises, a smaller relative, for unknown reasons. Give any dolphin its due and don't treat it like a toy or a pet.
The Wild Ocean
In a wild setting, swimming with dolphins carries a more fundamental risk: the vagaries of the ocean. Rip tides, sneaker waves, storms are all realities of any maritime activity and ultimately pose perhaps the greatest danger in swimming with dolphins. Stay in groups and be aware of your own limitations. While sharks have been unfairly vilified, it's true they pose some risk to humans swimming with wild dolphins; large species like great white and tiger sharks actively hunt dolphins and porpoises, and smaller sharks are apt to be drawn to the same food sources attracting dolphin pods.
Dolphin Health
Feeding and swimming with wild dolphins is illegal in the United States under the 1972 Marine Mammal Protection Act. The long-term effects of this kind of contact on wild cetaceans aren't entirely clear, but harassing or stressing any animal is never a good idea---and habituating them to humans generally isn't, either. Always make sure you're dealing with a reputable company that respects both the law and the animals. Consider chartered whale- and dolphin-watching tours, which restrict interaction to on-board viewing.