Cavendish, who has made noises about wanting to leave the team, is one of the big names on the Tour along with Tour de France winner Bradley Wiggins, but was caught in a crash a kilometre from the end of the 200km stage, a loop between Ipswich and Norwich.
Cavendish suffered a cut to the left shoulder, but managed to finish the stage.
Rowe sealed a win for Team Sky regardless, besting Boy van Poppel and Russell Downing to claim the stage.
An early breakaway had seen four riders, including Briton Kristian House of Rapha Condor as well as Rony Martias, Niels Wytinck and Jonny Clarke build a lead worth three-and-a-half minutes by the halfway stage.
The peloton, which had Team Sky and Garmin at the helm for most of the day, slowly reeled them in, making a bunch sprint inevitable with 15km to go.
Cavendish had stayed out of trouble with several crashes taking place in the final stages, and looked a strong favourite to sprint to victory, only to go over and take several riders with him.
The eight-stage Tour of Britain continues tomorrow with a stage from Nottingham to Knowsley.