The teams shared four goals in a remarkable opening quarter of an hour as Borja Valero put Fiorentina ahead in the fifth minute, Romulo levelled within a minute, Juan Iturbe gave the visitors the lead and Valero equalised again.
Peru midfielder Juan Vargas gave Fiorentina a halftime lead with an angled shot and Rossi scored what turned out to be the decisive goal with a 54th-minute penalty awarded for handball by Bosko Jankovic, who was sent off.
It was Rossi's 12th goal of a season which has seen him make an impressive comeback after a knee injury which sidelined him for the best part of two years.
Despite playing with a man less, Verona pulled one back in the 72nd minute when Jorginho scored from a rebound.
Fiorentina are equal fourth with Inter Milan on 27 points while promoted Verona are sixth with 22.