The former Auxerre, Marseille and Liverpool man scored the opening goal of the game after 11 minutes to get off the mark for his new club.
The 32-year-old, who returned to France last month after an unhappy spell at Russian club Kuban Krasnodar, fired home from the edge of the penalty box only for Israel striker Eden Ben Basat to level things up for Toulouse on 25 minutes.
But the Corsican side found themselves back in front before the break when Francois-Joseph Modesto found the back of the net on the stroke of half-time.
And after Toulouse midfielder Clement Chantome was shown a red card with 17 minutes remaining, Bastia sealed all three points through substitute forward Gianni Bruno's penalty in injury time.
The win sees Bastia move up to 10th in the Ligue 1 table, while Toulouse remain 12th.