Minus Six Nations players such as Jocelino Suta, Mathieu Bastareaud and Frederic Michalak, Toulon were punished by a Castres team eager to take full advantage.
The home side were ahead 19-9 at the interval after winger Romain Martial went over in the 14th minute and South African scrum-half Rory Kockott kicked 14 points.
Toulon's cause was not helped by yellow cards shown to both prop Andrew Sheridan and hooker Sebastien Bruno before the break.
They did hit back in the second half, Australian centre Matt Giteau scoring a try in the 68th minute, but their only consolation came in the form of a defensive bonus point thanks largely to 15 points from the boot of Jonny Wilkinson.
Victory for Castres sees them move onto 54 points in fourth place ahead of the rest of the weekend's action.
