Dan Biggar fired a hat-trick of penalties at Stade Pierre Antoine but it was not enough as Rory Kockett and Geoffrey Palis booted the French champions to victory.
The Welsh region battled to snatch a victory at the death as Castres saw Palis and Seremaia Bai sent to the sin-bin but the hosts clung on.
The defeat leaves the Ospreys all but mathematically out of Europe at the group stages for the fourth season in a row, following earlier losses to Leinster and Northampton.
Backs coach Rees said: "Effectively we are out of the competition. But it was a huge effort by all the boys, from Alun Wyn Jones to young Daniel Suter - they all gave it everything.
"We have had a very difficult week with only one full training day but the boys applied themselves and gave everything they could for the shirt.
"There were opportunities at the end to win but it didn't happen and we have to look forward to the next game."
The victory sees Castres maintain the pressure on Leinster and Northampton in Pool One.
They drew level on eight points with the Dubliners ahead of Saturday's clash at Franklin's Gardens but would have been targeting a crucial bonus point which they failed to secure.
Castres lock Richie Gray admitted the Welsh region frustrated Castres but he was delighted to secure the win to kick-start their European campaign.
He added: "The Ospreys made life difficult for us at the breakdown and defended very well. But we expected that, they are a very competitive side.
"We were down to 13 men at the end but we managed to hold out and I think that shows how hard this team works for each other.
"We did so well to hold out in that period. It was a real team effort.
"The win puts us in a good position in the group. Now we have a very difficult trip to the Ospreys next week but it is one we feel we can do well in."