O'Brien (19) got on the end of a long ball from Jean-Yves M'voto and then lobbed goalkeeper Joe Murphy to put Barnsley in front.
Barnsley's Tomasz Cywka then twice went close to putting the home side further ahead, testing Murphy with a 30-yard free-kick and hit the keeper's left-hand post with a curling effort.
Moussa equalised in the 78th minute after bursting through the middle before beating Luke Steele to set up a stirring finish.
Clarke struck the winner just a minute from time after holding off a couple of defenders.
Mellon said: "We were disappointed with the second half.
"We felt that we lost a lot of the individual battles that you need to win. We lost too many of them, certainly in the second half.
"We value the competition highly as a football club but we never showed the fire in the second half to progress in the tournament and that was disappointing.
"We didn't put the chances away and we were poor in the second half in the important moments.
"We'll need t get going again. They know that they let themselves down.
"As professional people, we'll get going again, there's no doubt about that because that's what we do.
"There were a lot of things said in the dressing room that needed to be said and we will get a reaction."
- Sports & Recreation
- Franck Moussa
- Joe Murphy
- Leon Clarke