Jockey Joseph O'Brien had the 2-5 favourite towards the back of the pack on the final run to the post.
He attacked from deep but was just unable to close the gap to Godolphin's Encke, ridden by Mickael Barzelona and a 25-1 shot, who took the win by three-parts-of-a-length.
Michelangelo (10-1) was third three lengths back from Camelot, who was looking to become the first horse to win the St Leger, 2000 Guineas and Derby in the same year since Nijinsky 42 years ago.
Indeed Nijinsky, like Camelot housed at Ballydoyle, was the last horse to attempt it - and the previous occasion it was achieved was by Bahram in 1935.
Only 15 horses have completed the treble.
Aidan O'Brien was also disappointed in his bid to become the first man to train all five British Classic winners in a single season: Camelot's two previous victories added to Homecoming Queen's triumph in the 1000 Guineas and the Onvestec Oaks won by Was.
"In the Guineas he quickened and in the Derby he quickened, but he just stayed on here," he said.
"He ran a great race but just got beat. It's disappointing for everybody but that's the way it is. That's racing.
"We expected him to win and if I thought they were going to go that steady I would have had a pacemaker in or two pacemakers. But that's my fault."