The Briton, who is 14 points behind Formula One championship-leading team mate Nico Rosberg after 10 races, had been fastest in all three practice sessions and was favourite for pole position.
Red Bull's quadruple world champion Sebastian Vettel qualified second, equalling his best grid placing of the season, in an all-German front row.
2008 world champion Hamilton, who has won for the past two years in Hungary and four times in all, faces another fight through the field in Sunday's race at a slow circuit where overtaking is difficult.
"Guys, I'm on fire," Hamilton, who has had a run of bad luck this season with two race retirements and qualifying problems as well as five wins, said over the car radio before coasting to a halt.
"OK, Lewis, just stop the car where there is a fire marshal," his race engineer replied.
"I can't stop it," said the Briton, who finally did so at the pitlane entry and scrambled out as marshals extinguished the blaze. He then walked away, helmet still on and head down. Hamilton slapped the barrier lightly in frustration as he went.
"There's a lot going through my mind, but I just have to try to turn it into positives until tomorrow," Hamilton told the BBC. "I think it's getting to the point beyond bad luck - it's something else. We just need to do better."
Mercedes said a fuel leak, the source of which was unclear, had led to the fire in the engine. The 29-year-old had to start last weekend's German Grand Prix in 20th place after he crashed in the first phase of qualifying due to a front brake disc failure.
He ended up finishing third. Hamilton said the team would have to change the engine and gearbox, bringing with it a mandatory five-place penalty. That will mean he is sure to start in the pitlane.
"I honestly don't know what I can do tomorrow," he told Sky Sports television. "This is a track that you cannot overtake on so I think I will struggle to get in the top 10 tomorrow or at least the top five. I will probably leave here more than 20 points behind Nico, but there are still races to go.
"I don't know what to say, I will try my best."
Rain at the start of Q3 made the world championship leader's task difficult.
Rosberg was the first onto the track on slick tyres with conditions worsening, but with the first corner particularly wet he slid off into the runoff area on his first flier, ruining his lap.
But Kevin Magnussen, who was second on the road, also outbraked himself, hitting the tyre barrier on the outside of the corner and bringing out the red flag before anyone could set a time.
When qualifying resumed after an eight-minute delay, the rain had eased and while the track was slippery, everyone opted to continue on slicks.
With the track drying up, everyone went back out on slicks, with Rosberg going fastest on his second flying lap after a brief spell at the top of the times for Bottas.
But unlike Rosberg, who stayed out and used only one set of softs in Q3, Vettel pitted for new rubber and took top spot from the Mercedes driver on his final lap.
Moments later Rosberg improved, taking pole position by almost half-a-second.
Vettel held onto second ahead of Bottas, the latter also opting to pit for new tyres, with Ricciardo fourth fastest.
Fernando Alonso and Felipe Massa were fifth and sixth, both also having two runs in the restarted Q3 segment, with Jenson Button eighth ahead of Jean-Eric Vergne.
Force India driver Nico Hulkenberg was the slowest of those to set a time in Q3 in ninth, with Magnussen classified 10th having not set a time before his crash.
Daniil Kvyat was in contention for a top 10 slot, but he locked the rears at Turn 12 on his final lap in Q2 and spun, ending up 11th.
Sauber driver Adrian Sutil was 12th ahead of Sergio Perez, who suffered a hydraulic leak during Q2 and could only attempt one run.
Esteban Gutierrez took 14th ahead of Romain Grosjean, with Jules Bianchi ending up 16th after an excellent performance to make the second stage of qualifying.
Kimi Raikkonen was a shock casuality during Q1, ending the session 17th fastest.
With Hamilton and Pastor Maldonado, whose Lotus had stopped early in Q1, not running, most teams opted not to send their cars out for a second run on the basis that they only needed to finish ahead of the Caterhams and the Marussias to make the next phase.
But Raikkonen was the slowest of those who did not attempt a run on soft rubber and when Bianchi put in his final lap, the Finn was relegated to the dropzone while sat in the garage.
Caterham driver Kamui Kobayashi was 18th ahead of Max Chilton, who suffered a fuel pressure problem on his final run, with Marcus Ericsson slowest of those who set a lap.
Hamilton was classified 21st ahead of Maldonado thanks to starting a flying lap.
- Motor Racing
- Sports & Recreation
- Nico Rosberg
- Sebastian Vettel
- Valtteri Bottas
- Daniel Ricciardo