The Wimbledon champion started out well as he took the opening set and led 3-0 in the second, but struggled thereafter against the world number 40 as he lost 3-6 6-4 6-2.
World number four Murray struggled on serve throughout, however, getting less than half his first serves in during the deciding set - while the 30-year-old Mayer was almost untouchable, losing just three points on his own serve in the closing set.
The manner of the defeat was the big worry for Murray's fans: in what was only his ATP Tour second match since back surgery last September, the way his game crumbled prompted fears that he may not yet be fully recovered.
Any such worries appeared to have been allayed by his dominant win in the opening round - a 6-0 6-0 victory over Mousa Zayed - but the defeat against the wily German veteran spells major trouble for Murray's hopes at the Australian Open, where he has made the final in three of the last four years.
The Scot will head to Melbourne - where the first Grand Slam of the year begins on January 13 - having played just two ATP Tour matches, in addition to two exhibition matches.
"To be honest, in my mind, the match was finished at 3-6, 0-3," Mayer told the ATP website. "[But] I tried to play more aggressively, tried to risk something more. Of course for sure he didn't play his best tennis.
"I knew before the match if I would have a chance, it's probably today. He didn't play for a long time. [So] I took my chance."
Second seed David Ferrer was beaten though as he was upset 6-4 7-5 by Daniel Brands.
Murray and Ferrer were not the only seeds to fall in Qatar either: eighth seed Fernando Verdasco also went out, losing 4-6 7-6(0) 6-2 against Romania's Victor Hanescu while sixth seed Philipp Kohlschreiber lost 7-6(4) 7-6(7) to Peter Gojowczyk in an all-German affair.
Results from the Qatar Open Men's Singles Round 2 matches on Wednesday
Daniel Brands (Germany) beat 2-David Ferrer (Spain) 6-4 7-5
1-Rafa Nadal (Spain) beat Tobias Kamke (Germany) 6-3 6-7(3) 6-3
7-Ernests Gulbis (Latvia) beat Lukasz Kubot (Poland) 6-2 4-6 6-3
Dustin Brown (Germany) beat Ivo Karlovic (Croatia) 3-6 7-6(13) 6-4
Florian Mayer (Germany) beat 3-Andy Murray (Britain) 3-6 6-4 6-2
Peter Gojowczyk (Germany) beat 6-Philipp Kohlschreiber (Germany) 7-6(4) 7-6(7)
Results from the Chennai Open Men's Singles Round 2 matches on Wednesday
7-Edouard Roger-Vasselin (France) beat Jiri Vesely (Czech Republic) 6-4 6-4
Dudi Sela (Israel) beat 2-Mikhail Youzhny (Russia) 3-1
Results from the Brisbane International Men's Singles Round 2 matches on Wednesday
1-Roger Federer (Switzerland) beat Jarkko Nieminen (Finland) 6-4 6-2
2-Kei Nishikori (Japan) beat Matthew Ebden (Australia) 6-2 6-4
Marin Cilic (Croatia) beat 5-Grigor Dimitrov (Bulgaria) 7-5 7-5
Marinko Matosevic (Australia) beat Sam Querrey (U.S.) 5-7 7-6(3) 6-4