Maguire conceded with enough points available to win the eighth frame after missing a red as he fell 5-3 behind. Runs of 56 and 131 saw him level at 5-5, but Dott did enough to scramble over the line in the final frame just before midnight to secure a match against Judd Trump or Barry Hawkins in the last eight on Friday.
"I thought all the way through the match: 'there is no way I'm winning this," said the 2006 world champion Dott. "I've lost a few 6-5s here, and I thought it was going to happen to me again.
"I'm ecstatic to be through. I will need to play better than tonight to win the tournament, but I have a chance if I keep feeling the way I'm feeling. My game is a lot better."
Dott won the first frame in 33 minutes aided by a solid enough 61 after Maguire had spread the reds with a poor safety shot.
Larkhall's Dott then ran in a 69 to move two frames clear with his opponent continuing to falter at key moments.
Maguire - the 2004 UK champion - finally clicked into gear in the third frame as a knock of 110 saw him close to 2-1 behind.
Both men had opportunities in the fourth frame, but Maguire was suddenly appearing the more composed of the two players as a closing 41 helped him level at 2-2.
A missed attempt at a longish red by a faltering Dott saw Maguire compile 65 for a third successive frame and a 3-2 lead.
But Dott was quick to halt the bleeding as he produced a dominant performance in the sixth frame to level at 3-3 before an excellent 92 break saw him move 4-3 clear.
The frustration of the evening was getting to Maguire as he conceded the next frame after missing an easy red with Dott leading 51-15.
There was enough points available for Maguire to mount a recovery as the red mist descended.
He will be fined £250 by World Snooker for conceding with enough points available to win the frame.
"You can normally tell with Stephen when he starts to bubble," added Dott. "I wasn't sure that he had conceded the frame as people are getting fined now for doing it. But it didn't really affect him afterwards as he played like a machine. It is just the way he is."
Maguire steadied himself as a 56 saw him claw back to within one frame at 5-4 behind before an immaculate run of 131 saw him level at 5-5 after holing a lovely red to get the break started.
Both men had chances in the deciding frame, but it was Dott who held himself together as Maguire left himself needing snookers when he unfortunately saw a pink drop in trying to play safe.
Dott holed a double on the green moments later to ensure his progress.
"I can't think of a more horrible way to lose a match because the shot Stephen has played is a great shot. And he has got me in all sorts of trouble if the pink doesn't go in," said Dott.
"Obviously, you are glad to win, but you would rather win a different way. We are good friends, and it is a horrible way to lose."