Miller slammed in a 76th-minute winner, just three minutes after his introduction at London Road.
Boro had earlier seen Faris Haroun's early brace wiped out by Peterborough's on-loan striker Dwight Gayle, who struck twice in an impressive display.
"There's no doubt in my mind we deserved to win it but we have conceded three very poor goals - one from our own corner, one when we have failed to pick up a man at the back post and the third from a goalkeeping mistake."
Haroun's first came in the ninth minute when he capped off a fine counter-attack to put away Emmanuel Ledesma's pass and 11 minutes later he converted Stuart Parnaby's cross at the back post.
But Posh fought back through on-loan Gayle with a rasping effort before the break and shortly into the second half by converting Michael Bostwick's header, before Miller's strike.
