Lolley, who snubbed a move to Posh last week, blasted a 30-yard screamer past keeper Bobby Olejnik to secure a famous replay victory with 14 minutes to go.
Posh had hit the front in the 26th minute when Tommy Rowe converted from point-blank range after a Danny Kearns cross hit an upright but Harriers transformed the tie with a double-salvo early in the second half.
Michael Gash's powerful header provided an exquisite 48th minute Danny Jackman cross with the finish it deserved before Jack Byrne fired the Conference Premier side ahead after a surging run from Lolley four minutes later.
Posh man Mark Little struck a post in between the two Harriers goals but the hosts had to wait until the 74th minute to level when top scorer Britt Assombalonga earned and converted a penalty.
But Lolley came up with the perfect response to send the non-league men through in what his likely to be his last game for the club.
Substitute Joe Garner smashed in a second-half hat-trick to fire Preston to a 3-2 victory over Ipswich and earn a trip to Nottingham Forest in the fourth round.
After a drab first half, Championship side Ipswich took the lead when Frank Nouble capitalised on a defensive error just before the hour mark.
That was enough for the hosts to throw on Garner who scored a quick-fire double to put the League One side in front.
And while Ipswich got back level through their own substitute, David McGoldrick, there was no stopping Garner who won it in the 88th minute to dump the Tractor Boys out.
Jennison Myrie-Williams slammed in a 75th-minute winner as Port Vale came from behind to claim a 3-2 win at Plymouth.
Vale goalkeeper Chris Neal saved Reuben Reid's 80th-minute penalty following Chris Lines' foul on home skipper Conor Hourihane, for which Lines was dismissed, but the visitors held on.
Substitute Nick Powell struck twice to send FA Cup holders Wigan through as they came from behind to defeat MK Dons 3-1 after extra-time.
Skipper Tommy Elphick's near-post finish sent Bournemouth on their way to an emphatic 4-1 victory over League Two Burton which teed up an fourth round clash with his boyhood club Liverpool.
Sheffield Wednesday survived a scare as they finished strongly to see off non-league Macclesfield at Hillsborough, winning 4-1.
Captain Paul Robinson scored his first goal for five years as Birmingham booked a tie against Swansea with a 3-0 win over Bristol Rovers.
Watford will travel to either Manchester City or Blackburn in the fourth round after ensuring there was no upset against Bristol City at Vicarage Road.
Yann Kermorgant's late strike ensured Charlton's FA Cup hopes did not sink in the Valley mud as they drew 2-2 with Oxford in the rescheduled third-round tie.
FA Cup Third Round Results
Charlton 2-2 Oxford
FA Cup Third Round Replay Results
Birmingham 3-0 Bristol Rovers
Bournemouth 4-1 Burton
Fulham 3-0 Norwich
MK Dons 1-3 Wigan (After Extra Time)
Peterborough 2-3 Kidderminster
Plymouth 2-3 Port Vale
Preston 3-2 Ipswich
Sheffield Wednesday 4-1 Macclesfield
Watford 2-0 Bristol City