Kris Commons drove the champions into the lead in the third minute but Saints striker Gregory Tade levelled in the 18th minute before the visitors were denied what looked a clear penalty when Steven Anderson challenged Commons inside the box.
But Celtic were well off the pace all afternoon and in the 80th minute they paid dearly when former QPR striker Vine curled the ball in from the edge of the penalty area to hand the Perth club their first win of the league campaign.
It was not the result or performance Neil Lennon was looking for ahead of their Champions League opener against Benfica at Parkhead on Wednesday night.
However, the result will delight St Johnstone manager Steve Lomas who had seven players down with a virus on Thursday, which led him to cancel training.
A Michael Higdon double, meanwhile, helped Motherwell come from behind to defeat Dundee 2-1 and remain top of the Premier League.
Colin Nish had opened his Dundee account with a fine header after 20 minutes, but Higdon equalised from close range in the 74th minute before heading home the winner 10 minutes later to deny the Dens Park side a first home win of the season.
The result preserves The Steelmen's unbeaten start to the season but was harsh on a Dundee side that had defended bravely for the majority of the match.
Leigh Griffiths was on target as Hibernian stretched their unbeaten league run to five matches with a 2-1 victory over Kilmarnock.
An inswinging Griffiths corner ricocheted in off Kilmarnock goalkeeper Kyle Letheren and defender Rory McKeown as Hibs took a 14th-minute lead before Danny Racchi tucked in an equaliser after 32 minutes.
Griffiths converted a 45th-minute penalty which proved decisive as Hibs claimed their second win at Easter Road this season, just one behind their tally of home SPL wins in the whole of the last campaign.
Aberdeen substitute Cammy Smith's late header denied Inverness their first win of the season in a 1-1 draw which saw both sides finish with 10 men.
The 17-year-old's first senior goal with six minutes to go cancelled out Richie Foran's penalty, which gave the hosts the lead mid-way through the second half.
Inverness's Owain Tudur Jones was dismissed in the 53rd minute and Aberdeen captain Russell Anderson was given his marching orders 13 minutes later, both for dangerous challenges.
And St Mirren registered their first home Premier League victory of the season as they defeated Hearts 2-0.
Jim Goodwin fired a superb 35-yard strike beyond Jamie McDonald seven minutes before half-time to give the hosts the lead.
And Lewis Guy doubled their advantage three minutes after the interval after a lightening fast counter-attack ended with Steven Thompson cutting the ball back for the striker to blast into the net.