The win pushed the Jets (3-2) into a tie for second place in the AFC East while Atlanta, top seeds last year in the NFC after posting a 13-3 record, dropped to a dismal 1-4 for the season.
"It took every one of us," said Jets head coach Rex Ryan.
"The biggest thing I'm proud of is that we didn't turn the football over. We were not reckless, we were just the opposite, we were fearless. And that's how we have to play."
The Falcons took full advantage of a Jets penalty after a failed fourth down attempt near the end zone, quarterback Matt Ryan's three-yard pass to receiver Levine Toilolo giving the hosts a 28-27 lead with just under two minutes remaining.
However, Jets rookie quarterback Geno Smith drove New York from their own 20-yard line to the Atlanta 25 to set up the game-winning field goal.
Smith, who has led the Jets to an NFL-leading three game-winning drives for New York's three wins, completed four successive passes and scrambled for an eight-yard gain that set up Folk's game-winning kick.
The young signal caller had committed 11 turnovers in New York's first four games, including two interceptions and a pair of lost fumbles in last week's lopsided loss to Tennessee.
"Everybody has complete confidence in me," Smith said about taking the field for the last drive. "They told me to go out there and let it rip."
Eliminating turnovers was a key focus on offence, he added.
"I made it my duty not to put the ball on the ground," Smith said. "It came down to a last-second drive against a tough Atlanta team on the road. But we stood up tonight and that speaks volumes about the team's character."
Smith completed 16-of-20 passes for 199 yards with three touchdowns and no interceptions in an error-free performance.
He hit tight end Jeff Cumberland on a 20-yard touchdown strike, connected with Jeremy Kerley for a 10-yard score and threw a one-yard touchdown to Kellen Winslow.
Said Jets coach Ryan: "I'm proud of Geno, proud of everybody."
Falcons quarterback Ryan hit 36-of-45 throws for 319 yards and a pair of touchdowns, connecting with Jason Snelling for a four-yard score before his short scoring pass to Toilolo.
Running back Jacquizz Rodgers scampered 19 yards for a touchdown in Atlanta's fourth-quarter comeback.
The Jets defence made a big contribution with a rousing goal line stand at the end of the first half, snuffing the Falcons' rushing attempt from the one yard on their fifth try from the shadow of the goal line after a penalty gave Atlanta an additional crack at it.
New York went into intermission with a 17-7 lead.
"Felt like we could score those points, but we didn't get it done," said Atlanta coach Mike Smith. "In hindsight it wasn't probably the (right) decision the way the game ended up.