South Africa, who lost their opening match to Sri Lanka, struggled early on and needed a blistering unbeaten 86 off 43 balls by Duminy to reach 170 for six.
New Zealand seemed to be cruising in their reply thanks to Kane Williamson's 35-ball 51 and Ross Taylor's 62.
They needed 31 off 24 deliveries with seven wickets in hand but a brilliant two-over spell from Steyn (4-17) turned the match on its head.
The fiery paceman dismissed the dangerous Corey Anderson for seven in the 18th over and then picked up two for four in the last over with the Kiwis, who beat England in their first match, requiring seven for victory.
Taylor, needing three off the last ball, was run out.
"I was thinking all the time, Dale is my trump card and I need to bring him back in the game because he was bowling beautifully," South Africa captain Faf du Plessis, who missed the Sri Lanka match with injury, said at the presentation.
"But with the way they were hitting the ball... it was a fantastic effort from Dale.
"We were in huge trouble at the start. JP took his time and then hit it sweetly at the end. Fantastic effort and one of the best innings I have seen him play."
Leg-spinner Imran Tahir also played a key role, taking two for 27, but paceman Morne Morkel had a day to forget as he went for 50 in three overs.
"Credit to South Africa for the way they hung in there... We probably should have won it pretty easily," New Zealand captain Brendon McCullum said.
"But you can't probably take it to the last over against such a quality bowler like Dale Steyn.
"Little bit of luck today and we would have got across the line but credit to South Africa for their performance."
Earlier, Hashim Amla (41) and Duminy fought back with a 55-run stand after South Africa had slumped to 42 for three.
The fourth-wicket partnership was broken in bizarre fashion when Amla thumped a ball from Anderson and it hit Duminy's bat at the non-striker's end before spooning up for the bowler to complete an easy catch.
Duminy was full of improvised strokes, bludgeoning 10 fours and three sixes as his side added 70 runs in the last five overs.
- Sports & Recreation
- South Africa
- Corey Anderson
- Dale Steyn