On a night when Ravenhill paid tribute to Nevin Spence, who died a year ago in a farm tragedy along with his father Noel and brother Graham, the home side wasted a glorious opportunity to seal a game which they led 12-6 at half-time.
Instead Gregor Townsend's men stole away with the four points to remain unbeaten after two games while Ulster had only a losing bonus point - the same gain they managed from their shock 15-8 defeat at Newport Gwent Dragons the week before.
Harry Leonard's last-gasp penalty gave Edinburgh's new coach Alan Solomons a much-needed 16-13 win over a Newport Gwent Dragons side that have been a bogey team for them in recent seasons.
The Dragons looked on course to build on their opening day RaboDirect Pro12 win over Ulster with a draw at Murrayfield, but Leonard's penalty at the death broke their hearts.
It was a key victory for Edinburgh, who were beaten by Munster last week and were grateful for Tim Visser's converted try and Leonard's three penalties.
A second-half resurgence from Munster denied Zebre their first ever win in the Pro12 as the Irish province recorded a 43-21 victory.
Zebre went in at half-time 18-9 ahead after tries from Luciano Orquera and Andrea De Marchi but tries Paddy Butler, Denis Hurley, CJ Stander and Ian Keatley helped Munster to the win.
In the night's fourth fixture young Cardiff fly-half Rhys Patchell scored all 21 points for the Blues, who claimed their first win of the season in a 21-10 triumph over Connacht.