The Russian side went through thanks to Porto's 2-0 defeat at Group G table-toppers Atletico Madrid who finished on 16 points.
Zenit dominated most of the match but Vienna keeper Heinz Lindner was in great form, saving three times in quick succession in the first half to deny Hulk twice and Cristian Ansaldi.
Zenit deservedly went ahead on 35 minutes when a fine through ball by Hulk found Aleksandr Kerzhakov who rounded Lindner and fired home his seventh goal of the season.
A minute before halftime, Vienna were level when Philipp Hosiner collected the ball on the edge of the box and hit a low right-foot drive past Yuri Lodygin.
Tomas Jun made it 2-1 after 48 minutes when he got on the end of a great cross from Markus Suttner and tucked it away from close range and three minutes later it was 3-1. Hosiner was quickest to react when Yuri Lodygin pushed out an effort from Florian Mader.
Romain Kienast made it 4-1 in stoppage-time for Vienna's first Champions League win in their debut season.
Porto and Vienna finished on five points but Porto ended third because of a better head-to-head record against the Austrians.