A resounding 71.94 percent of voters said no to the proposal with 28.06 percent in favour.
Austria hosted the 1964 and 1976 Winter Olympics, both in Innsbruck, and co-hosted the Euro 2008 soccer championship with Switzerland but has never staged a summer Games.
The rejection came 10 days after Swiss voters in the Graubuenden canton voted against a plan to bid for the 2022 Winter Olympics.
"We accept the result of the referendum," Karl Stoss, president of the Austrian Olympic Committee, which had backed the proposal, told Austrian media.
"We would welcome any investments in sporting infrastructure by the city of Vienna so that we can offer top training conditions to our athletes and so that, from now on, Vienna can regularly stage big international sporting events, even if they are not on the scale of the Olympics."
