Contested in the first modern Olympics in 1896 and part of the ancient Games in Olympia, wrestling will now join seven other candidate sports battling for one spot in a revamped programme.
It is unlikely, however, that it will get a reprieve when the IOC session in Buenos Aires votes on the recommendation in September as this would defeat the purpose of Tuesday's vote.
"This is not the end of the process, this is purely a recommendation," IOC spokesman Mark Adams told reporters following an Executive Board meeting. "It is the session which is sovereign."
"It was a decision to look at the core sports, what works best for the Olympic games. This was the best programme for the 2020 Olympics. This is not about what's wrong with wrestling but what is good for the Games."
The vote came as a major shock after other sports, including modern pentathlon and taekwondo, were seen as more at risk of losing out due to their low global appeal.
"FILA was greatly astonished by today's recommendation of the IOC Executive Board not to maintain wrestling among the 25 core sports for the 2020 Olympic Games," the international wrestling federation said in a statement.
It said the federation was represented in 180 countries, "with wrestling being the national sport in a fair amount of them and the only possibility for athletes to represent their country at the Olympic Games, thus contributing to their universality.
"FILA will take all necessary measures to convince the IOC... of the aberration of such a decision against one of the founding sports of the ancient and modern Olympic Games," it said, adding it would meet next week to discuss its next steps.
Board members were given a report on each of the Olympic sports which provided details on 39 criteria such as popularity, finances, tickets sold, anti-doping and governance, before a secret vote.
"There were different rounds of voting necessary to come to this conclusion," said IOC vice president Thomas Bach. "It is an extremely difficult decision to take."
"I cannot look into the heads of my colleagues. Such a decision is never based on one single reason. It is always a series of reasons. Of course, different members take a different approach."
"The common understanding is the purpose of this was to modernise, to look into the future of the Olympics," added Bach, a potential IOC presidential candidate later this year.
The 15-member executive board needed four rounds of voting to decide on wrestling with pentathlon, hockey, canoeing and taekwondo also getting votes to be dropped from the Games.
IOC president Jacques Rogge did not vote.
Canoeing scored the fewest and was eliminated first with taekwondo following it to safety in the next round, leaving three sports in the decider.
Wrestling got eight votes against it in the final round of voting with hockey and pentathlon tied at three votes each.
"I am very surprised by the result," board member and president of the International ice hockey federation Rene Fasel told Reuters. "Personally, I do not know why but that is what the majority wanted."
The IOC said 25 of the 26 Olympic sports were elected as core sports for the 2020 Games which will also include rugby and golf, making their first appearance in 2016.
Wrestling joins baseball and softball, making a joint bid, martial arts karate and wushu, rollersports, wakeboarding, squash and sports climbing as candidates for the one open spot.
Baseball and softball were taken off the programme in 2005.
The IOC executive board will meet in St Petersburg in May to determine which of these will be put to the vote in September.
Wrestling had 344 athletes at the London Olympics, competing in greco-roman and freestyle disciplines. Women's events were introduced at the Athens 2004 Olympics.
Russian wrestling federation chief Mikhail Mamiashvili was shocked by the decision but was confident his sport would remain in the Games.
"I'm absolutely convinced this ancient sport will retain its status," Mamiashvili, who won an Olympic gold medal in 1988, told Reuters.
"But FILA (the world amateur wrestling federation), the whole wrestling community must take a more active role in the process. We need to make some drastic changes in the sport, make it more attractive, especially for TV audiences," he said.
Olympic exclusion will be a major blow to the sport's popularity and financial stability as the Games are a global platform for the promotion of smaller, less established sports.
"It is very unfortunate," Satpal Singh, coach of India's twice Olympic medal winner Sushil Kumar, told Reuters. "It is being played from the first Olympics and is played all over the world."
International Modern Pentathlon Union president Klaus Schormann welcomed the news.
"In the last few years we acted and took decisions to make our sport more telegenic and more compact," Schormann told Reuters. "So every good news is further motivation for us."
Madrid, Tokyo and Istanbul are bidding to host the 2020 Olympics with a decision also to be taken in September.