The Italian, who joined the club in late 2009, won the FA Cup in 2011 and the Premier League title last season.
This season, however, the club have lost out to neighbours Manchester United in the league, suffered a shock defeat to Wigan Athletic in Sunday's FA Cup final, and were eliminated in the group stages of the Champions League.
Rumours have been swelling that Malaga manager Manuel Pellegrini will replace the Italian, though the Chilean denied any such agreement was in place.
Mancini, who had four years left on a five-year contract, has been reportedly paid £28 million in compensation by the club for the termination of his contract - although contrasting reports suggest that could be limited to £7m, the equivalent of a single year's wages, by a clause in the deal he signed last summer.
Assistant manager Brian Kidd will take the reins for the club's two remaining Premier League fixtures this season.
"It is with regret that Manchester City Football Club announces that Roberto Mancini has been relieved of his duties as Manchester City Manager," the club wrote in a statement.
"This has been a difficult decision for the owner, Chairman and Board to make and it is the outcome of a planned end of season review process that has been brought forward in light of recent speculation and out of respect for Roberto and his extensive contributions to the Football Club."
Chairman Khaldoon Al Mubarak paid tribute to Mancini, despite his axing: "Roberto’s record speaks for itself and he has the respect and gratitude of Sheikh Mansour, myself and the board for all of his hard work and commitment over the last three and a half years. He has clearly also secured the love and respect of our fans.
"He has done as he promised and delivered silverware and success, breaking the club’s 35-year trophy drought and securing the title in 2012. I would like to personally and publicly thank him for his dedication to the progress that he has overseen and for his support and continued friendship."