Few had given the Welsh club a chance of pulling off an upset in their first Premier League home game, and they fell behind to Edin Dzeko's stunning strike early in the second half.
But they hit back as Aron Gunnarsson tapped in from close range before Fraizer Campbell lost Pablo Zabaleta twice from corners to head home and spark scenes of delirium at Cardiff City Stadium.
Alvaro Negredo got a goal back for City in stoppage time, and spurned a last-gasp chance for an equaliser, but it was not enough as Pellegrini was given a rude awakening after their handsome opening win over Newcastle.
He said: "I have not seen them yet, but defending set-pieces is the duty of all the team. Not just the goalkeepers and the defenders.
"It does not matter who is guilty with the two corners, conceding from two corners is too much, it is too important an area.
"When we scored the first goal I thought it would be easier for us as we were playing a team defending with 10 players inside their box.
"It was very difficult for us and I think while we did not do a very good match we did not play badly.
"We had the patience to try and score, we did it but it was a pity with the two corners."
Cardiff manager Malky Mackay said he had visions of the last-gasp goal that won City the title in 2012 as his side hung on.
City scored twice in stoppage time on the final day of the 2011/12 season.
"They do not lie down and I had visions of Sergio Aguero scoring against QPR in the last game of the season a couple of years ago in injury time," Mackay said.
"You want to play against the best in the best league in the world and we are doing that and I am proud of everyone at the club. There have been lots of downs but everyone here will have a smile on their face tonight."
It was The Bluebirds' first top-flight victory since 1962 with Cardiff having lost their Premier League opener at West Ham United last weekend.
"We deserve to be here. We have a great record at home," Mackay said. "Everyone realises how difficult it is to win away from home. We have to make sure Cardiff is the same sort of fortress it was in the Championship last season."
The South Wales club, who won the Championship comfortably last term, have seen plenty of change since they were last in the top tier.
Now under the ownership of Malaysian billionaire Vincent Tan, they have changed their club colours from blue to red, and spent heavily since sealing promotion to try to ensure their stay in the Premier League lasts more than a single season.
Mackay said this win was just rewards for the support shown to the club in a part of the country where they must compete with rugby for attention.
"The atmosphere was fantastic before the game and to go and give a performance like that against a team like that was amazing," he said.
"We have to believe we can compete. The team were incredibly disciplined and as the game grew, we grew. We knew we had the ability to counter them but they are a very good team and if we didn't do our job properly they would punish us."