The Magpies looked to be heading into the fourth round when Papiss Cisse capitalised on a period of pressure by beating Cardiff goalkeeper David Marshall from close range in the 61st-minute.
However, it was Ole Gunnar Solskjaer's decision, in his first game in charge of the Welsh club, to send on Fraizer Campbell and then Craig Noone which eventually won the day as the midfielder blasted home a 73rd-minute equaliser within seconds of his arrival before the striker headed home the winner from Peter Whittingham's corner seven minutes later.
Asked how disappointing that was, Pardew said: "It is and I am bemused because I think we had a more than strong enough team to win that game.
"I felt they were in good shape, had good control of the game and really and truly, the last period just got away from us, and we can't afford that to happen.
"We want to win a trophy here, they (the fans) are desperate for a trophy. It's been a long, long time and we are not going to win a trophy going out in the third round."
As Pardew headed off to lick his wounds, Solskjaer was preparing to start work in earnest, although asked if that would include a January recruitment drive, he was coy.
He said: "You will see when and if we make something happen. This was a great start for me, I don't want to start talking about signings.
"We could go with the local lads - I have been thinking to ask the owner could we sign Giggsy, Aaron Ramsey, Gareth Bale? They might be some signings that we could so with, so maybe they are the targets."