Armand Gnauduillet's header in the 87th minute and Tendayi Darikwa's strike two minutes into stoppage time snatched the Spireites their fourth League Two win in their opening five games.
But Pompey's hopes of victory suffered a huge blow seven minutes before the break when Connolly was sent off for an off-the-ball incident with Chesterfield's Sam Hird.
He said: "David has apologised for it, but I didn't see the incident. Of course, you are a little bit surprised by it. He's a seasoned pro.
"I think he was wound up by one or two decisions he didn't get. That's no excuse, but the referee is probably the only one who didn't think it was a penalty, Even the Chesterfield staff said it was a stonewall penalty."
"We all feel disappointed we didn't come out of the game with any points, but we have showed people there is a real desire and spirit to do well in front of our great fans.
"The positives outweigh the negatives. We fought for everything. We created chances even though we were down to 10 men."
Chesterfield manager Paul Cook admitted the Spireites lacked cutting edge in attack after they left it late to seal the points.
"We were making bad decisions all over the pitch," he said. "But to come to a place like this and get a win; we must give the lads great credit.
"It was a fantastic result. It wasn't a fantastic performance, but it was certainly a fantastic result."