The visitors looked set to hang on for victory after left-back Ben Coker had given them the lead with a 23rd-minute free-kick.
But after Coker was sent off in the 88th minute, Anton Forrester headed home from a free-kick in the first minute of stoppage time to spare Bury from a fourth straight defeat.
While Southend still ended their four-match losing streak, Brown questioned his defenders' discipline and some of referee Eddie Ilderton's decisions.
"Abstract naivety is costing us games," he said. "We've lost four on the bounce so to throw away two points at the end was so poor, so naive, so I'm bitterly disappointed.
"It was the referee's performance, first and foremost, and also the free-kicks that we gave away.
"We must have given away 25 free-kicks, which invited Bury to put the ball into our six-yard box. In 90 minutes of football something will give - and it did.
"Up until then we were very comfortable. People will say you're never comfortable at 1-0. Well, you should be. One goal should be enough to win a game but then we conceded a sucker punch at the end.
"We have to learn from it. If we do then we'll have a chance of getting out of League One."
Bury had started strongly, with Tom Soares having a shot turned wide by Southend keeper Dan Bentley.
The visitors replied and although Bury keeper Trevor Carson saved centre-half Mark Phillips' header, Coker curled in a free-kick from a tight angle.
Winger Anthony Straker then caught Nathan Cameron in possession and squared for Freddy Eastwood but his shot was smothered by Carson.
Southend's other winger, Kevan Hurst, volleyed narrowly over just after the break while Eastwood wasted another chance.
As Bury struggled to turn their possession into chances, Will Atkinson fired wide on the break for the Shrimpers.
But Coker was dismissed after being booked twice in eight minutes before Forrester headed home from substitute Jordan Sinnott's free-kick.