Daggers striker Rhys Murphy bagged the only goal of the game at Roots Hall, but the hosts were forced to play for more than an hour with 10 men.
"I really got in to the players at half-time, more than I have done before here, and they responded to that," said Brown.
"The first half wasn't good enough so we're looking at that rather than blaming the referee.
"We had a mountain to climb after the red card but the players wore the shirt with pride. They worked hard and couldn't really have given any more.
"There were plenty of positives for us in that second half and we could have got something from the game as their goalkeeper made a couple of very good saves, especially the one from Cauley Woodrow."
Dagenham also triumphed at Roots Hall in the Johnstone's Paint Trophy just a fortnight ago, winning 5-2 and they were helped on their way this time by a red card to Phillips after just 27 minutes.
The Shrimpers centre-back was ordered off after a reckless challenge on Murphy who, just four minutes later, scored the game's winning goal as he coolly fired past the advancing Daniel Bentley.
After the break, Murphy missed a good chance to double Daggers' lead before Medy Elito volleyed inches wide of the target.
But the Shrimpers fought back well with Woodrow having a scorching 30-yard shot tipped over by Dagenham goalkeeper Chris Lewington.
The shot-stopper then also did well to parry a powerful Freddy Eastwood shot around the right post.