Romaine Sawyers and Craig Westcarr put the hosts in control at the Banks's Stadium, with Francois Zoko's effort mere consolation before Lucas Akins was sent off in the 80th minute as the Saddlers ran out 2-1 winners.
Westley defended striker Akins, who was sent off in the 80th minute for an apparent punch into the stomach of Saddlers defender Paul Downing, with referee Andy Madley producing the red card after the fourth official had drawn his attention to the incident.
But Westley insisted: "It's the first time I've seen a fourth official get involved like that.
"Lucas was trying to break away from the defender, who was holding his shirt. He was just trying to get away.
"In the first half several of the lads were very unwell in terms of wearing the shirt. I made three changes in the second half and we brightened up a bit and took a bit of control, but then we had the situation with the fourth official.
"The last thing we wanted was to be reduced to 10 men, with the sickness in the camp, and in such a bizarre way."