Goals from Christian Benteke and Charles N'Zogbia put Villa in control and all West Ham could manage in reply was an Ashley Westwood own goal as the home side held on.
Although in the final moments Villa `keeper Brad Guzan pulled off two fine saves to deny Kevin Nolan and Carlton Cole.
Allardyce said: "I thought we had saved it but we should never have been in that position. That's the biggest disappointment.
"We should never have got ourselves in a losing position because we'd created the chances. We can't play any better than we played.
"We then created the chances to get ourselves back to 2-2 and we failed to do so and that's been our problem away from home this season.
"It was great goalkeeping in the end. Guzan pulled off some really good saves.
"I thought we sat comfortable from the chances Villa created and we gifted them a soft penalty when we should not have been in that position.
"It was a great free-kick from Charles N'Zogbia so we can't criticise that too much. The overall situation was disappointing because of the way we played. We dominated and need to put the ball in the back of the net when we get a chance."
Defender Joey Brien will miss West Ham's warm weather break in Dubai this week after suffering a thigh injury.