Shaun Batt netted a hat-trick to continue the London club's remarkable start to the season which also sees them top League One with five wins out of five.
Bradley Dack headed a consolation for the Gills but it was Orient who progressed into the second round.
"We're disappointed to lose but I thought it was a really good effort from our team," Allen said. "I thought we played very well.
"We were 2-0 down at half-time and there were times when we thought 'how did that happen?'
"We had some good possession and good chances but just weren't able to unlock the door.
"On the whole it's a challenge now to work out how to make it better because the performances have been good.
"I think the supporters who have paid to come and watch will say we gave it a good go but we've been punished and it was a tough one.
"Leyton Orient are top of the league and won five out of five. I watched them dominate Hull the other week so we can take a lot from this result."