OM, who played better than the visitors for most of the Group F game at their Stade Velodrome on Wednesday, conceded two second-half goals from Theo Walcott and Aaron Ramsey before Jordan Ayew reduced the arrears in injury time.
"It's frustrating because before they opened the scoring we were in control," said coach Elie Baup.
Marseille's chances of qualifying from a group also featuring Napoli and last season's runners-up Borussia Dortmund have already taken a serious blow, with the Ligue 1 team travelling to Germany in two weeks.
Arsenal lead Group F with Napoli, who beat Borussia 2-1 on Wednesday.
"We will need to build on what we did good (against Arsenal) and be more efficient," Baup explained.
"There is no room for error in the Champions League - that's what high level is about. Arsenal weathered the storm and then took their chance."
The London side, who barely created an opportunity in the first half, stepped up a gear after the break and Walcott made the most of Jeremy Morel's defensive blunder to open the scoring after 65 minutes with a fine half volley.
Ramsey added a second goal with seven minutes left, firing home with a low drive after being left unchallenged just outside the area.
"Marseille played a good game but we were patient enough to win it in the second half," said Arsenal manager Arsene Wenger.
"We had one or two opportunities and we converted them," said defender Laurent Koscielny.
Two years ago, Arsenal beat OM at the Velodrome in a Champions League Group game, winning 1-0 with a late goal.
Although Wednesday's result suggested OM have not improved in two years, the Marseille players believe they are on the right path.
"We proved that we are a match for great teams like Arsenal," forward Dimitri Payet said.
"We now have to win an away game to make up for this defeat. We have no choice," said Alaixys Romao.
That is exactly what they did two years ago, when a brilliant strike by Mathieu Valbuena earned them a 3-2 win at Borussia Dortmund and sent them into the last 16.