Bilbao demolished their century-old San Mames arena, popularly known as "the cathedral", at the end of last season and played their first match in the new stadium nearby even though one end is still to be completed.
Celta forward Charles dampened the party atmosphere when he put the visitors ahead in the 14th minute. Bilbao levelled four minutes later when Mikel San Jose pounced on a loose ball and fired past goalkeeper Yoel.
Captain Andoni Iraola made it 2-1 to Bilbao in the 61st minute and Benat sped clear and finished clinically to put them two goals ahead in the 68th before Santi Mina scored for Celta in the 79th to set up a nervous finale for the home side.
"Let's see if it's a good omen for us," Bilbao coach Ernesto Valverde said.
"The stadium is incredible and it will be even better when it's finished."
Bilbao are fifth on nine points from four matches, one behind Real Madrid and Villarreal and three adrift of joint leaders Barcelona and Atletico Madrid, the only teams with perfect records.
In the night's other La Liga match, Elche and Valladolid played out a goalless draw at the Estadio Martínez Valero.