Valentin Eysseric put the hosts in front from the penalty spot in the 11th minute and centre back Renato Civelli headed his fourth of the season with five minutes to go as Nice moved to 45 points from 26 games.
The southern club, 13th in the Ligue 1 table last season with 42 points, are hot on the heels of Marseille, who occupy the last Champions League qualifying spot and visit leaders Paris St Germain on Sunday.
"We have been in fourth or fifth place for a little while now so it gives us the will to stay there," Nice forward Eric Bautheac told French channel beIN Sport.
"It's early to talk about Europe but if we are still in the position to qualify in the last five games, we'll give it everything."
Nice were quickly rewarded for a good start when Alexy Bosetti was awarded a penalty after centre back Mickael Tacalfred fouled him in the box. Eysseric converted the kick.
The home team could have extended their lead but Bosetti narrowly missed the goal twice.
Stade Reims' timid reponse was not helped by full back Aissa Mandi's dismissal for a second booking in the 69th minute and Nice eventually sealed the victory when Civelli headed home a Bautheac free-kick.
Reims lie 17th with 24 points, one above the relegation zone.