The home side had resumed on 380 for seven in reply to New Zealand's first innings of 469 all out but Gazi led a spirited fightback, including a ninth-wicket stand of 105 with Robiul Islam.
Gazi struck 10 fours and three sixes in his unbeaten innings of 101 before running out of partners as the hosts were dismissed for 501 after lunch.
Bangladesh lost Abdur Razzak in the fifth over of the day when he was trapped leg before wicket by Trent Boult for seven.
Robiul had made a patient 33 when seamer Doug Bracewell had him dismissed after a top edge which Ross Taylor caught on the run from first slip.
Taylor then took a more conventional catch to dismiss Rubel Hossain for four off the bowling of spinner Ish Sodhi to wrap up the innings.
The tourists were quick to eat into the 32 run lead and reached 117 for one in their second innings before rain brought an end to proceedings just before 4pm at the Zahur Ahmed Chowdhury Stadium.
Opener Hamish Rutherford was the man out, trapped leg before wicket by Nasir Hossain for 32 leaving Peter Fulton (44 not out) and Kane Williamson (28) to resume on the fifth day when more rain is forecast.
This is the 10th Test match between the teams with Bangladesh looking for their first win after one draw and eight defeats. The second and final Test starts in Mirpur on October 21.