However, Constable equalised after 54 minutes and just two minutes later Williams headed home to secure the win and earn the League Two leaders a trip to The Valley.
After a slow start Oxford grew into the game and a Johnny Mullins attempt was tipped over by Andy Coughlin before the Wrexham goalkeeper saved from Danny Rose.
Yet just as the visiting team were beginning to take control, Wrexham struck with Dean Keates crossing deep to Clarke who volleyed home after 29 minutes.
The Welsh side led at the break but two goals in two minutes saw them trail, with Constable equalising on 54 minutes, firing in from the edge of the box.
Jay Harris crashed a 25-yard shot wide as Wrexham pressed for an equaliser before Ryan Clarke produced a fine save to deny Robert Ogleby's fine effort.