Williams - whose sister crashed out of the event earlier in the week - lost the opening set in a lengthy tie-break, but hit back to claim the second with ease.
But Candian Bouchard held firm in the final set to claim the match 7-6(6) 2-6 6-4 and set up a quarter-final against either Jelena Jankovic or Ajla Tomljanovic.
Third seed Sara Errani came safely through her third round match, holding her nerve to beat China's Peng Shuai 7-6(6) 7-6(5).
WTA Charleston round three matches
6-Eugenie Bouchard (Canada) beat 11-Venus Williams (U.S.) 7-6(6) 2-6 6-4
Jana Cepelova (Slovakia) beat 13-Elena Vesnina (Russia) 7-6(4) 3-6 6-3
3-Sara Errani (Italy) beat Peng Shuai (China) 7-6(6) 7-6(5)
12-Daniela Hantuchova (Slovakia) beat Teliana Pereira (Brazil) 6-2 6-3
- Sports & Recreation
- Sara Errani
- Peng Shuai