Phelps won the male athlete of the year award for the third time after the swimmer captured four gold medals in London to finish his career with an unprecedented tally of 18 golds, twice as many as the next best from any Olympic sport.
Felix won the women's award after scooping winning gold in the 200 metres sprint and the 4x100m and 4x400m relays on the running track.
The women's rowing eight crew won the team award while swimmer Jessica Long and wheelchair racer Raymond Martin won the top individual Paralympic awards and the men's quad doubles wheelchair tennis pairing won the Paralympic team award.
"We are so proud to honour these young men and women who represent the finest ambassadors of sport, Team USA and the Olympic Movement," USOC chief executive Scott Blackmun said.
"Through their commitment to excellence on and off the field, these athletes represent our country with incredible grace and composure, and their outstanding achievements are an inspiration to all Americans."