The absence of the 17-times Grand Slam winner was announced a day after world number three Rafa Nadal withdrew from the August 4-12 tournament with a knee injury and robs the event of two of its biggest attractions.
"I have always enjoyed playing there but after a long stretch of tournaments, I will need some time to recover," Federer said in a statement. "I look forward to coming back to Canada next year."
Federer's decision came shortly after he defeated Argentine Juan Martin del Potro 3-6 7-6 19-17 in the Olympic men's semi-final on Friday, which was the longest men's three-set match played in the professional era.
The 30-year-old Swiss will play either world number two Novak Djokovic from Serbia or Britain's Andy Murray in Sunday's final.
Additional main draw withdrawals for the Toronto Masters include David Ferrer (Spain), Gael Monfils (France) and Fernando Verdasco (Spain).
In an bid to avoid withdrawals stemming from participation in the London Games, the Toronto field was trimmed to 48 players from 56 with the top 16 seeds receiving first-round byes.