A senior Russian official said the accusations by New York-based Human Rights Watch were exaggerated and that the government was monitoring workers' rights closely.
Human Rights Watch released its report as President Vladimir Putin visited Sochi to inspect preparations for the Games, which start a year from now.
It said more than 16,000 migrant workers had come to Sochi, on Russia's Black Sea coast, in the hope of finding work, largely in the construction of Olympic facilities.
In a report based on interviews with 66 of the workers, it said they were suffering "consistent patterns of abuse" in poorly paid jobs. Such workers, it said, had come from Armenia, Kyrgyzstan, Serbia, Tajikistan, Uzbekistan and Ukraine.
"People work, they don't get paid, and leave. Then a bus comes and unloads a fresh group of workers to repeat the cycle," it quoted a Ukrainian worker as saying.
It accused employers of illegally withholding pay and failing to provide sick pay, sufficient time off or suitable food and housing.
Human Rights Watch said migrant workers were particularly vulnerable as their legal status in Russia depends on them being in work. Employers frequently failed to provide copies of work permits and documentation, and threatened to denounce them to authorities as illegal immigrants, the report said.
The group also called for the International Olympic Committee to take a more active role in ensuring workers' human rights are respected.
"There is an Olympic charter that talks about dignity and the spirit of Olympism," said Human Rights Watch researcher Yulia Gorbunova.
"It is really not compatible with using and abusing people that are involved in constructing those incredible venues."
A spokesman for the IOC said it had "a longstanding commitment to follow-up" on human rights issues connected to the games and had taken steps ensure a handful of reported instances of non-payment of wages were resolved.
Deputy Prime Minister Dmitry Kozak, accompanying Putin in Sochi, said Russia was monitoring the workers' situation.
"There have not been enough complaints to deserve an international report," Kozak said, adding that 96,000 workers and 500 construction firms were employed in Sochi.
"It's clear that migrant workers don't have it very easy, in Sochi or in Moscow. But I cannot say that massive violations of people's rights are taking place during the preparations for the Olympics."
Olympstroy, a Russian state corporation created to oversee construction for the games, said that it had received only five complaints relating to wages, all of which had been resolved.