Chelsea lodged an official complaint to the Premier League season after last Sunday's 3-2 home defeat by Manchester United, saying Mark Clattenburg, who sent off two of their players in, had used "inappropriate language" to two of their players.
Later in the week they issued another statement saying they did not have evidence to support a second complaint of verbal abuse but said they had lodged a complaint with the Football Association regarding comments to Nigerian John Obi Mikel.
The Football Association, which has had to deal with high-profile racism cases involving Chelsea's former England skipper John Terry and Liverpool striker Luis Suarez this year, has opened an investigation into the claims.
And Clive Wilkes told The Sun that there is a large amount of discontent amongst referees.
Wilkes said: “I keep in touch with a lot of the lads and there is such a strength of feeling about what is happening to Mark.
“I know a few referees who are even talking about boycotting Chelsea games because of all this.
“Remember, it’s not the first time Chelsea have targeted a referee — there’s been Anders Frisk and Graham Poll.
“And some refs are now saying enough is enough.
“It’s no exaggeration to say the refs in this country have never felt lower.
“They are so disillusioned. They feel vulnerable, feel that they are being left isolated — that they are getting no backing, no support.
“They want to speak out themselves but are too frightened, they fear they will be sacked if they go public with their grievances.
“There have even been murmurings about strike action but that’s very much the last resort.
“They’d much prefer to get around the table with the FA, PFA, the clubs and players to sort this out as quickly as possible.”
FIFA referee Clattenburg, who will not be officiating at any Premier League games this weekend in the wake of Chelsea's claims, was at the centre of a tumultuous clash at Stamford Bridge last week.
He sent off Chelsea's Branislav Ivanovic and then showed a red card to the home side's Fernando Torres after believing the Spaniard had dived to try and win a free kick.
He later allowed United's 75th minute winner, scored by Javier Hernandez, despite it looking suspiciously offside.
Some Chelsea players and officials are understood to have visited Clattenburg in his dressing room after the match.
Chelsea did not name any of the players involved in the claims regarding Clattenburg although would not deny that Mikel was one of them.
The claims made by Chelsea come in the wake of Terry being banned for four matches after the FA found him guilty of racially abusing Queen Park Rangers defender Anton Ferdinand.
Chelsea found themselves involved in another racism incident this week after a fan was photographed making monkey gestures during the 5-4 League Cup victory over United on Wednesday.