Ponting, who stood down after nine years in the post in 2011, said the concerns over his successor-designate and vice captain - who he called "Pup" - had prevented him from considering resigning earlier.
"It wasn't that he was disruptive or treacherous and publicly he said all the right things," Ponting said in an extract from his autobiography, "At The Close Of Play", serialised in the Herald Sun newspaper.
"But he had never been one to get too involved in planning sessions or debriefs at the end of a day's play, or to volunteer to take on any of the captain's workload.
"More than once, (coach) Tim Nielsen and I had encouraged him to take on more of a leadership role within the group, but when Pup was down on form or if he had a problem away from cricket, he'd go into his shell.
"I knew he was an excellent thinker on the game, but for a long time I was concerned that he wouldn't be able to handle the huge variety of 'little things' that go with being Australian captain."
Those feelings had been compounded by the way Clarke distanced himself from the team as he became more and more involved in a celebrity lifestyle off the field.
"Pup remained a good trainer and we could all see that he loved playing for Australia and was determined to do well," Ponting recalled.
"But away from cricket, he moved in a different world to the rest of us.
"It never worried me if a bloke didn't want a drink in the dressing room, but I did wonder about blokes who didn't see the value in sticking around for a chat and a laugh and a post-mortem on the day's play.
"This was the time when we could revel in our success, pick up the blokes who were struggling, and acknowledge the guys who were at the peak of their powers. Pup hardly bought into this tradition for a couple of years and the team noticed."
Ponting said Clarke's attitude changed after his team mates "closed ranks around him" during his well-publicised personal problems with his then fiancée Lara Bingle in early 2010, which caused him to return from the tour of New Zealand.
"His official reign as Australian captain started on a high, with ODI wins in Bangladesh and ODI and test wins in Sri Lanka, and he quickly took his batting to a new level, to the point that it seemed he could almost score big hundreds at will," Ponting wrote.
"He now seemed happy to take on the planning, media and administrative duties that he'd veered away from when he was vice-captain and the mood in the Aussie dressing room was positive.
"Perhaps I'd been wrong to be so concerned for so long."