Gardiner denied the claims and subsequently threatened to take legal action against the reality star.
Now, the Australian choreographer has claimed on the White Wine Question Time podcast that ITV producers gave him an ultimatum - if he continued with the legal action against Collins he would be fired from the show as he could no longer be impartial as a judge.
Gardiner also accused producers of wanting to maintain the tension on the show, and says they indulged Collins's “unprofessional behaviour” in order to keep the show “fiery”.
He told podcast host Kate Thornton: "There were many situations over the years where I was hung out to dry, as it were, and I wasn't supported.
"Yet, by the same token, I was very encouraged by the producers to 'keep doing what you're doing, because you're bringing in the viewers, you're also bringing in the press.'
"I was in a very difficult position of getting abused for doing what I was doing, and then getting praised by the producers to continue to do it."
WATCH: Jason Gardiner on his Dancing On Ice exit, his new life in nature, and his return to panto this year
Gardiner says the abuse he received included death threats that left him feeling paranoid about his safety.
He said the relationship with producers and with ITV had turned "toxic" and he couldn't deal with it anymore.
"I had an epiphanous moment where I knew that if I went back, I was opening myself up to the same abuse," he said.
"That's exactly what it is. It was abuse. I said: 'I'm not a victim of abuse, I'm not going to put myself in that position ever again.'"
While ITV has declined to comment to Yahoo, we do know that in June 2019 it published a Duty of Care Charter which laid out its commitment to the mental and physical well-being of all people working for, or with, ITV.
The same month ITV Studios introduced refreshed processes and guidance throughout their content-making business to manage and support the mental health and well-being of programme participants before, during and after production.
Gardiner said he didn't think the processes had “trickled down” by the time he left the show in August 2019 and “not once” was he offered support while working on the programme.
"On the one hand," he said, "they're pushing this narrative of men's mental health. And then on the other hand, in real terms, I was not getting any help, whatsoever.
"Not once did they say to me, how are you doing through all of this? Because for them, at the end of the day, everybody was talking about the show.
"So there's no such thing as bad publicity. It's good publicity for them, because everybody was talking about the show, it was driving interest."
Pressure from the press, death threats and “heavyweights like Piers Morgan berating you on national TV” all led him to call it a day because he didn't want to “destroy” his mental health.
He said he was “worried for his life” and it became a “paranoia”.
"I thought, I've got to stop this, and it's my choice for putting myself in that spotlight," he explained.
"In that position, that is causing this knock-on effect. What do I need to do? I need to cut out that toxic environment and that toxic job and get on with my life."
Like a divorce
Gardiner is now working in perma-culture - the practice of creating self-sufficient eco-systems - and has spent time since leaving the show living in tents, building schools and sustainable housing for refugees across Europe.
He said being in isolated settings had given him time to think: "I'm starting to remember all of the good times, all of the good memories, and the laughs that I've had on that show.
"It was an amazing show, and I loved being a part of it."
But he says the moment he left the programme was like a divorce and remains resentful at how it ended.
He says the producers chose not to wade in on the row between him and Collins, despite being people he had built up "years of working relationships with".
Calling Collins “very unprofessional” throughout the 2019 series, he said everyone, including hosts Phillip Schofield and Holly Willoughby, was left waiting “for hours and hours, just because she didn't like her hair and makeup”.
He claimed this behaviour was 'indulged' by producers because it would create tension.
"These shows really work under the premise of there being a lot of tension, it's more fiery, isn't it?" he said.
"So [Collins] was worked up, we were all p****d off, because most of our day was waiting for her to come out of hair and makeup.
"Then when she got on to set, she decided that she didn't really want to partake in anything, she would just sit and watch.
After she accused him live on air of selling stories about her, Gardiner said his lawyers approached him and asked if he wanted to pursue legal action, and he told producers he was “really seriously” thinking about it.
He claimed he asked producers for backing in public that he hadn't sold stories but “that didn't happen”.
"Instead," he said, "An hour before we were to go live, the big wigs - four of them - from ITV came up to the studios, piled into my dressing room.
"I was given the ultimatum: If I was going to pursue any legal recourse against Gemma Collins, because she's terrified that I'm going to do this, that they would fire me from the show."
Gardiner says he had no intention of launching legal proceedings during the show because he understood it would impact his impartiality as a judge. But he says ITV refused to back him publicly following Collins’ allegations.
"I've never [sold stories], I will never do it. And so to be accused of that live on air in front of millions of people, then the pickup in the press just propagates it more and more and more.
"You get to the point where it's like: 'I have a right of reply'. And ITV were not allowing me to do a right of reply, they said that it would just be making a mountain out of a molehill."
WATCH: Jason Gardiner on his major new life shift, from talent show judge to building sustainable houses