Channel 4 announced this week it was not renewing the former British soldier's contract due to his "personal conduct" but Middleton insists he had already made the decision to leave the reality show.
The 40-year-old ex Royal Marine told Good Morning Britain: "Channel 4's statement is very reckless and desperate. Especially backed by excuses like the BLM tweet [which] I done a year ago, settled that, filmed two more series of SAS: Who Dares Wins.
"It's all smoke and mirrors. From the last episode of SAS: Who Dares Wins when I filmed the last episode, it became a half-scripted reality show. Full control was taken from me, from health and safety to production. And after that I said I'm not going to film any more, I'm done with SAS: Who Dares Wins.
"I wanted to part in a positive way and they knew I was going. And all of a sudden they dropped this bomb and ultimately they're protecting their brand. They got in there before me."
Middleton said the show – which pits contestants against each other as they undergo a two-week military training regime – was originally designed to make people feel uncomfortable.
Watch: A day in the life of Ant Middleton
He said: "If you're a snowflake, if you're easily offended – you do not belong on the set of SAS: Who Dares Wins. This is not Celebrity Big Brother – even though it's turned into that.
"You've got a group of military alpha males that are going to put people through a gruelling process. And if you don't like what you see then don't come and work on the set.
"But now it's turned into this PC, woke, everyone in their safety bubble – if you hear anything that offends you make sure you complain. That's a show gone. It's a reality show now. The PC patrol and the woke patrol have taken over."
Middleton admitted: "I have butted heads with a lot of people... Yeah, I probably haven't been easy to work with, but only because the course wasn't what I created."
He insisted he had wanted to leave the show as it was no longer the strict military course he had designed.
He said: "When we started it allowed us to be true to the military ethos to give an authentic programme. Over the years the PC patrol have kicked in, the woke patrol have kicked in. To the point where we can't say anything.
"We can't even act on the ground without health and safety jumping in and going, 'You can't put them in the water at shoulder level, you've got to put them in at waste height. It's so inauthentic now I decided to walk away."
Filming has already completed on the sixth series of SAS: Who Dares Wins, along with a celebrity special, both due to air later this year featuring Middleton.
The father-of-five – who resigned from his role as the Royal Navy’s chief cadet last year – came under criticism for comments he made about the Black Lives Matter protests as well as the coronavirus pandemic. He later apologised for both social media posts.
Middleton joined the army at the age of 17 in 1998 and served for 14 years. He served four years of a 14-month prison sentence in 2013 for unlawful wounding and common assault on two police officers outside a nightclub in Essex.
He recently took part in celebrity survival show Straight Talking with Australian actor Rebel Wilson.