Union boss John Setka is threatening the Labor Party with a legal challenge and a freeze on all donations after Anthony Albanese said he wants him expelled.

Labor leader Anthony Albanese has called for the removal of the Construction, Forestry, Mining and Energy Union (CFMEU) secretary from the Labor Party, calling his behaviour “completely unacceptable”.

Mr Setka reportedly attacked Ms Batty, the 2015 Australian of the Year, saying her anti-domestic violence advocacy harms men’s rights.

ACTU’s Sally Mcmanus has told John Setka “it’s in the best interest of the union movement” for him to resign.

Now 12 more unions are calling for Mr Setka to resign, despite his insistence he will not be stepping down.

Former Labor senator Graham Richardson tells Ben Fordham Mr Setka will eventually have to quit.

“In the end, this bloke’s a goner.

“He will fight it tooth and nail but he won’t succeed in his fight.

“He’s done so many things over such a long period that I think, cumulatively, he should have been knocked off about a dozen times.”

Peta Credlin agrees and says he must resign.

“If Labor wants to get back in the game it has got to deal with Setka.

“It’s not about Rosie Batty, this guy’s got a rap sheet… as long as my arm and then some.

“This is the problem with the most militant union we’ve got in this country, it needs to clean up its act.”

Click PLAY to hear the full interview