In a letter to the citizenship ministers in both states, the IPA's Simon Breheny writes: "I question your commitment to the fundamental human rights to freedom of speech and freedom of thought.

"It is troubling that state Liberal governments are failing to defend the values they have been elected to uphold.

"Section 18C makes it unlawful to 'offend, insult, humiliate or intimidate' a person on the grounds of that person's race, colour or national or ethnic origin.

"Section 18C is a substantial restriction on freedom of speech. Most famously, it was the provision used to silence conservative journalist Andrew Bolt in September 2011.

"Yet, not once in your media release do you mention freedom of speech or the limitation this provision places on Australians' political freedoms."

The NSW and Victorian governments lodged separate submissions opposing the changes championed by Attorney-General George Brandis but released a joint statement on May 2, saying a watered-down law would "threaten the social cohesion and well-being of not just our states' culturally and religiously diverse communities, but also the wider Australian community".

The government has received more than 5500 submissions from the public on its draft exposure bill on changes to the Racial Discrimination Act but Senator Brandis has refused to make any of them public.

Fairfax Media recently revealed that of 60 submissions made public by ethnic and community groups, just four submissions - including one by Holocaust denier Fredrick Toben - support the repeal of Section 18C and D of the act.

On Monday, the Attorney-General's department knocked back a freedom-of-information request by Fairfax for access to the submissions, claiming it would take "400 hours" work to fulfil the request.

Senator Brandis has denied reports that the government was backing away from its original changes but has faced public criticism from the IPA, which said any weakening of the draft exposure would amount to a broken election promise.

Making the case for the controversial changes in the Senate in March, Senator Brandis said: ''People do have a right to be bigots, you know.''

In his letter, which was also sent to all state Liberal MPs in Victoria and NSW, Mr Breheny "reminds" the ministers of the values of the Liberal Party and the words of party founder Robert Menzies, who said "the whole essence of freedom is that it is freedom for others as well as ourselves".

Mr Breheny writes: "Expanding our political freedoms is the duty of every member of parliament that has been elected to defend liberal values. I encourage you to re-commit to the values and beliefs of the Liberal Party, and to reconsider your position on this issue of vital importance."

NSW Minister for Citizenship and Communities Victor Dominello said the NSW government opposed any change to the staus quo because "no compelling case has been made that the current system is broken".

"Freedom of speech is critical to our way of life, but it has never been an unfettered right and must be balanced against other protections," he said. "There are a number of limitations to our freedom of speech - for example, the laws of defamation, privacy and trade practices.''

Victorian Minister for Multicultural Affairs and Citizenship Matthew Guy declined to comment.