"If Australian passports are being used or forged by any state, let alone for the purpose of assassination, this is of the deepest concern and we are getting to the bottom of this now," Rudd told public broadcaster ABC.
"We will not leave a single stone unturned."
Rudd said Australian officials had worked through the night after Dubai police named Australian, British, French and Irish passport-holders among the new group of suspects.
Dubai police strongly suspect Mossad, the Israeli spy agency, of carrying out the assassination in a luxury Dubai hotel where Mabhuh was found dead in his room on January 20.
"The new list of suspects includes people who offered prior logistical support and preparations to facilitate the crime, and others who played a central role," the emirate's police force said in a statement.
Foreign Minister Stephen Smith, who summoned Israeli ambassador Yuval Rotem for an explanation, said initial investigations showed the Australian passports were probably "duplicated or altered".
"We have made no conclusion about what to us, from our preliminary investigation, seems to be a serious abuse of three Australian passports either through forgery or identity fraud," Smith told reporters.
"But I made it crystal clear to the ambassador that if the results of that investigation cause us to come to the conclusion that the abuse of Australian passports was in any way sponsored or condoned by Israeli officials, then Australia would not regard that as the act of a friend."
The European Union also voiced outrage over the use of fake passports after an earlier list of 11 people, including a woman, was released. Israeli ambassadors in four European countries have been summoned for talks.
Rudd said he had discussed the latest development "at length" with Smith, describing it as "a matter of the deepest concern to Australia". Australia's ambassador in Tel Aviv was also seeking a meeting with Israeli officials.
Smith said Australian officials had reached the three passport-holders -- Bruce Joshua Daniel, Nicole Sandra McCabe and Adam Korman, who all live in Israel -- or their families.
"At this stage Australian officials have no information to suggest the three Australian passport-holders were involved in any way other than as victims of passport or identity fraud," he told parliament.
Israeli media reported that Australian-born Korman, 34, who works in a shop in Tel Aviv, had already voiced "shocked" at the news.
"I am shocked, it's identity theft -- simply unbelievable," Korman told the Ynetnews.com website.
"I have been frightened and shocked since receiving the news," he added. "It's irresponsible and a violation of human and individual rights to do such a thing."
Mabhuh, who masterminded a number of attacks on Israeli targets, was electrocuted and strangled to death during a mission in Dubai for the Islamist Hamas movement, according to his brother.
Israel has sought to play down the row, maintaining there is no evidence of its involvement. It has described Dubai police's calls for the arrest of Mossad chief Meir Dagan "baseless" and "absurd".
Britain's Sunday Times newspaper, quoting unnamed sources, has reported that Israeli Prime Minister Benjamin Netanyahu authorised the mission and met members of the hit squad shortly before their departure.
-------------------------------------------------------------------