"If we are to defeat this threat, I believe it's time to turn the page on the failed policies of the past," he told the conference during a panel discussion attended by German Chancellor Angela Merkel.
But his remarks sparked a chorus of condemnation at home, with the Muslim Council of Britain expressing anger that the Muslim community was still being treated "as part of the problem as opposed to part of the solution."
Many criticised his timing, on the day that 2,000 members of the far-right English Defence League (EDL) held a rally in the ethnically mixed city of Luton to protest against the spread of Islam in Britain.
Cameron did not mention the EDL but a lawmaker for the opposition Labour party, Sadiq Khan, accused him of "writing propaganda for the EDL".
The prime minister's speech was the first of his premiership on Islamist extremism, a major concern for British governments since four home-grown suicide bombers attacked London in 2005, killing 52 people.
It echoed controversial comments made by Merkel last year, when she also called multiculturalism a failure, saying Germany had not devoted enough attention to the integration of immigrants.
Cameron, who took power in May 2010, argued that "under the doctrine of state multiculturalism, we have encouraged different cultures to live separate lives, apart from each other and the mainstream".
"All this leaves some young Muslims feeling rootless. And the search for something to belong to and believe in can lead them to this extremist ideology," he said.
The response, he argued, should be "a lot less of the passive tolerance of recent years and much more active, muscular liberalism".
"A passively tolerant society says to its citizens: as long as you obey the law, we will leave you alone. It stands neutral between different values," he said.
"A genuinely liberal country does much more. It believes in certain values and actively promotes them."
Cameron was careful to distinguish between Islam the religion and the political ideology of Islamist extremism, saying they "are not the same thing."
But Muslim groups in Britain reacted with a mixture of disappointment and outrage.
"It is disappointing," Faisal Hanjra, assistant secretary general of the Muslim Council of Britain, told BBC radio, adding that it did not appear to mark a shift in the approach to tackling terrorism.
"Again it just seems the Muslim community is very much in the spotlight, being treated as part of the problem as opposed to part of the solution."
Mohammed Shafiq, chief executive of British Muslim welfare group the Ramadhan Foundation, added: "Singling out Muslims as he has done feeds the hysteria and paranoia about Islam and Muslims."