"It's when there are a lot of them that there are problems," he said in the film Le Monde said shows him getting ready to pose for a photo with a young man from France's large community of Arab origin.
The young man, referred to as Amin, is seen in the middle of a small group of laughing and joking activists from Sarkozy's UMP party at the party's summer gathering in Seignosse in southwest France on September 5.
One woman can be heard saying about the man, who is also seen laughing and apparently enjoying himself: "He eats pork, he drinks beer."
To which Hortefeux replies: "He doesn't correspond at all to the prototype." He then goes on to make his comment that it is "when there are a lot of them that there are problems."
The Socialists, the main opposition party, called for Hortefeux's resignation and described the comments as "shameful and unspeakable."
"The question is not whether he (Hortefeux) should resign or not, but what he is still doing in the government," said Socialist party spokesman Benoit Hamon.
A range of rights groups and opposition politicians expressed outrage at what they said was a clearly racist remark, with many calling for the minister to step down from the job Sarkozy appointed him to in June this year.
But Hortefeux issued a statement to denounce "a vain and ridiculous attempt to create a controversy" and said that not a single word uttered by the minister made "reference to the supposed ethnic origin of the young activist."
And the young activist himself told Le Monde that the minister's comments had been taken out of context.
"It's disgraceful. I am Arab but he completely respected me, it wasn't at all out of place. And I do not consider that it was a blunder," said the man, who was not named by the newspaper.
Prime Minister Francois Fillon stepped into the row to stand by his interior minister, declaring that Hortefeux "was the "victim of a fairly scandalous campaign of defamation."
"Nothing he said could be held against him," Fillon told French television station TF1.
The controversy came just two days after Hortefeux put an end to a top French official's career after the man was accused of a racial slur at a Paris airport.
Hortefeux issued a statement condemning racism when he initially suspended the official in August.
"I will never tolerate racist or discriminatory speech in our country, especially by a state official of any kind. This behaviour violates the values of our republic," he said.
When Sarkozy came to power in 2007, he made much ado about appointing to his government several ministers from a range of ethnic backgrounds.
France has several million people of immigrant stock, many of them from former colonies in north Africa, but only a small number have made it into the country's business, political or social elites.
Hortefeux was immigration minister from 2007 until early this year.
During that time he increased the numbers of illegal immigrants forcibly expelled from France and developed the network of detention centres for illegal migrants.
-------------------------------------------------------------------------------------------------------------------------------