Authorities said the robbery took place around 11:30 am local time (0930 GMT) and targeted an exhibition entitled "Extraordinary Diamonds" by the Leviev jewellery house.
A lone gunman, wearing a baseball cap with his face hidden by a "bandana or scarf", entered the hotel through a front entrance undisturbed as security guards patrolled the premises, according to the prosecutor's office.
"It all happened very quickly and without any violence," it said, adding that the robber, armed with a semi-automatic pistol, escaped with a briefcase containing jewels and watches encrusted with diamonds.
Police said they had not been alerted that the exhibition was taking place, which while not a legal requirement would ordinarily be standard procedure for luxury hotels such as the Carlton.
The "Extraordinary Diamonds" exhibition was held in a wing of the hotel with direct access to the street.
Huge posters of the diamonds with images of pink flowers had been affixed to the walls to advertise the jewels' presence to the public.
On its website, Leviev, which opened its first jewellery store in London on Old Bond Street in 2006, claims to have access to stones "among the most outstanding in the world."
An inventory was now underway, and should be completed by Monday, a source close to the investigation said. The estimate of 40 million could not be confirmed, but "tens of millions of euros" were likely to have been lost, the source said.
The management at the Carlton Hotel declined to comment when contacted by AFP, saying they had "instructions not to say anything."
The Carlton was the location for Alfred Hitchcock's 1955 Oscar-winning thriller "To Catch A Thief", starring Cary Grant and Grace Kelly.
The diamond heist was one of the largest in recent years in France.
In the country's biggest ever robbery, which took place in 2008, three men stole almost every piece on display at a jewellery exhibition in Paris with an estimated value of 85 million euros.
Cannes has fallen prey to thieves several times recently, notably during this year's film festival, which attracts a glittering array of celebrities from the movie world.
In a pre-dawn heist at a hotel during the festival in May, thieves stole jewellery worth $1.4 million due to be loaned to movie stars.
That robbery took place in the hotel room of an American employee of Swiss jeweller Chopard while she was out for the evening, police said.
In a scene straight from a Hollywood film, a strongbox containing jewels was ripped out of the wardrobe and carried off, they said.
In a second theft during the festival, robbers made off with a diamond necklace with an estimated value of $1.9 million.
At least two apartments rented by film executives were also burgled during the 2013 festival, with thieves taking cash, jewellery and other personal items.
And in 2012, thieves made off with four watches worth 400,000 euros belonging to football stars Souleymane Diawara and Mamadou Niang, who were in Cannes for the festival.
The world's biggest diamond theft occurred in February 2003 in the northern Belgian city of Antwerp, when robbers got away with jewels worth an estimated 100 million euros.
----------------------------------------------------------------------------------------------------