"While finding a way forward on Syria will not be easy -- it's not going to be automatic -- it is the most promising opportunity for a political opening we have seen," Kerry said just before he was to set off for Vienna.
"The challenge that we face in Syria today is nothing less than to chart a course out of hell," he added.
Washington is at loggerheads with Moscow over Syria, accusing Russian forces of concentrating their air campaign there on moderate opposition groups opposed to Syrian President Bashar al-Assad's rule.
Moscow says its bombing campaign is targeting Islamic State jihadists and also other "terrorist" groups.
But Kerry stressed that the US and Russia also shared "common ground", arguing that both want "a united, secular Syria" in which citizens can choose their own leader through elections.
The inclusion of Iran -- a key ally of Assad -- in this week's meetings marks a crucial shift after Tehran was excluded from earlier talks, mainly because of opposition from Washington and Riyadh.
"We have reviewed the invitation, and it was decided that the foreign minister would attend the talks," Iranian foreign ministry spokeswoman Marzieh Afkham said.
It will be the first time all the major international players in the conflict are in the same room, but there has been no mention of either the Syrian government or the opposition attending.
- Divisions over Assad's future -
Top diplomats from Russia, the United States, Saudi Arabia and Turkey will also meet beforehand on Thursday evening, the second time the quartet will have met in less than a week.
The following day will see them joined by Britain, Egypt, France, Germany, Italy, Lebanon and the European Union.
Britain's Foreign Office said the United Arab Emirates, Qatar, and Jordan will also be present.
After months of failure, efforts to find a breakthrough have gained pace as hundreds of thousands of Syrians have fled their shattered country.
Serious divisions remain over when or whether Assad should step down -- and four-way Russia-US-Saudi-Turkey talks in Vienna last Friday failed to make a breakthrough.
On one side, Russia and Iran are backing Assad's forces on the ground and say Damascus must be helped to defeat "terrorism" before a political process can take shape.
On the other, the United States and its key regional allies Turkey and Saudi Arabia are supporting groups fighting Assad and insist he must go.
French Foreign Minister Laurent Fabius said Wednesday that France and its allies had agreed on the need for a "precise timetable" for Assad's departure.
But Kremlin spokesman Dmitry Peskov said it was impossible to discuss the details of a political solution before delivering "the final blow to terrorists".
"It would be absolutely illogical to act otherwise," he was quoted as saying by Russian news agencies.
Meanwhile, the Pentagon announced plans to step up attacks on Islamic State jihadists in Syria and Iraq, with Defence Secretary Ashton Carter saying he expected more airstrikes and even possible "direct action on the ground".
- 'Stuck in quagmire' -
The dynamic in the Syrian conflict shifted after Russia launched its air campaign on September 30, claiming it was targeting IS fighters.
Russian warplanes have struck 118 "terrorist" targets in Syria over the past 24 hours -- the highest total yet -- the defence ministry in Moscow said Wednesday.
Iran is believed to have sent thousands of troops and Hezbollah militia fighters to support Assad's forces.
But the US believes they will struggle to defend Assad for long, which is why they have been forced to engage diplomatically.
More than 250,000 people have been killed in Syria's brutal conflict since it began in March 2011 following a bloody crackdown on protests against Assad's rule.
Doctors Without Borders (MSF) said Thursday a "significant increase" in air strikes on Syrian hospitals since late September has killed at least 35 patients and medical staff and wounded 72.
"After more than four years of war, I remain flabbergasted at how international humanitarian law can be so easily flouted by all parties to this conflict," said Sylvain Groulx, MSF chief for Syria.