The Taliban had broken off contact with the Americans last year and has refused to negotiate with Kabul, but spokesman Zabiullah Mujahid told AFP the office was intended "to open dialogue between the Taliban and the world".
With US officials telling reporters that contact would be established in the coming days, President Barack Obama called the talks an "important first step" but warned of a long and bumpy road ahead.
"An Afghan-led and Afghan-owned peace process is the best way to end the violence and to ensure lasting stability in Afghanistan and the region," Obama said on the sidelines of the Group of Eight summit in Northern Ireland.
Obama insisted the Taliban would have to renounce ties to Al-Qaeda, halt violence and commit to the protection of women and minorities, and warned that US-led NATO forces remain "fully committed" to battling Al-Qaeda.
The Taliban, which was driven from power in the wake of the September 11, 2001 attacks by US-backed rebels, has since mounted a guerrilla war against the Afghan government and maintains rear bases in Pakistan.
Pakistan, which has battled its own Islamist insurgency but is widely seen as quietly backing the Afghan Taliban, said it welcomed the peace talks.
UN Secretary General Ban Ki-moon also welcomed the talks, with a spokesman saying dialogue is "the only way to end the violence."
A huge US-led NATO army has proved incapable of defeating the insurgency and is concentrating on training Afghan government troops to take charge when international forces end combat operations in the course of next year.
In parallel, US envoys are attempting to establish a dialogue with the main Taliban faction, in the hope of convincing it to repudiate ties to the Al-Qaeda extremist network and to reach a political deal with Karzai.
A US official said US and Taliban envoys would meet in Doha "in a couple of days," after which the Taliban would meet with a "High Peace Council" set up by Karzai to conduct the negotiations.
In opening its mission, the Taliban did not explicitly renounce Al-Qaeda, which prior to 9/11 and the US intervention had bases in Afghanistan, but it did vow not to allow attacks to be launched from Afghan soil.
This proved a sufficient first step for US officials, and State Department spokeswoman Jen Psaki said that an eventual disavowal of Al-Qaeda ties by the Taliban was only an "end goal of the process."
Psaki could not say when the meeting would take place, but said Ambassador James Dobbins, US special representative for Afghanistan and Pakistan, would leave Washington later Tuesday bound for Doha via Ankara.
The divided Afghan insurgency could complicate talks, amid US doubts as to whether the powerful "Haqqani network" of warlord Jalaluddin Haqqani, a former CIA asset turned Al-Qaeda ally, was ready to embrace negotiation.
US officials said the Taliban envoys in Doha had been authorized to talk by Mullah Omar, the main Taliban figurehead, and that Haqqani's group is "a fully subordinate part of the overall insurgency."
But General Joseph Dunford, the US commander in Afghanistan, said he was skeptical the Haqqanis would back a peace deal.
"All I've seen of the Haqqanis would make it hard for me to believe they were reconcilable," he told reporters by video link from Kabul.
Officials said this week's ground-breaking meeting would amount to "an exchange of agendas," followed by another within about two weeks.
The Taliban said it supports a "political and peaceful solution that ends Afghanistan's occupation and guarantees the Islamic system and nationwide security."
Karzai, who has long called for peace talks, said he had ordered government envoys to travel to Qatar to try to open negotiations.
He pledged that Afghan forces were ready to take on the insurgents, but the enduring threat was underlined when a bomb targeting a lawmaker killed three people just before the security handover ceremony Karzai attended.
The turnover of the last districts from NATO to Afghan control included areas in the south and east where the Taliban are at their strongest.
Doubts remain over the ability of Afghan forces to maintain security, and the 98,000 foreign troops still in Afghanistan will retain an important function in training, logistics, air support and in combat emergencies.
"The reality is Afghan forces are not dreadful, but they're probably not sufficiently capable to drive the war to a conclusion," Stephen Biddle, professor of international affairs at George Washington University, told AFP.
"My guess is they will be able to maintain the stalemate, provided the US pays their bills."