"I just couldn't bear it."
The deal to evacuate the towns is the latest in a string of such agreements through Syria's six-year civil war.
They have been touted by the government as the best way to end the fighting but rebels say they are forced out by siege and bombardment.
Critics say deals are permanently changing the ethnic and religious map, but in an exclusive interview with AFP this week President Bashar al-Assad insisted the evacuations were only temporary and people would return once the "terrorists" had been defeated.
At least 80 buses left Fuaa and Kafraya in Idlib province in the northwest, an AFP correspondent in rebel-held territory said.
They arrived at a marshalling point in Rashidin, west of second city Aleppo, followed by 20 ambulances.
The Syrian Observatory for Human Rights said 5,000 people had left the two towns, including 1,300 pro-government fighters.
- 'Madaya cried today' -
The Observatory said 2,200 people from Zabadani and Madaya had left, among them 400 rebels.
Madaya resident Amjad al-Maleh, speaking from a departing bus, told AFP that rebels among the evacuees had been allowed to keep light weapons.
"It is a very bad feeling when you see those who besieged you and killed you with hunger and bombardment right in front of you," Maleh said.
"Madaya cried today -- the ones who stayed and the ones who left."
Late Friday, the buses from Madaya arrived in Rashidin, where the evacuees from Fuaa and Kefraya were waiting, according to a resident aboard one of the buses.
The Observatory said the Fuaa and Kafraya buses were expected to head to government-held Aleppo city, while the Madaya evacuees were to head to Idlib.
More than 30,000 people are expected to be evacuated under the deal, which began on Wednesday with an exchange of prisoners.
All 16,000 residents of Fuaa and Kafraya are expected to leave, heading to Aleppo, the coastal province of Latakia, or Damascus.
Civilians from Madaya and Zabadani will reportedly be allowed to remain if they choose. Those who opt to leave will head to rebel-held territory in Idlib.
The four towns are party to a longstanding agreement reached in 2015 that requires aid deliveries and evacuations to be carried out simultaneously.
But access has been limited, with food and medical shortages causing malnutrition, illness and even death among besieged residents.
The UN says 4.72 million Syrians are in hard-to-reach areas, including 600,000 people under siege, mostly by the Syrian army, but also by rebels or the Islamic State group.
- 'They're going back' -
There has been a series of evacuations in recent month, mostly around the capital Damascus but also from the last rebel-held district of Syria's third city Homs.
The rebels have charged that Assad's regime is deliberately forcing civilians to leave to alter the country's sectarian map.
But in Wednesday's interview, the president told AFP that it was the rebels who were driving people from their homes.
"We wish that everyone could stay in his village and his city, but those people like many other civilians in different areas were surrounded and besieged by the terrorists, and they've been killed on (a) daily basis, so they had to leave," he said.
"But of course they're going to go back to their cities after the liberation."
Assad is facing renewed international pressure after accusations his government carried out a suspected April 4 chemical weapons attack on a northern rebel-held town.
He told AFP that reports of the attack were a "fabrication."
French Foreign Minister Jean-Marc Ayrault on Friday dismissed Assad's comments as "100 percent lies and propaganda."
The Organisation for the Prohibition of Chemical Weapons is probing the incident, but Russian Foreign Minister Sergei Lavrov on Friday criticised the watchdog for failing to go to the site.
"We consider it unacceptable to analyse events from a distance," he said after meeting his Syrian and Iranian counterparts.
"They say still that it is not very safe, but they cannot put forward convincing arguments," Lavrov said.