Ankara, which hosts a string of Syrian refugee camps near its border, reacted furiously to the Syrian attack, which claimed a woman and three children among the five victims.
NATO, in an emergency evening meeting at its Brussels headquarters, condemned the attack against member-state Turkey and called on Syria to respect international law.
The incident was the most serious between the neighbours since June when Syria shot down a Turkish fighter plane with the loss of its two-man crew and Ankara took the matter to NATO.
The same month, two Syrians in Turkey were killed by cross-border bullets.
Turkey's Prime Minister Recep Tayyip Erdogan said after the shells hit Wednesday, "our armed forces in the border region immediately retaliated against this heinous attack... by shelling targets spotted by radar.
"Turkey will never leave unanswered such provocations by the Syrian regime targeting our national security, in line with engagement rules and international law," his office said in a statement.
NATO said after its meeting that it "continues to stand by Turkey and demands the immediate cessation of such aggressive acts against an Ally and urges the Syrian regime to end flagrant violations of international law."
The alliance said Syria's "recent aggressive acts at NATO's southeastern border" were a "flagrant breach of international law and a clear and present danger" to an ally.
"The most recent shelling... which caused the death of five Turkish citizens and injured many, constitutes a cause of greatest concern and is strongly condemned by all Allies."
US Secretary of State Hillary Clinton said "we are outraged that the Syrians have been shooting across the border. We are very regretful about the loss of life on the Turkish side."
She added it was a "very, very dangerous" situation, and said she would be discussing with her Turkish counterpart "what the best way forward would be."
Pentagon spokesman George Little said: "This is yet another example of the depraved behavior of the Syrian regime, and why it must go."
UN chief Ban Ki-moon told Damascus to respect its neighbours' sovereignty.
France issued a swift condemnation and affirmed its full support and "solidarity" with its Turkish ally, the foreign ministry said.
British Deputy Prime Minister Nick Clegg said:
“We condemn all violence by the Syrian regime and demand that it avoids any repetition of today’s incident on the border with Turkey."
Relations between Syria and Turkey have steadily deteriorated since the uprising against Syrian President Bashar al-Assad's regime first broke out in March 2011.
Akcakale town has been hit by Syrian fire several times recently and schools have been shut over the past two weeks because of the violence.
A week ago, the Turkish army deployed artillery and anti-aircraft missiles along the border in the region.
Regional governor Celalettin Guvenc told Anatolia news agency that "three or four Syrian shells fell inside Turkey at around 4:30 pm local time... and one of them hit a house. As a result, we unfortunately lost our five citizens."
Ten others were wounded and two of them were in critical condition, he said.
Activists in the Syria conflict said Turkey hit back at Syrian army positions.
"Syrian tanks in Raqa province shelled Tal al-Abyad on the (Syrian side of the) border, and some of the shells fell on Turkish territory," a Raqa-based activist who identified himself only as Wael told AFP via the Internet.
"The Turks then fired shells in response," he added. His account was corroborated by another activist and the Syrian Observatory for Human Rights.
"The last incident went too far," Deputy Prime Minister Bulent Arinc was quoted as saying by Anatolia.
"NATO treaty articles bring about certain liabilities when one of its members is attacked.
"We will not be blinded by rage, but we will protect our rights to the end in the face of an attack on our soil killing our people."
Syria was looking into the origin of the deadly cross-border shelling, its Information Omran Zoabi said in a statement reported by state television.
"Syria offers it sincere condolences to the families of the victims and to our friends the Turkish people," he added.
The minister reiterated his government's charges that it is battling foreign-backed "terrorists," whom it has frequently accused Ankara of training and financing.
Since late July, Syrian rebels have seized control of at least three key border crossings with Turkey and others on the eastern border with Iraq.
Turkey, once a strong ally of Syria, broke with the regime after it began cracking down on dissent, in a conflict which monitoring groups say has claimed more than 31,000 lives.
------------------------------------------------------------------------------------------