Aso's Liberal Democratic Party (LDP) won 38 seats while its coalition partner New Komeito gained 23 in Sunday's vote for the 127-strong assembly in the capital, leaving the ruling bloc three seats short of the 64 needed to secure a majority, election officials said.
The main opposition Democratic Party of Japan (DPJ) won 54 seats, meaning that the LDP was no longer the assembly's largest party, the officials said.
The remaining 12 seats went to other opposition and independent candidates.
Before the vote, the ruling coalition had 70 seats against 34 for the DPJ.
Aso's coalition acknowledged the apparent bloody nose from voters in the sprawling capital.
"We have to sincerely face the severe judgement from Tokyo residents," said senior LDP official Nobuteru Ishihara. "We are afraid that this impact on a general election will be quite big."
Opposition leader Yukio Hatoyama said his party, which controls the upper house, would submit a motion of no-confidence against Aso in parliament "as soon as possible".
"We will urge (Aso) to ask for the people's voice by dissolving the lower house and calling a general election," he said.
"With this result, Tokyo residents said 'No' to the LDP and New Komeito," said Naoto Kan, deputy president of the DPJ.
Opinion polls suggest the LDP, having lost the upper house in 2007, is also likely to lose its large majority in the lower house to the DPJ in the nationwide vote, triggering a major shift in Japan's political landscape.
"With the defeat, the chances are increasing that the LDP will lose power in the general election," said Yoshinobu Yamamoto, a professor of politics at Aoyama Gakuin University in Tokyo.
"Pressure on Aso to resign will grow but I doubt a change in leadership would help the LDP a lot," he said.
Since taking power in 1955, the LDP has ruled Japan continuously except for one 10-month stretch, guiding the nation from the post-war gloom through its "economic miracle" to become the world's second-biggest economy.
But it has badly stumbled since the popular premier Junichiro Koizumi stepped down in 2006.
Aso, Japan's third prime minister since Koizumi, took office in September with a mission to revive the LDP ahead of general elections.
But the 68-year-old premier's approval ratings have slumped over a series of ministerial resignations, policy flip-flops and concern about his handling of the economy during Japan's worst recession since World War II.
"This Tokyo election is the beginning of the path to a power change," said Tomoko Amatsuji, a 70-year-old office worker who voted for an opposition candidate. "I'm not satisfied with the government. It's time to see a change."
While he may face fresh calls from some of his own lawmakers to step down if his party does badly in the Tokyo vote, Aso has indicated he intends to fight on.
"There is no direct link between the Tokyo assembly election and national politics," he was quoted by a government spokesman as saying on Saturday.
Voter turnout was estimated at 54.49 percent, up from 43.99 percent in the previous vote, according to electoral officials. Some 16.6 million people were eligible to vote.
------------------------------------------------------------------------------------------------------------------------