"My supposition is, somebody knew" about bin Laden's presence in the garrison town of Abbottabad, near Islamabad, Gates told a news conference.
But there was no evidence that leaders in Islamabad were aware of the Al-Qaeda chief's whereabouts before the US raid this month on bin Laden's compound, he said.
And some evidence suggested that the leadership definitely was not aware, he said.
"It's my supposition, I think it's a supposition shared by a number in this government, that somebody had to know, but we have no idea who and no proof and no evidence."
Gates said he shared the "frustration" felt by US lawmakers towards Pakistan but stressed that President Barack Obama's administration could not make allegations without evidence.
"It's hard to go to them with an accusation when we have no proof that anybody knew."
Gates said the aftermath of the raid presented an "opportunity," with Pakistan pledging to take more action.
"The Pakistanis, over the last couple of weeks, have expressed the view that they are willing to go after some of these people and that we should not repeat the bin Laden operation, because... they will undertake this themselves," he said.
The Pentagon chief, in his first press conference since bin Laden was killed on May 2, argued against punishing Pakistan by suspending aid as Islamabad had suffered a blow as a result of the covert raid.
"I think we have to proceed with some caution," Gates said. "My own view is we need to continue the assistance that we have provided that benefits the Pakistani people.
"If I were in Pakistani shoes, I've already paid a price. I've been humiliated, I've been shown the Americans can come in here and do this with impunity," he said.
Mullen, chairman of the US Joint Chiefs of Staff, said it would take time to learn who may have known about bin Laden in Pakistan as intelligence agencies are still poring over a large amount of material found at the Al-Qaeda leader's compound.
The four-star admiral said it was important to maintain strong ties with Pakistan and that his counterpart and "friend," army chief General Ashfaq Kayani, had pledged to pursue the Haqqani militant network -- blamed for launching attacks on US forces in neighboring Afghanistan.
"I think one of the issues that is just a challenge for us is our clock moves a lot faster than his (Kayani's) clock," Mullen said.
"That has been the case so far, and I think it will be the case in the future. I'm not trying to give him an excuse, but matching those clocks has been pretty difficult."
Lawmakers in Congress have voiced dismay that bin Laden was hiding in a military town not far from the Pakistani capital and have urged the Obama administration to review security aid to the country.
Pakistan received a total of $2.7 billion dollars in aid and reimbursements from Washington in fiscal year 2010, which ended on October 1, making it the third-largest recipient of US aid after Afghanistan and Israel.
Finding bin Laden in a Pakistani garrison town "indicates, at a minimum, a lack of commitment by the Pakistani military to aggressive cooperation with the United States," a group of Democratic senators wrote in a letter this week to Gates and Secretary of State Hillary Clinton.
-------------------------------------------------------------------------------------