Simply establishing the existence of the prison has been difficult. Rebels and doctors refused to say where prisoners are taken.
Then, a fighter who laid down his arms after the battle for Tripoli was essentially won on Tuesday, called a friend who mentioned a prison to him.
The friend said we could come to this prison, in a small town where a revolutionary committee now holds sway, pending the establishment of a centralised government in Tripoli and the drawing up of a new constitution.
The prison's location cannot be revealed because "we do not know what Kadhafi's people are still capable of," said Yacoub Amar Mohammed, who heads the town's legal committee charged with investigating the prisoners' alleged crimes.
He says that 371 prisoners are yet to be investigated, 30 have been freed and four have been deemed chargeable.
"From the first day we have tried to find a legal system and avoid what the ex-regime did for 42 years," said Mohammed. "We are trying to establish standards of human rights and to follow them in the new, free Libya."
Asked if conditions elsewhere are as good as here, or if he is concerned about rebels exacting revenge on prisoners, as alleged by rights groups, he said, "you cannot always control personal behaviour, there may be one or two cases."
He said that some of those freed had asked to come back to prison for their own safety "because they did horrible things to the Libyan people."
"Tripoli is 95 percent free, so we will wait for the National Transitional Council to start work and give a legal structure so that they can get the sentence they deserve. We have strong evidence against many of the people here."
Some in the West have voiced concern that fundamentalist Sharia Islamic law may be established in Libya, but Mohammed says that "our Islam is simple and modern."
"The rebels are young people who like the West and support European football teams, they are moderate and want to build a new Libya, worshipping God and feeling free, with respect for other people."
"We are trying to avoid the negatives, of which there are many after 42 years outside of the modern world, so we have to catch up. The negatives will soon be gone, through education."
A man who gave his name as Dr Khayree pointed out that, conveniently, even Kadhafi-built elementary schools "look like prisons."
Steel mesh covers all the windows, and doors inside have bars. "Even the hospital looks like a military installation," sneered Khayree.
The prisoners are mostly locals, "so we try to give them the best," said Mohammed. They will be moved to a proper prison in two days and cannot be photographed "because of the Geneva Convention," he said.
Rebels' detention facilities are not so well organised at the former headquarters of the feared Istakhbarat secret police, in Tripoli.
Rebel guard Abdullah, sporting the latest freedom fighter fashion of a revolutionary flag tattooed on his arm, took us down a dark corridor lined with cells.
One hot room held two men who said they were from Mali, construction workers who were detained while looking for food. Sharing the relatively clean cell with mattresses and dates on the floor was a hysterical Libyan man.
Abdullah said the Malians would probably be released soon. However, he had doubts about the Libyan.
Behind the bars of a cell opposite, two Libyan men paced around. Asked if they were okay, they said they were fine, thanks to God.
They were detained while sniping for Kadhafi two days ago, but they laid down their arms and surrendered. Abdullah said they also would be released, unwilling conscripts into Kadhafi's cruel and now defunct regime.