"We are facing many obstacles, mostly snipers and car bombs," said one CTS fighter, First Lieutenant Bashar Hussein, from a position in Dhubbat neighbourhood, just south of Hoz.
The terrain allows a small number of determined men to hold off a larger force.
Iraqi soldiers were around 500 metres (yards) away from the compound on Thursday and had only inched a little closer by Friday.
"Daesh (IS) resistance got stiffer as Iraqi forces moved closer to the government compound," said an army brigadier general, speaking to AFP on condition of anonymity.
"Our forces are now just over 300 metres away from those buildings," he said.
The number of IS fighters still holding out in the city was estimated at fewer than 400, with reports of some retreating from the front by using civilians as human shields.
"Operations to liberate Ramadi need time. It isn't easy to retake it quickly," said Ibrahim al-Fahdawi, the head of the security committee for nearby Khaldiya district.
"Booby-trapped houses, suicide attacks, improvised explosive devices, snipers, mortars, rockets: Daesh (IS) is using everything it's got to stop the progress of the security forces," said Fahdawi.
- Trapped civilians -
IS posted several statements online over the past two days claiming to have inflicted serious losses in multiple attacks across Ramadi.
Iraqi security officials only admitted to limited casualties and said they were able to repulse suicide car bomb attacks.
According to two military sources, at least three government fighters were killed and 13 wounded Friday during the fighting in Hoz.
The loss of Ramadi, which lies about 100 kilometres (60 miles) west of Baghdad, was Baghdad's most stinging defeat in the war against IS since the jihadists took over a third of the country last year.
A key component of IS's victory in May was the use of dozens of massive suicide car bombs to blitz government positions but, more than six months on, the security forces came prepared.
"They've tried but they've had less success for several reasons," said Colonel Steve Warren, spokesman for the US-led coalition that has been carrying out daily air strikes.
"Both Iraqis and the coalition have become much better at identifying these threats earlier on," he said.
"The Iraqis have been equipped with AT4 shoulder-fired anti-armour rockets, so the coalition has trained them on how to deploy these weapons, provided them with 5,000 of them and the Iraqis have used them to great effect," Warren said.
Also slowing the security forces' advance was the presence of civilians trapped in their homes.
Dozens of families that had stayed in central Ramadi, many of them prevented from leaving by IS when the security forces launched their big push on Tuesday, have managed to flee.
But Fahdawi said dozens remained trapped, mostly in the neighbourhoods of Al-Thaylah and Al-Jamaiyah.
"Daesh detained all the men and left the women and children in their homes... It may be because they want to prevent a revolt against them from what's left of the male population," he said.
The operation to retake Ramadi started months ago with Iraqi forces cutting IS supply lines to parts of Anbar before gradually closing in on the city, taking key bridges, roads and positions one after the other.
Retaking the city would provide a welcome morale boost to the much-criticised military.
It would also isolate the IS bastion of Fallujah, further down the Euphrates valley on the way to Baghdad, and continue to shrink the "caliphate" IS proclaimed last year.
Besides Fallujah, IS still controls Iraq's second city of Mosul, around which preparations for an offensive to retake are still at a very early stage.
Iraqi Prime Minister Haider al-Abadi said in a Christmas message that "retaking our beloved Mosul will be achieved with the cooperation and unity of all Iraqis after victory in the city of Ramadi."