A video shows how the woman pulls him over, refusing to let go and then Francis slaps her hand away with an angry look on his face.
The incident prompted nasty comments on social networks.
Francis apologized for the slip on Wednesday. "Sometimes we lose patience; I apologize for my bad example," he said at the Angelus prayer after New Year's Day Mass.
It is not the first time that the pontiff has refused physical contact: In March, he pulled his hand away from several people during an audience in Italy because they wanted to kiss his ring.
The Vatican later said that Francis did this for hygiene reasons because he wanted to prevent the spread of germs.
----------------------------------------------------------------------------------------