All perfect praise be to Allaah, The Lord of the worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and messenger.
If your father demanded the money as charity from this man or if this man told your father that the money he gave to him is provided by a Zakah committee or charitable organization, then this man has no right to consider the money as a debt that your father owes him or ask him to pay it.
However, if your father just asked the man for money without clearly stating that he needed it as charity and the man did not tell him that it was paid by a Zakah committee or charitable organization then, in principle, this money is a debt that your father owes to this man and is required to pay it off whenever he is able to do so.