All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
If this person gives you back your money, then take it from him. In which case, it is permissible for you to use it or spend it in charity on poor and needy people.
It is obvious from your saying "I give you this amount as Sadaqah from my children" that you did not give it out of your own will. Moreover, the person you mentioned is not eligible to receive the money of Zakaah, as you said.
But if you had intended from your statement to give him the money as a gift, then you should execute that and not take it back since the Prophet prohibited taking gifts back.
Allah knows best.