All perfect praise be to Allah, the Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad, sallallahu ‘alayhi wa sallam, is His slave and Messenger.
If the client retracted in matters which he was lenient on with you, you have the right to do the same and retract your leniency with him and claim your rights.
If these people have delegated you to spend their money on the construction and then denied this delegation and claimed their money back, they have no right to such money. If you do not have evidence indicating that they have delegated you and you cannot obtain your right except by means of this trick, then some scholars held that it is allowable for the Muslim to take his own rights via trickery provided that he must not take anything beyond his own right and there must not be any fear of grave evil consequences.
However, other scholars held that it is prohibited to do this and relied on the following hadeeth: Abu Hurayrah, may Allah be pleased with him, reported that the Prophet, sallallahu ‘alayhi wa sallam, said, "Fulfill the trust for the one who entrusted you, and do not cheat the one who cheated you." [At-Tirmithi and Abu Daawood]
This view is more prudent and safer in terms of clearing oneself of their liability.
In any case, we advise you to be more cautious in the future and to document the contracts and agreements and have witnesses for them.
Allah knows best.