All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His Slave and Messenger.
The example you mentioned clearly indicates that the wife accepted both to return the dowry and to renounce some of her financial rights on the husband as compensation. If this is the case, it is obligatory on the wife to fulfill what they agreed upon and forfeit these rights. There is no consideration for what she intended. For further information about the definition of Khul‘, please refer to Fatwa 89039.
However, if the husband took the dowry and uttered divorce on condition that he would demand her to pay the rest and she did not accept that condition, there would be no consideration for that (condition). The husband would not be entitled to anything except the compensation he took in return for uttering divorce because nothing would be obligatory on the wife except the things she made obligatory on herself.
In any case, it is better to take matters of dispute to the Sharee‘ah courts or its substitutes such as the Islamic authorities, in case Sharee‘ah courts are not available.
Allaah Knows best.