All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
If the mental problem from which your mother suffers has reached the extent of making her unconscious of what she says and does, then she is no longer competent for religious assignment and, accordingly, her oath is not valid nor binding and has no implications. Asna Al-Mataalib Fi Sharh Rawdh At-Taalib (4/273) reads: “An oath made by a child, an insane person, or someone who was forced to make it is not binding because their statement is not valid according to the Sharee‘ah.” [End of quote]
However, if her mental problem does not reach being unconscious of what she says or does, then her oath is valid and binding, and if she took the medicine that she swore not to take, she has broken her oath and it is incumbent on her to offer the due expiation for breaking an oath. You mentioned that you forced her to take the medicine; if you mean by forcing her that you merely kept urging and persuading her to take it until she did, this is not considered coercion according to the Sharee‘ah as long as she was able to abstain. In this case, such coercion does not change the fact that she broke her oath, nor does it waive the due expiation for which she is liable, and it does not make you liable to pay the expiation instead of her. But if you mean that you literally forced her to take the medicine against her will, her oath is not considered broken in this case. Mataalib Uli An-Nuha Fi Sharh Ghaayat Al-Muntaha (6/369) reads: “If he (the oath-maker) did what he swore not to do under duress, it does not count as breaking an oath; when one makes an oath not to enter a house, for example, and is forcibly carried inside it, he has not broken his oath, because the actions one performs under duress are not to be attributed to him (he is not considered their doer).” [End of quote]
Allah Knows best.