All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and Messenger.
Using magic is forbidden, if what your mother engaged in was indeed magic. In fact, it is among the major sins, as we have clarified in Fatwa 83703. However, if she did not know that what she was doing was a kind of magic, then she is not sinful. Allaah says (what means): {And there is no blame upon you for that in which you have erred but (only for) what your hearts intended. And ever is Allaah Forgiving and Merciful.} [Quran 33:5]
If she did this with the purpose of harming the family of your husband, then she is sinful for that and she is obligated to repent. To learn about the conditions of repentance, please refer to Fatwa 86527.
In principle, she should ask the pardon of the family of your husband, but since the consequences will most likely be unfavorable, it is better for her to just supplicate for them and ask Allaah to forgive them.
The illness that she was afflicted with may not have been caused by this book, and it is not necessarily a punishment for a sin; it may just be a test from Allaah. So she should seek treatment and consult specialists in the field. In any case, she should perform Ruqyah as it is advantageous for all kinds of illnesses. For more benefit, please refer to Fatwa 82918.
She is not obligated to pay any Kaffaarah (expiation) or give charity or the like, but if she gives charity, it would be good. The Prophet, sallallaahu ‘alayhi wa sallam, said: “Treat your patients with charity” [At-Tabaraani - Al-Albaani graded it Hasan (good)]
Allaah Knows best.