All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ʻalayhi wa sallam, is His slave and Messenger.
Firstly, we commend your zeal for the religion of Allaah and for its sanctity; this is an indication of your piousness and goodness. We implore Allaah to reward you generously for that. It should be noted that declaring a Muslim to be an apostate is a serious issue and should not be taken lightly; not everyone has the knowledge of the actions and words which take the Muslim out of the fold of Islam and which require declaring a Muslim to be an apostate. In fact, a Muslim may commit major sins and other misdeeds or even an act of disbelief that takes the doer out of the fold of Islam, but he is not to be declared a non-Muslim in the absence of one of the due conditions for Takfeer (declaring a Muslim group or individual as non-Muslim) or in the presence of an impediment to it. Please refer to Fatwa 8106.
Hence, we say that if your brother's wife has actually apostatized, then their marriage contract is void and she is no longer his lawful wife. Accordingly, it is impermissible for him to have conjugal relations with her; if he does, it is considered fornication. Please refer to Fatwa 134052 about apostasy and its effects on marriage.
You did well by offering sincere advice to your brother and his wife concerning this. We advise you to continue advising them. It is also better to seek the help of reliable scholars and virtuous people to whom they would most likely listen; hopefully, they will turn back to Allaah and repent from their sins.
It is prescribed for the Muslim to shun a persistent sinner; however, if shunning such a person will most likely not be of any avail, then it is advisable not to shun him but to try to win him over. Indeed, the matter revolves around the general good. Please refer to Fatwa 86090.
As for obedience to your mother, it is not unconditional or unrestricted. Children are obliged to obey their parents as long as there is a legitimate benefit in obedience to them and it does not incur considerable harm to them (children); please refer to Fatwa 84942.
Allaah Knows best.