All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger. We ask Allaah to exalt his mention as well as that of his family and all his companions.
First of all, you have to know that adultery is a great major sin and Allaah warned us against it as He says (which means): {And do not approach unlawful sexual intercourse. Indeed, it is a Faahishah (i.e. anything that transgresses its limits; a great sin), and an evil way (that leads one to Hell unless Allaah forgives him.}[Quran 17:32]. Therefore, it is an obligation on you to sincerely repent to Allaah and avoid any means that could lead you to committing the same sin again.
If a person commits adultery or fornication, he has to conceal his sin. The Prophet said: "Whoever happens to commit any of these filthy things, should conceal himself/herself as Allaah did not expose him." [Al-Bayhaqi] Besides, it is not permissible for your wife to ask you to admit that you have committed adultery and you are not permitted to inform her even if she asks you about it. So you may use for instance Tawriyah (saying something which has more than one meaning and intending a meaning different from what the listener is likely to understand), as this is better than openly lying, and if necessary it would even be permissible to lie as it is Islamicly permissible to lie to one's wife (in certain conditions). Imaam Muslim reported that Um Kalthoom Bint 'Uqbah said: "I have not heard the Prophet permitting telling lies in anything except in three situations: war, reconciling between people, and the talking of the husband to his wife and of the wife to her husband."
Some scholars even permitted swearing about something that is a lie if it is to achieve a considerable religious benefit or to repel a considerable harm.
Therefore, there is no harm on you for swearing while putting your hand on that book while your wife believes that you had put your hand on the Quran. There is no harm on you as well for swearing to a lie if you did so to achieve a religious benefit and you do not have to expiate for it.
Allaah Knows best.