All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
First of all, we welcome you in our site and we ask Allaah to guide you to Islam which is the true religion and the religion of all the Prophets and Messengers, may Allaah exalt their mention, as we clarified in Fataawa (plural of Fatwa) 88906 and 88051.
You should know that Zina (fornication or adultery) is forbidden in all the revealed Books and not only in Islam. Committing fornication is a grave major sin and an act of disobedience to Allaah which is forbidden in all the legislations which the Messengers, may Allaah exalt their mention, came with; all of which were revelations from Allaah. For more benefit, please refer to Fataawa 85254 and 88051.
As regards your inquiry as to how you could keep your relationship with that man, then you should know that you are able to keep your relationship with him in a way free from sins, and you are able to rid him from the feeling of guilt. You can be easily done by embracing Islam. Islam is the religion upon which 'Eesa (Jesus) and all the Prophets, may Allaah exalt their mention, were, and that there were glad tidings in the Torah and the Injeel (Gospel) about the coming of the Prophet Muhammad .
It seems that Allaah wanted good for you as He enabled you to visit our site so that we may clarify to you this reality which you properly do not know. The reference books from where many people take religion were forged by monks and priests to the point that you would nearly not find two identical copies.
Therefore, you should embrace Islam and be a true follower of 'Eesa (Jesus) and a follower of Muhammad, and all other Prophets, may Allaah exalt their mention.
You should save yourself from hellfire before the Day of Judgement when it is too late to repent.
If you become a Muslim, it is permissible for you to marry this man whom you love after he repents to Allaah as well.
Finally, we ask Allaah The Lord of The Mighty Throne to guide you to the truth and enable you to follow it and act upon it.