All perfect praise be to Allah, the Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad, sallallahu ‘alayhi wa sallam, is His slave and Messenger.
Performing jihaad by guarding the Muslims is called Ribaat. Ribaat means guarding the frontiers and securing the borders of the Muslim lands in order to block the attacks of the disbelievers. The Kuwaiti Encyclopedia of Fiqh reads, "Ribaat means staying in a place beyond which there are no Muslim lands, where the enemies of Muslims are expected to attack, with the intention of warding off the enemy for the sake of Allah."
Apparently, guarding masjids does not fall under Ribaat, even if the masjids are in non-Muslim lands. However, if the guard's intention is to seek the reward of Allah and help the Muslims to perform their worship safely, he shall earn great rewards, Allah willing. Verily, Allah does not allow the rewards of the doers of good to be lost. If it is feared that the masjid and the worshipers might be harmed, it is permissible for the guard to miss the congregational prayer in order to carry out his guard duty. If another congregational prayer would be held after the main one ends, he should wait until the second congregational prayer is held and join it and assign someone else to guard the masjid, if that is possible; otherwise, he bears no sin for performing the prayer individually.
It is hoped that he would earn the rewards of performing the prayer in congregation if he missed it with the sincere intention of caring for the interests of the Muslims, whether it be the Taraaweeh prayer or any other congregational prayer, Allah willing.
Allah knows best.