All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His Slave and Messenger.
You are not required to repeat the Ghusl because you merely doubt the impurity of the towel. In principle, things are pure, not impure, particularly since the towel is not originally used to remove impurities.
However, supposing that the towel was dirty, still not every dirt is considered impure. Furthermore, if we suppose that the towel is impure, it does not invalidate the obligatory Ghusl, such as Ghusl after Janaabah (sexual defilement) or menstruation. It is only obligatory to wash the impurity which gets on the body if it is tangible and can be transferred from the towel to the body. If what is on the towel is just the ruling of physical impurity, then it is not transferred (unlike the ruling of ritual impurity, i.e. if one becomes ritually impure, then all his body takes the same ruling).