I have a (very nice and professional) colleague who is from the same religion as me, we are the only two from our religion in the department and that maybe explains why he would enjoy me being his company and would like me to practice the religion with him.

We come from the same origins, same native language & accent, same race.. etc. However, the problem is that I am not religious, at least not as much as he is. For ex: I do not pray on time, and I do not visit the mosque.. etc.

Usually, on Fridays, he approaches me and invites me to go to the mosque with him to do the Friday Practices (equivalent to Church's Sunday Practices for Christians). 

The workplace has no problem with employees going for their religious activities, and this colleague never showed attitude if he knew I did not pray that day, or attend that Friday.. etc. 

With that being said, he invites me every Friday to go with him. Even that I am not into going, every time he invites me I feel embarrassed I am not "doing the right thing" so I end up going with him. It seems like he is "killing me with being nice".

How can I stop this from happening, without telling him **no** or/and look like I am the "bad guy"?

**EDIT:**

Adding what @LawrencePayne pointed out: It is considered, at least in our culture, impolite to NOT invite your Muslim peer to the prayer.