How to say "Don't count on me" in Cantonese?
In your situation, the most colloquial and natural expression that I can think of is "我嚟得就嚟” （If I can come, I will.) The speaker is not committing, nor is s/he ruling out coming. 

Another expression closer to "don't count on me/ don't rely on me (to do something, not necessarily joining the speaker for lunch)" is "唔好旨意嗮我。”