Conversation between freshman from senior high school and retired professor who suffers from Alzheimer's disease
One of the things that helps Alzheimer's patients the most is routines - especially routines from earlier parts of their lives (such as when he was a full-time faculty member). Keeping the brain active by thinking through number theory is also very helpful in staving off further progression of his dementia.

So if it was part of his routine when he was active faculty to go to lunch everyday in the dining hall and then to lecture, then you have now become a living part of his past and current world and helping him (and his family) stay functional.  As long as you are both enjoying it, there is no reason not to continue.

As gnometorule says, I think his daughter would be heartened to hear that 1) you enjoy this and 2) you would not have known he had Alzheimers had she not told you.  As gnome says, though, I would keep her apprised if you notice any changes.