How to politely reject a request for a letter of recommendation for a professor
If you want to decline to write, then it would be reasonable to reply to the person who requested the letter, saying something like: "My interactions with Professor X were three to four years ago and involved very little interaction with him beyond attendance at two of his classes.  So I do not consider myself to be in a position to provide useful information for award Y."