There are times I meet someone for a brief time, maybe a couple hours, and their personality impresses me enough to create a fictional character about them. Obviously I am adding my own attributes to the fictional character, and I don't know them well enough to write about many of the real person's attributes, but the original impression on me will be central to their character. Is it illegal to create a character this way, where the character development is loosely based on a real person this way?

When I am writing I make sure the connection is loose enough that the person could not be identified. I guess my main concern is if someone would read my book (which is a work in progress) and be like "oh I remember seeing this author for a couple of hours and this fictional person has a similar personality to me" Could they sue me?