What can be inferred and how common is that a potential PhD supervisor talks directly to my recommender?
Sounds like a polite chat. Professors are humans. They go to a conference and meet all kinds of people they don't know well and they need a topic to talk about, and a common acquaintance is a good topic for small talk. It is a bit tricky in the sense that there is an application involved here, but overall I would not over-interpret this.

If you want to interpret this, then you can see this as a slightly positive sign: (S)he is also signaling that (s)he is interested in supervising you. That helps: if one committee member says that (s)he wants to supervise a potential students, then there needs to be a good reason not to go along with that (e.g. lack of funding).