Replacing bearings in pivot links
"Bore" or "bearing bore" are the words I know for it. Yes, you're right to worry about this, especially if it's actually more or less loose in there and not just a slip fit. One possible negative consequence of the fit being loose is the bore could get deformed and/or ovalized until there's knocking or play you can feel while riding as a result. It could also creak. It probably also doesn't help bearing life, and I'm sure there are other problems that could result from this depending on the factors at work. If it's more like a slip fit, where it's somewhat snug but can be pushed in and out by hand, that's fairly common and while not ideal, it's unlikely to do much more than creak, although having it develop into something worse isn't impossible.

The common fix for this is to use Loctite 609 or equivalent, which is designed for this kind of application. It expands to create a tight fit, but generally not so much you'll hate life too much when you have to get it apart. I wouldn't be optimistic about the medium-strength threadlockers like 243 being strong enough here.