What is this plastic part on caliper brakes called?
With the information provided by Criggie's comment I was able to find an exploded diagram of the Shimano Exage brake with a part list.

That part is called (descriptively) a "Sleeve". It's part number 83E 0900




Searching for the part number reveals that Shimano uses that same part number on several different brakes.

I asked The Google as nicely as I could but found no replacement sleeves or brake rebuild kits available.
You are looking at:


Making your own per Criggie's comment
Scavenging sleeves from another caliper
Replacing the caliper
Living with the creaking sound
Duct tape?