Metal glue that cures in coldish weather?
Just about any 2 part epoxy would meet your requirements. However, gluing both sides is asking for trouble in the future.
Couple of ways around this. You could use loctite blue (thanks @jwh20 for steering us away from red) on the grub screws, which would be reversible but more secure than your current setup. You could lightly dimple the axle with a drill bit so that the grub screw is more secure. Or both. (If it was me, I'd just use the loctite.)