How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation?
What you want to do is called multi-task learning. Here's what you do:

Create a second Input.
Attach it to 1D CNN (2-3 layers), so it aggregates this tabular information.
Concatenate this feature with the intermediate feature generated by the U-Net using Concatenate layer.
Put a dense layer of 2 after this.
Put softmax with units = number of classes.
Add CE loss calculated with this one and the ground labels to the loss of U-Net.

This is in regards to TensorFlow. The same can be done in PyTorch easily.