Morphological transformations for images with discrete values
Morphological operators are primarily defined over binary images where values 0,1 could correspond to areas identified as foreground and background respectively.

In your case, you can create a binary image of the region you want to apply the operator on by assigning the "islands" of noisy classified regions to the "foreground" (e.g 4  now becomes the value 1) and the broad area around them that is consistently classified in a stable way to the "foreground" (e.g value 3). Apply erosion to this image and then substitute the pixels back from the new binary image to the segmentation mask.

Alternatively, if you want to eliminate spurious classifications so that you end up with consistent areas, you could apply nonlinear filters such as median, max and min over some sliding m by n window.

For more information, please see this link and this link

As far as Python is concerned please see this or this and this.  Another excellent python module for this job is this one.

(Also, please note that you could apply any kind of filter that could even produce a Real type output (e.g 2.43, 1.76, etc) but either round or truncate its final output to make it to an integer. Not sure these operators would be very helpful though in this specific case.)

Hope this helps.