Implementing U-Net segmentation model without padding
$\hspace{3cm}$
If we follow the definition of each arrow.
Gray =&gt; Copy and Crop

Every step in the expansive path consists of an upsampling of the
feature map followed by a 2x2 convolution (“up-convolution”) that halves the
number of feature channels, a concatenation with the correspondingly cropped
feature map from the contracting path, and two 3x3 convolutions, each followed by a ReLU. The cropping is necessary due to the loss of border pixels in
every convolution. Paper

So, believe(I have added 3 coloured circles)

Blue - 28x28 is upsampled and become 56x56, 1024 is halved to 512
Red - 64x64 is cropped to 56x56. Then Concatenated along FM axis.
Black - 3x3 convolutions, followed by a ReLU