Using extracted feature vector to perform zero shot detection
The differentiation of shape starts in the initial layers. Colour information is exploited in further layers. Your network has less layers, try increasing your layers (keep filters low in the initial layers).
Also, if you are looking at similarity models, look at contrastive training or ArcFace loss or siamese type models. If you want to stick with classification networks, look at GeM pooling.
Best way would be to use MobileNet (or better networks such as EffNetB4) to finetune with GeM pooling.