face landmark detection cnn loss not converging in tensorflow
Use a standard pretrained network like ResNet, Effnet or even VGG16 for feature extraction and put a regression layer in the end.
Now, talking about your network. FIrst of all, the image size is not square, make it! Don't use even valued kernels (3 and 5 are mostly used). Number of filters increase with depth, yours don't follow a pattern.
Lastly, 100 images are nothing for these behemoths, give it more data.
Better stick to a pretrained model.