CNN Multi-Class Model Only Predicts 1 class for all test images
73 millions trainable parms

- When using Transfer learning we first freeze the base model
- Train it till you reach good accuracy
- Then unfreeze it and train for just few epochs. Keep LR small
Other probable issues -
- If your labels are not One-Hot coded, please use sparse_categorical_crossentropy
- Add validation_split in fit method
- Suggest you add a keras.layers.GlobalAveragePooling2D after the base model and before flattening it

&quot;setting include_top=False: this excludes the global average pooling layer and the dense output layer&quot;

Can use this code as guidance
base_model = keras.applications.xception.Xception(weights=&quot;imagenet&quot;, include_top=False)
model = keras.layers.GlobalAveragePooling2D()(base_model.output)
output = keras.layers.Dense(n_classes, activation=&quot;softmax&quot;)(model)
model = keras.Model(inputs=base_model.input, outputs=output)

for layer in base_model.layers:
    layer.trainable = False

optimizer = keras.optimizers.SGD(lr=0.2, momentum=0.9, decay=0.01)
model.compile(loss=&quot;sparse_categorical_crossentropy&quot;, optimizer=optimizer, metrics=[&quot;accuracy&quot;])
history = model.fit(train_set, epochs=15, validation_data=valid_set)

for layer in base_model.layers:
    layer.trainable = True

optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, decay=0.001)
model.compile(loss=&quot;sparse_categorical_crossentropy&quot;, optimizer=optimizer, metrics=[&quot;accuracy&quot;])
history = model.fit(train_set, epochs=5, validation_data=valid_set)

Code Ref - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition