High train and val results. Bad test and predict results
I figured it out.

First: I was making the labels the wrong way.

Instead of this: 

train_labels = to_categorical(np.array(
[0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2)), 4)


do this:

train_labels = to_categorical(training_generator.classes, 4)


The first snippet wrongly generated a 4-ary array, where only the first and second element got a 1 assigned.

Second:

I found out my method of evaluation was wrong. I tried to use model.evaluate_generator which yielded the wrong results. I think it was because I tried to feed the model raw images, instead of creating bottleneck features from the raw test images and then feeding them to the model.

so instead of this:

from keras.models import load_model
conv_base = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))
loaded_model = load_model('bottleneck_fc_model.h5')

model = Sequential()
model.add(conv_base)
model.add(loaded_model)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

test_datagen = ImageDataGenerator(rescale=1./255)


test_generator = test_datagen.flow_from_directory(
        '/content/output/test',
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical')

test_loss, test_acc = model.evaluate_generator(test_generator, 216 // batch_size)
print('test acc:', test_acc)
print('test loss:', test_loss)


I did this (Which worked):

from keras.models import load_model
model = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))

bottleneck_features_test = model.predict_generator(
    test_generator, 261 // batch_size)

loaded_model = load_model('bottleneck_fc_model.h5')

model = Sequential()
model.add(loaded_model)

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

test_datagen = ImageDataGenerator(rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    '/content/output/test',
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical',
    classes = folders,
    shuffle=False)

bottleneck_features_test = conv_base.predict_generator(
    test_generator, 261 // batch_size)

test_labels = to_categorical(test_generator.classes, 4)

(test_eval_loss, test_eval_accuracy) = model.evaluate(  
   bottleneck_features_test[:256], test_labels[:256], batch_size=batch_size, verbose=1)

print("[INFO] test accuracy: {:.2f}%".format(test_eval_accuracy * 100))  
print("[INFO] test Loss: {}".format(test_eval_loss)) 


I hope this helps someone who's struggling with the same!