Applying a keras model working with greyscale images to RGB images I followed this basic classification TensorFlow tutorial using the Fashion MNIST dataset.
The training set contains 60,000 28x28 pixels greyscale images, split into 10 classes (trouser, pullover, shoe, etc...).
The tutorial uses a simple model:

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])


This model reaches 91% accuracy after 10 epochs.

I am now practicing with another dataset called CIFAR-10, which consists of 50,000 32*32 pixels RGB images, also split into 10 classes (frog, horse, boat, etc...). 

Considering that both the Fashion MNIST and CIFAR-10 datasets are pretty similar in terms of number of images and image size and that they have the same number of classes, I naively tried training a similar model, simply adjusting the input shape:

  model = keras.Sequential([
     keras.layers.Flatten(input_shape=(32, 32, 3)),
     keras.layers.Dense(128, activation='relu'),
     keras.layers.Dense(10)
  ])


Alas, after 10 epochs, the model reaches an accuracy of 45%. What am I doing wrong?

I am aware that I have thrice as many samples in an RGB image than in a grayscale image, so I tried increasing the number of epochs as well as the size of the intermediate dense layer, but to no avail.



Below is my full code:

import tensorflow as tf
import IPython.display as display
from PIL import Image
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import pdb
import pathlib
import os
from tensorflow.keras import layers #Needed to make the model
from tensorflow.keras import datasets, layers, models

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

IMG_HEIGHT = 32
IMG_WIDTH = 32

class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']


train_images = train_images / 255.0
test_images = test_images / 255.0

def make_model():
      model = keras.Sequential([
         keras.layers.Flatten(input_shape=(IMG_HEIGHT, IMG_WIDTH, 3)),
         keras.layers.Dense(512, activation='relu'),
         keras.layers.Dense(10)
      ])
      model.compile(optimizer='adam',
                   loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                   metrics=['accuracy'])
      return model

model=make_model()
history = model.fit(train_images, train_labels, epochs=10)
Your model is not sufficiently complex to adequately classify the CIFAR 10 data set. CIFAR-10 is considerably more complex than the  Fashion-MNIST data set and therefore you need a more complex model.You can add more hidden layers to your model to achieve this. You should also add DROPOUT layers to prevent over fitting. Perhaps the easiest solution is to use transfer learning. I would recommend using the MobileNet CNN model if you want to try transfer learning. Documentation for that can be found here. Since CIFAR-10 has 50,000 sample images I do not think you will need data augmentation. First try a more complex model without augmentation and see what accuracy you achieve. If it is not adequate then use the keras ImageData Generator to provide data augmentation. Documentation for that is here.