How to improve accuracy in the following code? I have the about 43 different categories of traffic signs images data. If I am using the small data of 3 categories the maximum accuracy I am getting is around 65% and I have tried a lot of different layer values as well as losses and optimizers. When I am using the complete dataset I am only getting accuracy of about 5%. Please give some pointers on how to improve my accuracy. Please tell me should I add more layers and if somebody has any good example somewhere, please provide the link.

import cv2
import numpy as np
import os
import sys
import tensorflow as tf
import glob

from sklearn.model_selection import train_test_split

EPOCHS = 10
IMG_WIDTH = 30
IMG_HEIGHT = 30
NUM_CATEGORIES = 10
TEST_SIZE = 0.4     
path = []
data = []
label = []

def main():

    # Check command-line arguments
    if len(sys.argv) not in [2, 3]:
        sys.exit("Usage: python traffic.py data_directory [model.h5]")

    # Get image arrays and labels for all image files
    images, labels = load_data(sys.argv[1])

    # Split data into training and testing sets
    labels = tf.keras.utils.to_categorical(labels)
    x_train, x_test, y_train, y_test = train_test_split(
        np.array(images), np.array(labels), test_size=0.2
    )
    # Get a compiled neural network
    model = get_model()

    # Fit model on training data
    model.fit(x_train, y_train, epochs=EPOCHS)

    # Evaluate neural network performance
    model.evaluate(x_test,  y_test, verbose=2)

    # Save model to file
    if len(sys.argv) == 3:
        filename = sys.argv[2]
        model.save(filename)
        print(f"Model saved to {filename}.")


def load_data(data_dir):
    """
    Load image data from directory `data_dir`.

    Assume `data_dir` has one directory named after each category, numbered
    0 through NUM_CATEGORIES - 1. Inside each category directory will be some
    number of image files.

    Return tuple `(images, labels)`. `images` should be a list of all
    of the images in the data directory, where each image is formatted as a
    numpy ndarray with dimensions IMG_WIDTH x IMG_HEIGHT x 3. `labels` should
    be a list of integer labels, representing the categories for each of the
    corresponding `images`.
    """
    subdirs = [x[0] for x in os.walk('C:\\Users\\LENOVO\\traffic\\gtsrb')]
    for i in range(1,NUM_CATEGORIES+1):
        subdirs[i] = subdirs[i] + "\*.*"
    for i in range(1,NUM_CATEGORIES+1):
        for file in glob.glob(subdirs[i]):
            a = cv2.imread(file)
            a = cv2.resize(a,(IMG_WIDTH,IMG_HEIGHT))
            data.append(a)
            label.append(i-1)

    return(data,label)

def get_model():
    """
    Returns a compiled convolutional neural network model. Assume that the
    `input_shape` of the first layer is `(IMG_WIDTH, IMG_HEIGHT, 3)`.
    The output layer should have `NUM_CATEGORIES` units, one for each category.
    """
    model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(
        32, (3, 3), activation="relu", input_shape=(30,30,3)
    ),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.5),

    # Add an output layer with output units for all 10 digits
    tf.keras.layers.Dense(NUM_CATEGORIES, activation="softmax")])
    model.compile(
    optimizer="Nadam",
    loss="categorical_crossentropy",
    metrics=["accuracy"])
    return (model)
    raise NotImplementedError


if __name__ == "__main__":
    main()
Adding more images in the training set is one way to increase accuracy. You can also do transfer learning, i.e. using layers trained by a larger dataset if you are classifying common things such as animals which have networks that have already been trained such as the ResNet50.

When you have enough images that the accuracy does not increase anymore, you can try adding more layers, with convolutional and ReLu layers alternating. If you train for many epochs and the model starts to overfit i.e. training set accuracy still trends up but test set accuracy trends down, you can add layers that reduce overfitting, such as the dropout layer. You should first get the model to overfit and then try reducing overfitting.

If you don't have enough training data you could also do augmentation, e.g. adding rotated, shifted and horizontally flipped images to the training set.

The thing with traffic signs is that if you are classifying images with also the landscape instead of just the cropped traffic signs, you are going to have problems. You should then also detect the traffic signs (YOLO v3 could be used as transfer learning) instead of just classifying them.

Here's also a good example of traffic sign classification using transfer learning on Kaggle: https://www.kaggle.com/valentynsichkar/traffic-signs-classification-with-cnn