Why does only Adam optimiser perform well in my case? I am trying to classify texts into categories (one text can have multiple categories), to do this I used one-hot encoded labels (with sklearn.preprocessing.MultiLbaleBinarizer()). My texts are also one-hot encoded but with keras.preprocessing.text.one_hot.

This is my code :

csv_data_name = 'learning_base'
df = pd.read_csv('../../datasets/csv/' + csv_data_name + '.csv ')
df['label_list'] = np.array(df.labels.str.split('|'))
df = df.drop(columns=['labels'])


# Remove incomplete lines (lines without labels)
df = df.dropna()

encoder = MultiLabelBinarizer()
labels_binary = encoder.fit_transform(df.label_list)
output_dim = labels_binary.shape[1]

vocab_size = 200000
encoded_docs = [one_hot(d, vocab_size) for d in df.text]
max_length = 200
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

# Data division
training_set_data, testing_set_data_whole, training_set_target, testing_set_target_whole = train_test_split(
    padded_docs,
    labels_binary,
    test_size=0.1,
    random_state=0,
    shuffle=False)


model = Sequential()
model.add(Embedding(vocab_size, 200, input_length=max_length))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(output_dim, activation='sigmoid'))

model.compile(optimizer=TFOptimizer(tf.train.AdamOptimizer()), loss='mean_squared_error',
              metrics=['acc'])

model.fit(training_set_data, training_set_target, epochs=50, verbose=True, batch_size=200,
          callbacks=[EarlyStopping(monitor='acc', patience=3, verbose=True), tensorboard])


Everything works fine except when I try another optimizer the learning seems incorrect, only Adam seems to work as intended.

This is my results :

|Optimizer|Time by epoch|Strict classification succes|Precision|Recall|epoch|
|-------- | ----------- |--------------------------- | ------- |------|---|
|GradientDescent                          |00:21|0%|12.5%|0%|3 (ES)|
|GradientDescent (`learning_rate = 0.01`) |00:21|0%|3.9%|16.1%|7 (ES)|
|Adadelta                                 |00:24|0%|3.1%|25.7%|3 (ES)|
|Adagrad                                  |00:26|0%|/|/|13 (ES)|
|Ftrl                                     |00:27|0%|/|/|8 (ES)|
|Momentum (`momentum = 0.01`)             |00:22|0%|/|/|18 (ES)|
|Adam (`learning_rate = 0.01` : default)  |02:37|53%|80%|60.54%|13 (ES)|


And here is the tensorboard for accuracy :
There could be a couple of possible reasons:


One reason could be the Adam optimizer is a combination of several other
optimization techniques (e.g., momentum and running average of
gradient squares). The combination of those techniques works well on
multi-label text classification.
Another reason could be that multi-label text classification is a sparse
problem. The Adam optimizer in TensorFlow has a sparse
implementation.