There are many resources online about how to implement MLP in tensorflow, and most of the samples do work :) But I am interested in a particular one, that I learned from https://www.coursera.org/learn/machine-learning. In which, it uses a cost function defined as follow:$J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K} \left[ -y_k^{(i)} \log((h_\theta(x^{(i)}))_k - (1 - y_k^{(i)}) \log(1 - (h_\theta(x^{(i)}))_k \right]$$h_\theta$ is the sigmoid function.And there's my implementation:# one hidden layer MLPx = tf.placeholder(tf.float32, shape=[None, 784])y = tf.placeholder(tf.float32, shape=[None, 10])W_h1 = tf.Variable(tf.random_normal([784, 512]))h1 = tf.nn.sigmoid(tf.matmul(x, W_h1))W_out = tf.Variable(tf.random_normal([512, 10]))y_ = tf.matmul(h1, W_out)# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)cross_entropy = tf.reduce_sum(- y * tf.log(y_) - (1 - y) * tf.log(1 - y_), 1)loss = tf.reduce_mean(cross_entropy)train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))# trainwith tf.Session() as s:    s.run(tf.initialize_all_variables())    for i in range(10000):        batch_x, batch_y = mnist.train.next_batch(100)        s.run(train_step, feed_dict={x: batch_x, y: batch_y})        if i % 100 == 0:            train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})            print('step {0}, training accuracy {1}'.format(i, train_accuracy))I think the definition for the layers are correct, but the problem is in the cross_entropy. If I use the first one, the one got commented out, the model converges quickly; but if I use the 2nd one, which I think/hope is the translation of the previous equation, the model won't converge.
Implement MLP in tensorflow
machine learning;tensorflow
You made three mistakes:You omitted the offset terms before the nonlinear transformations (variables b_1 and b_out). This increases the representative power of the neural network.You omitted the softmax transformation at the top layer. This makes the output a probability distributions, so you can calculate the cross-entropy, which is the usual cost function for classification.You used the binary form of the cross-entropy when you should have used the multi-class form.When I run this I get accuracies over 90%:import tensorflow as tffrom tensorflow.examples.tutorials.mnist import input_datamnist = input_data.read_data_sets('/tmp/MNIST_data', one_hot=True)x = tf.placeholder(tf.float32, shape=[None, 784])y = tf.placeholder(tf.float32, shape=[None, 10])W_h1 = tf.Variable(tf.random_normal([784, 512]))b_1 = tf.Variable(tf.random_normal([512]))h1 = tf.nn.sigmoid(tf.matmul(x, W_h1) + b_1)W_out = tf.Variable(tf.random_normal([512, 10]))b_out = tf.Variable(tf.random_normal([10]))y_ = tf.nn.softmax(tf.matmul(h1, W_out) + b_out)# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)cross_entropy = tf.reduce_sum(- y * tf.log(y_), 1)loss = tf.reduce_mean(cross_entropy)train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))# trainwith tf.Session() as s:    s.run(tf.initialize_all_variables())    for i in range(10000):        batch_x, batch_y = mnist.train.next_batch(100)        s.run(train_step, feed_dict={x: batch_x, y: batch_y})        if i % 1000 == 0:            train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})            print('step {0}, training accuracy {1}'.format(i, train_accuracy))