Logistic regression program does not give statistically correct result I wrote a very simple and compact logistic regression program using theano. I am initialising my data randomly, and I have restricted the number of training steps to 1, so since the weights are also random, I should expect the result to follow a standard distribution. I have ten classes and 10 initial examples, so I am expecting my accuracy to be close to 10, most of the time, however I am often (too often) getting results that are quite far from 10 (like 0 or 49). I should expect to get those results sometimes, but not so often. Is there something wrong in my code?

import theano
import theano.tensor as T
import numpy as np
rng = np.random
#rng = np.random.RandomState(0)

ninputs = 100
nfeatures = 784
noutputs = 10
nhiddens = 50

#training rate
tr_rate = 0.1 
training_steps = 1

D = (rng.randn(ninputs, nfeatures), rng.randint(size=ninputs, low=0, high=noutputs))

# Feed-forward
x = T.dmatrix('x')
t = T.ivector('t')

w = theano.shared(rng.randn(nfeatures, noutputs), name="w")
b = theano.shared(np.full(noutputs, 0.01), name="b")
sigma = T.nnet.softmax(T.dot(x,w) + b) 
prediction = T.argmax(sigma, axis=1)   


parameters = [w, b]

xent = -T.mean(T.log(sigma)[T.arange(t.shape[0]), t])          # Cost function

cost = xent.mean() + 0.01 * (w ** 2).sum()                     # Regularisation 

gparameters = [T.grad(cost, param) for param in parameters]    # Compute the gradient of the cost 

pars = zip(parameters, gparameters)

train = theano.function(
          inputs=[x,t],
          outputs=[xent],
          updates = [(param, param - tr_rate * gparam) for param, gparam in zip(parameters, gparameters)],
          allow_input_downcast = True )


predict = theano.function(inputs=[x], outputs=prediction)

# Train
for i in range(training_steps):
    train(D[0], D[1])


result = predict(D[0]) - D[1]

N = ninputs

error = 0
for index in result:
    if result[index] != 0:
        error += 1

correct_guesses = N - error
accuracy = (N - error)*100/N

print
print "correct predictions = %f over %i examples" % (correct_guesses, N)
print "accuracy = %i%%" % accuracy
If you initialized the weights randomly and predicted immediately, you should probably expect 1/num_classes accuracy. 

But you are training first. The data may be random, but it is still possible to overfit it. With a sample that small and wide, you could imagine some degree of separability of the classes by chance. In that scenario, your network can learn weights that are better than random (far from perfect in only one iteration).

The effect would likely be lessened by using much more data or holding out/using new random data to test accuracy.