Neural network for MNIST: very low accuracy
What loss function are you using? It looks to me like you're using squared error loss (am I right?). This might work, but consider using cross entropy loss instead, which is more well suited for classification problems.

Also, by using the logistic function as activation function in the last layer, you're treating the problem as ten separate binary classification problems. While this also might work, since the problem is a multi-class classification problem, you should probably change the activation function in the last layer to softmax.

One error I spot in your code is that you're not taking into account the derivative of the nonlinearity in the last layer. To fix this, you should change

delta3 = x3 - Y[:,j];


to

delta3 = (x3 - Y[:,j]) .* sigmoidGradient(z3);


similar to the way you calculate delta2.