Number of Iterations (Gradient Descent)
You can use validation sets in training. They will be a better indicator for the testing phase of your model performance than only the training error. Also, your learning_rate is quite small, which can cause the model to converge in large number of iterations. You can play with those parameters to see how it effects your training.
I am learning about the Gradient Descent Algorithm and I implemented one such**(in python)** over the Boston Housing data set(in sklearn). I have implemented a cost function as well which takes in all the coefficients, the intercept and the training data and returns the cost for those coefficients and intercept. I ran the algorithm over the Boston data set for 1500 iterations and learning_rate = 0.000003202, and It converged successfully, giving the least cost as 61.840725406571245, but when I trained the sklearn's LinearRegression() algorithm over the same training data, and found the cost using .coef_ and .intercept_. It gave the least cost as 19.958219814238042. So My question is How can one say the a certain number of iterations, will give me the least/optimal cost? Like say I run the algorithm for n times, and obtain a particular cost, then How will I know that this cost is good enough or do I need to reduce it further?