Relationship between train and test error I have some specific questions for which I could not extract answers from books. Therefore, I ask for help here and shall be extremely grateful for an intuitive explanation if possible.  

In general, neural networks have a bias/variance tradeoff and thus we need to have a regularizer. Higher bias --> underfitting; Higher Variance--->overfitting. 

To solve overfitting, we use regularization for contraining the weight. This is a hyperparameter and should be learned during training based on my understanding using cross-validation. Thus, the dataset is split into a train, validation and test set. The test set is independent and is unseen by the model during learning, but we have the labels available for it. We usually report the statistics such as false positives, confusion matrix, misclassification based on this test set.

Q1) Is this bias/variance problem encountered in other algorithms such as SVM, LSTM etc as well? 

In convolutional neural network (Matlab toolbox) I have not seen any option for specifying the regularization constant. So, does this mean that CNN's don't need a regularizer?

Q2) What is the condition if training error and test error are both zero? Is this the ideal best situation?

Q3) What is the condition if training error > test error?

Q4) What is the condition if training error > validation error?

Please correct me where wrong. Thank you very much.
First of all be very clear with the use of the Training set, Validation set and  Testing set. These play a crucial part in  tuning your DL model. Usually, a validation dataset is used for keeping a check over the model during the training. An intutive observations are noted with the training validation and testing accuracy during data fitting:


If the model is having high validation accuracy and low training accuracy, it is an underfit model.
If the model has higher training accuracy and low validation accuracy, it is overfit.


Bias-variance trade-off problem is a central problem of supervised machine learning algorithms.

The bias–variance decomposition is a way of analyzing a learning algorithm's expected generalization error with respect to a particular problem as a sum of three terms, the bias, variance, and a quantity called the irreducible error, resulting from noise in the problem itself. 

Ref