What happens when we train a linear SVM on non-linearly separable data?
I think basic support vector machine means hard-margin SVM. So, let's review:
What is a Hard-Margin SVM
In short, we want to find a hyperplane with the largest margin which be able to separate all observations correctly in our training sample space.
The optimisation problem in hard-margin SVM
Given the above definition, what is the optimisation problem which we need to solve?

The largest margin hyperplane: We want max(margin)
Be able to separate all observations correctly: We need to optimise margin and also satisfy the constraint: No in-sample errors

What happens when we train a linear SVM on non-linearly separable data?
Back to your question, since you mentioned the training data set is not linearly separable, by using hard-margin SVM without feature transformations, it's impossible to find any hyperplane which satisfies &quot;No in-sample errors&quot;.
Normally, we solve SVM optimisation problem by Quadratic Programming, because it can do optimisation tasks with constraints. If you use Gradient Descent or other optimisation algorithms which without satisfying the constraints of hard-margin SVM, you should still get a result, but that's not a hard-margin SVM hyperplane.
By the way, with non-linearly separable data, usually we choose

hard-margin SVM + feature transformations
use soft-margin SVM directly (In practical, soft-margin SVM usually get good results)