What is the difference between SVM and logistic regression?
Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule:

$$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$

Note that I am regarding the "primal", linear form of the SVM here.

In both cases the parameters $\mathbf{w}$ and $b$ are estimated by minimizing a certain function, and, as you correctly noted, the core difference between the models boils down to the use of different optimization objectives. For logistic regression:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i \log(1+e^{-z_i}),$$

where $z_i = y_if_{\mathbf{w},b}(\mathbf{x}_i)$.

For SVM:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i (1-z_i)_+ + \frac{1}{2C}\Vert \mathbf{w} \Vert^2$$

Note that the regularization term $\Vert \mathbf{w} \Vert^2$ may just as well be added to the logistic regression objective - this will result in regularized logistic regression.

You do not have to limit yourself to $\ell_2$-norm as the regularization term. Replace it with $\Vert \mathbf{w} \Vert_1$ in the SVM objective, and you will get $\ell_1$-SVM. Add both $\ell_1$ and $\ell_2$ regularizers to get the "elastic net regularization". In fact, feel free to pick your favourite loss, add your favourite regularizer, and voila - help yourself to a freshly baked machine learning algorithm.

This is not a coincidence. Any machine learning modeling problem can be phrased as the task of finding a probabilistic model $M$ which describes a given dataset $D$ sufficiently well. One general method for solving such a task is the technique of maximum a-posteriori (MAP) estimation, which suggests you should always choose the most probable model given the data:

$$M^* = \mathrm{argmax}_M P(M|D).$$

Using the Bayes rule and remembering that $P(D)$ is constant when the data is fixed:

\begin{align*}
\mathrm{argmax}_M P(M|D) &amp;= \mathrm{argmax}_M \frac{P(D|M)P(M)}{P(D)} \\
 &amp;= \mathrm{argmax}_M P(D|M)P(M) \\
&amp;= \mathrm{argmax}_M \log P(D|M)P(M) \\
&amp;= \mathrm{argmax}_M \log P(D|M) + \log P(M) \\
&amp;= \mathrm{argmin}_M (-\log P(D|M)) +  (-\log P(M))
\end{align*}

Observe how the loss turns out to be just another name for the (minus) log-likelihood of the data (under the chosen model) and the regularization penalty is the log-prior of the model. For example, the familiar $\ell_2$-penalty is just the minus logarithm of the Gaussian prior on the parameters:

$$ -\log\left((2\pi)^{-m/2}e^{-\frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2}\right) = \mathrm{const} + \frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2$$

Hence, another way to describe the difference between SVM and logistic regression (or any other model), is that these two postulate different probabilitic models for the data. In logistic regression the data likelihood is given via the Bernoulli distribution (with $p$=sigmoid), while the model prior is uniform (or simply ignored). In SVM the data likelihood is modeled via some $\mathrm{exp}(-\mathrm{hinge})$ distribution (not sure it even has a name, but I hope you get the idea that undoing the minus-logarithm would always bring you back to $P(D|M)$, up to a constant), and the model prior is the Gaussian.

In practice, the two models have different properties, of course. For example, SVM has sparse dual representations, which makes it possible to kernelize it efficiently. Logistic regression, on the other hand, is usually well-calibrated (which is not the case with SVM). Hence, you choose the model based on your needs (or, if you are unsure, on whatever cross-validation tells you).