What does it mean when someone says "Most of the data science algorithms are optimization problems"
Most algorithms try to minimizes some objecive functions. 

For example, in linear regresssion, given $(x_i, y_i)$, we try to find $\hat{y}_i= \alpha_0 + \sum_{j=1}^d \alpha_j x_{i,j}$ and we we want it to be close to $y$. We try to minimize the mean square error in our estimation. 

That is our objective function is 
$$\min _\alpha\frac1n \sum_{i=1}^n (y_i-\alpha_o- \sum_{j=1}^d \alpha_j x_{i,j})^2.$$

We might have a model with unknown parameters and we can use maximum likelihood to find out our model, in that case, we maximize the likelihood function. Again, we get another optimization problem.

In general supervised classication error, we are trying to minimize the error which is typically formulated as a minimization of a loss function.  In SVM, we are trying to find a boundary that maximizes the margin between the two classes, again, we are trying to maximize an objective function.