What is the point of getting rid of overfitting?
Basically, prediction error can be decomposed into three terms, two of which that you can control. MSE (mean square error) = Variance + Bias^2 + irreducible error. A model with high variance implies that small changes in the training dataset yield large changes in the fit of the model (in this context). A model with low bias essentially fits the training dataset extremely well. Irreducible error is the difficulty inherent to the problem itself (and therefore, not controllable) so lets ignore that.

In an ideal world, we would obviously want low variance AND low bias. However, this in reality is extremely difficult to do because intuitively, the two forces of variance and bias work against each other. The reason why is that I can arbitrarily make my model fit the data extremely well (and therefore, have low bias) by simply increasing the complexity of the model, modelling every quirk and essentially do what you refer to as "memorizing the dataset". You see, data are realizations of random variables that can potentially take on many different outcomes and therefore, there is noise in the dataset. Hence, modelling the exact pattern/trend in the dataset is not going to generalize well because the data itself is not a perfect representation of the population (which is our goal) but a sample of it.

By increasing the complexity of my model to arbitrarily improve the fit of the model on the dataset, I have effectively traded away all of my bias for large amounts of variance. I did this because clearly, by increasing the complexity of my model to the point where my model has even memorized the noise in the dataset, a relatively small change in the dataset with different observations (another random sample from the same population) will change the fitted model drastically. This is because noise is inherently random, not predictable and unique to that specific dataset (sample) only, not the general population from which the data comes from. As a consequence of my model changing a lot with my dataset that I trained it with (what many refer to as an unstable model), my models predictions will inherently be very random as well and therefore, my models performance on new held out data will be no better than say, a random guesser. This is what people mean when they say their model is "overfit". I've fit the training data set perfectly, at the cost of not modelling the actual signal from the data but a bunch of noise unique to the training set only.

So what people have figured out is that we need to make a tradeoff between fitting the training data extremely well whilst also being mindful of not modelling noise. Thus, stuff like dropout layers in NN, L1/L2 regularization, pruning in trees, learning rates, early stopping, and cost complexity penalties all work in purposely sacking some bias (fitting the training set not as well) in hopes of reducing variance (having a stable model and therefore, stable non random predictions). Striking the balance between the two is in essence what predictive modelling is about and all ML methods try to address this problem in their own way.

In your example, removing overfitting would imply moving the validation accuracy closer to the training accuracy, not the other way around. But of course, this is typically a pipe dream and you will almost always overfit somewhat. The degree to what you find acceptable, given the problem at hand, is key. If I have a training accuracy of 0.99 and a validation accuracy of 0.89 in many contexts I would still be satisfied with the model, for example.