What exactly do people mean when they say that the computer "learns" to do something? I'm watching a youtube video where a guy is talking about how computers can learn to go from some point A to some point B.

However, the way he does it is very disappointing: all he does is generate thousands of objects that go from A to some random destination, and then he updates all the random destinations by taking the average of the original destination + the destination of the object that got closest to B. This process is then repeated until convergence.

I don't see what's so impressive about this. If you wanted the machine to go from A to B, why did you not just ... you know .... tell it to go to B?. What's the point of essentially generating random instructions, and hoping one of those random instructions just happens to be: go to B, when you could just as well tell that instruction to the computer yourself?

Is this really what people mean when they say that a computer "learns" to do some task? And if so, what's so impressive about it? What am I missing?
Your confusion is a result of Moravec’s Paradox; the just tell it to... step is enormously more difficult than it sounds. So what researchers do is attempt to find a general approach to that problem. The problem is expressed as a matrix of numbers (a grid that may be a million on each side these days) and “learning” is the process of computing the values for each of those numbers. Obviously no human can solve an equation like that, so a machine solver is used. Therefore the work of the AI researcher is to find better ways to generate these large matrices i.e. to express go to X and to create better solvers. The holy grail is a matrix and a solver that will work for any X.