How do neural networks learn concepts?
Computers don't learn - they compute.  They are given a set of instructions and generally follow those instructions to the letter, with the occasional blue screen of death.

The computer uses an algorithm that is developed to solve a particular problem.  The algorithm will iterate on a problem using the criteria and input data it is given.  As the algorithm progresses it's internal state is modified in some way, which you could say it is "learning", based on however it is processing of the data.

So in the case of learning how to do basic math - the network would have to be given the input of the numbers and operations, it would need to be "trained" on how to use them, and then given data to operate on. 

The problem with that is... computers are really, really good at doing basic math - it's pretty much what they are designed for doing, but they are not good a "learning" things, because deep down they are just build to move the ones and zeros around really fast.

What could be more useful is to train the network to try and understand if a question is a math question or not, e.g "what is one plus one" vs "is that a banana over there", and/or do natural language processing on it to determine the logic of the problem, resolved as a trivial problem for a computer "1+1=?".  

i.e. the actual math itself is the easy bit for the computer - processing understanding how you are phrasing a question is what the network would need to do.  

And the answer of course is 42.