Mathematical foundations of the ability to learn
Computational learning theory (CLT or COLT) is devoted to the mathematical and computational analysis of machine learning algorithms, so it is concerned with the learnability (i.e. generalization, bounds, efficiency, etc.) of certain tasks, given a learner (or a learning algorithm), a hypothesis space, data, etc.
CLT can be divided into (at least) two subfields: statistical learning theory (SLT) and algorithmic learning theory (ALT). The most famous CLT frameworks are PAC learning and the VC theory (which extends PAC learning to infinite-dimensional hypothesis spaces).
The following are some good resources you can use to get started with CLT (but you can find more here).

The book Machine Learning (1997) by Mitchell
The book Foundations of Machine Learning (2012) by Mohri et al.
The book Understanding Machine Learning: From Theory to Algorithms (2014) by Shalev-Shwartz et al.
The paper An overview of statistical learning theory (1999) by Vapnik (who is one of the main contributors to SLT)
The paper Introduction to Statistical Learning Theory (2014) by Bousquet et al.

Here's a related question on this site: What sort of mathematical problems are there in AI that people are working on?.