Practical limitations of machine learning
How does one even start describing these practical limitations?

Most authoritative source on the main practical (and theoretical) problems related to machine learning is formal framework describing it. Mathematics.

Right here we have a problem, even tough you look at all of these ML algorithms and you conclude its math, we do not explain the whole process of a ML problem. I cant find a citation but Bengio or someone said that computer science is hard science because you have mathematical underpinning for everything, and ML is more of a soft science where you learn by trying, but only after you tried. (Not always the case ofcourse, thats why everyone is researching ML now to give it some structure)

Take a simple neural network. You know its matrix multiplication, backpropagation bla bla. Great. But what are the topological properties of certain architecture, convergence criteria, what functions can it approximate. Some of these questions are known and/or are being researched. Lets look at this one for example: What functions can we guarantee (in a formal sense) that a NN can approximate and under what conditions?

Well we can express a task as an optimization one. And in order to converge to optimal solution, under certain constraints, we need to satisfy certain assumptions.

Regarding DNN (deep neural networks) and mathematical theory behind it , convergence assurance is given with famous Universal Approximation theorem that states that every smooth function can be estimated given enough parameters.

Caveat just because we can do it in theory does not mean its possible. For example approximating a function that generates random numbers would require infinite recources







But what about non-smooth functions (such as Time-series) one?

Well the TL;DR of the DNNS FOR NON-SMOOTH FUNCTIONS is that for a special set of piecewise smooth functions "convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions"

What is piecewise smooth function? function whose domain can be partitioned locally into finitely many "pieces" relative on which smoothness holds, and continuity holds across the joins of the pieces.



Ok but WHY can a DNN approximate these types of functions?

" The most notable fact is that DNNs can approximate non-smooth functions with a small number of parameters, due to activation functions and multi-layer structures. A combination of two ReLU functions can approximate step functions, and a composition of the step functions in a combination of other parts of the network can easily express smooth functions restricted to pieces. In contrast, even though the other methods have the universal approximation property, they require a larger number of parameters to approximate non-smooth structures"

Conclusion there is a mathematical theory that insures approximations of a set of certain non-smooth functions using DNN. So if we have non-smooth function that satisfies these constraints, we can find an optimal architecture and get optimal convergance rates.

Concluding your question There are best practices that constantly evolve and you can get a check-list that just isnt relevant (Take computer vision problems, check list 2 years ago isnt same as today). BUT what is constant and remains the best authority is formal under pinnings such as mathematics. It can tell you directly the "best practice" of when it would be futile to even try to approximate a function.