Tactics to avoid feeling overwhelmed by machine learning Short version: despite lots of reading, machine learning still feels like being a monkey in the dark. Any advice?
For background, I'm a researcher in computer science, in a field non-related to machine learning.
I have been trying to get more proficient in machine learning*, yet no matter how much I read and fiddle with code/toy datasets, when I try to go to a harder problem, I always feel overwhelmed by the choices I need to make:

I have to choose the algorithm: This is the part I typically  find the most straightforward;
For said algorithm, I have to choose the objective function :  usually, many are applicable, and I find it difficult to gain a good intuition of what makes an objective function adapted in some cases rather than others, apart from the very classical ones for linear or logistic regression
And then, I should devise the features: this still feels completely arcane to me, apart from using content-based features readily available in the data.

I am under the impression that I have to &quot;create&quot; the tailored algorithm and the data.
Concerning the algorithm, I have spent some time into studying gradient boosting and the math behind it, to the point that I have a reasonably solid comprehension of how it works, and an intuition of parameter tuning for simple datasets. However, that knowledge does not generalize.
How are these issues typically approached? Are there any resources that can help?
* By taking the Machine Learning  Coursera course and its more in-depth version, reading more XgBoost-specific material (on its internals and parameter tuning and intuition), as well as playing with the Titanic dataset, and a housing market dataset.
We have to climb up a steep learning curve when we learn about machine learning. Your question is quite general: One of the tactics I use when learning is divide and conquer. Get some coarse overview about the whole area, then pick some particular area and dig deeper only there.
Perhaps the question is too general, the best tactic may vary and depend on the area you address. 
But I am not sure if learning the math is always helpful (although it may always be interesting for those who care). 

The algorithms can often be applied in a black box approach, and it may be sometimes not necessary to understand an algorithm in math terms (white box), but sufficient to know it's function, strengths and weaknesses (black box).  

You may be the first one that tests that algorithm for the domain, so pure experimentation is useful in the end.