What is the more efficient way of self-teaching, when the return is diminishing?
I can help you think about math and programming, at least. The key to both is to do a lot of exercises and get a lot of practice. In learning on your own, you need access to someone else who can give you feedback on your results. It is very hard to judge your own work and to draw the correct conclusions from mistakes or to overcome difficulties. 

Some online course structures may be fine for getting that feedback, but not all are. You need more than just the correct answers. You need to get a sense of why you went wrong in an exercise and how to avoid whatever that error was in the future. 

Reading and watching videos isn't sufficient and it will give you only shallow learning. Then, when you hit the next challenge, you aren't quite ready for it. If you have a good textbook (and there are bad ones as well) try to do every exercise. Try to get feedback. 

One way to get feedback, if you are working without a good guide, is to work in a group. You seemed to indicate that this was the case with you. You can each do a few exercises (the same ones for everyone) and then exchange work and discuss what you did and how. But even the feedback of other learners is better than none at all.