Why has Hadoop failed to become popular?
I wouldn't say Hadoop failed to become popular rather I would say, it is still the base of any production big data system. 

Python or R are handy at the beginning when you just need to try out things but when it comes to putting things in production, Hadoop is the way to go. It does not directly provide any data scientist tools what it does provide is the base where data would be stored, processed and can be used to apply machine learning algorithms using Spark. 

So to summarize, I see Hadoop being used in the following contexts


For Data Storage in Production Systems
Hadoop YARN as a cluster manager for tools like Spark, Flink
If someone still wants to code in R/Python/SAS - Hadoop being used as backend. 


Hope this helps.