What about the part where they went around CUDA to improve memory efficiency in the H800s used in the $5 million training run?

Does that mean CUDA is a bottleneck? #IDK 

High level languages are great but 50x greater efficiency is a big deal
#IDK