to be quite honest, i don't know what the big deal is. they not only certainly trained on benchmarks in a way, which are basically worthless at this point (see what garbage top the leaderboards. but i don't think their training cost is to be trusted. most likely they're mum on the costs due to export control cirumvention. which is understandable. inference has always been pretty cheap. the training costs require something else.  gpt-4 was a moe, too. i dunno, i haven't really read the paper. i'm far more interested in byte-level llms and other things to reduce the training costs and the amount of data needed. that's where the real innovation is happening in the us, not some slight tweaks to their training costs even if you believe them. which is what this is, at best.
United States