Since this is @OpenBMB crackedness awareness month: missed that the team had released a dense model with activation sparsity back in the summer (also sparsified a few others). I think sparsification is more interesting, inference time wise, than MoEs. Would like to see more.