baseline (experts zero)                    56.0 ms   saves   0.0 ms   runs [56.0, 55.8, 56.8]
  linear_attn (GDN, 36 layers)               34.1 ms   saves  21.9 ms   runs [34.1, 34.2, 34.1]
  self_attn (12 layers, incl. indexer)       49.1 ms   saves   6.8 ms   runs [49.3, 49.0, 49.1]
  hyper-connections (97)                     41.4 ms   saves  14.5 ms   runs [41.5, 41.3, 41.4]
  shared expert + gate (48)                  53.4 ms   saves   2.6 ms   runs [53.4, 54.1, 53.4]
  router (48)                                53.8 ms   saves   2.1 ms   runs [53.8, 54.9, 53.8]
  ple (n-gram embedding)                     54.9 ms   saves   1.0 ms   runs [54.9, 55.0, 54.9]
  lm_head                                    50.3 ms   saves   5.7 ms   runs [50.3, 50.3, 50.2]
  ALL of the above                            3.3 ms   saves  52.7 ms   runs [3.2, 3.4, 3.3]