At small batches, a chip uses at most 1 to 7% of its math units

At the small batches that keep replies fast, a chip uses at most 1 to 7% of its math units

Upper bound on math-unit use while generating text, by batch size (16-bit weights). Derived from vendor specifications: 16-bit dense peak divided by memory bandwidth, weight traffic only (paper, Appendix A1).