Yes, Transformers are probably close to the Pareto frontier in terms of efficiency if you want to train something that has a sequence as an input.

But they're not inherently *special*. There's a bunch of other model types around that pareto frontier. Transformers are just good at saturating memory bandwidth, which is the hardware frontier.