I have a blog post coming in a few days on this topic.
But basically, I agree, architecture doesn't really matter. The number of parameters and the training data matter.
The tradeoff transformers make is that you saturate the hardware really efficiently (so: easy to parallelize & scale) but the tradeoff is the n^2 scaling of the attention mechanism.
This turns out to be a great tradeoff if what you're doing is scaling models to 7B+ parameters.
Uszkoreit has mentioned that the global/quadratic attention was (paraphrasing) considered as overkill, but the brute force parallelism this simple approach allowed made that irrelevant.
But of course that changes when you scale up the context size enough, and the fix is simple since the key insight of the architecture was the hierarchical tree-like nature of language and thus dependence mostly on local (within branch) context, not global context. In Google's Big Bird attention (from their Elmo/Bert Muppet era!) they basically use sliding window local attention, but augmented with a fixed number of global tokens with global attention, and some random attention to further back non-local tokens. This mixture of attention patterns performs almost as well as global attention. Part of the reason (aside from the mostly local nature of language) is that as you ascend the transformer-layer hierarchy, receptive field sizes increase (same as they do in a CNN), so even with the attention gaps of random attention, there is still visibility at higher layers.
But basically, I agree, architecture doesn't really matter. The number of parameters and the training data matter.
The tradeoff transformers make is that you saturate the hardware really efficiently (so: easy to parallelize & scale) but the tradeoff is the n^2 scaling of the attention mechanism.
This turns out to be a great tradeoff if what you're doing is scaling models to 7B+ parameters.