If you believe in consciousness divorced from flesh, you're in strong intellectual company, but that's not a path I can go down. Metaphysically, I just can't accept the idea that the mind is anything but the computation performed by the brain.
I rather agree with you, but the interaction of the mind with the physical reality is also extremely important to shape it. GPT3 has no interaction with a physical world. Any formal system that cannot interact with something outside of itself will be intrinsically limited, for one thing by Gödel incompleteness theorem.
I agree about the need for an environment. The difference between GPT-3 and an agent in an environment is that GPT-3 only saw tons of static text, while an agent can design an action (an experiment), act it out and observe the results, drawing conclusions. Thus it can act in a way similar to the scientific method.
In the book the meaning of the quote is more to the effect of how truly great athletes can perform on the verge of what spectators would consider inhuman or possible. I'd be hard-pressed to believe that language and syntax would be responsible for these kinds of actions and flow. I'd argue that getting into such a state is not possible while the mind is caught up in the language of things rather than the experience itself, and reacting to it directly. This is what I meant by the quiet consciousness, devoid of language or syntax.
