That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is just a next word prediction. Gives the impression it's actually thinking, and this is by design. Personally I think it's dangerous in the sense it gives users a false sense of confidence in the LLM and so a LOT of people will blindly trust it. This isn't a good thing.
I'm curious how you think "word predictor" meaningfully describes an instruct model that has developed novel mathematical proofs that have eluded mathematicians for decades?
edit:
You cannot predict all the actions or words of someone smarter than you. If I could always predict Magnus Carlsen's next chess move, I'd be at least as good at chess as Magnus - and that would have to involve a deep understanding of chess, even if I can't explain my understanding.
I can't predict the next token in a novel mathematical proof unless I've already understood the solution.
I think that's more of a limitation in how people think about word predictors
If you can predict the words a bright person will say about X... Isn't that some truly astounding tool? That could be used in myriad useful ways if one is a little creative with it
Since it's also "alien" it can also detect and explore paths that we simply haven't noticed since their biases aren't quite the same as ours
Magnus Carlsen understands chess, a machine designed to simply predict his next move would not necessarily understand chess. This is essentially the Chinese Room experiment.
So I think "word predictor" makes sense here. A word predictor can be really really cool.
It’s an incoherent argument, or a meaningless semantic distinction.
There is no design of such a machine that does not encode a very deep understanding of the game.
Leela Chess Zero does understand chess. She plays at roughly 2300 strength with a search depth of 1 ply - purely on the strength of her gestalt evaluation of the position. Humans have learned a lot about chess from studying her (and AlphaZero’s) games. General, transferable knowledge she developed herself about - for example - the long term value of early rook pawn advances.
“Understanding” doesn’t imply anything about personhood or self reflection or awareness.
It's not, and unfortunately you cannot just dismiss perhaps the greatest refutation of functionalism as being incoherent, you have to actually address the argument.
Take a person (Fred) with no experience of knowledge of chess. They don't know how the game works, how the pieces move, or any of the rules. They memorise an algorithm, say how Leela does its search and evaluation, and they can then look at a position on a board, run the calculations, and come up with a move. Fred can now play chess really strongly, and simultaneously has no understanding of chess. Now in the original experiment it was a room with a person, and the person used a book to reply in Chinese. But the same idea applies.
There is no algorithm that can be memorized. Leela's understanding is in the weights, not neural net algorithms.
I'm familiar with the Chinese room argument and I've never accepted it because what it describes isn't real. It imagines some algorithm for which there is no evidence. Show me this process running and then ask me if it understands Chinese.
To me this is as philosophically dubious as the notion of p-zombies.
You can actually do calculations of LLMs and models like Leela on paper or in your head if you had enough time (and patience)! It's basically just a whole lotta matrix multiplication. It's a thought experiment and its validity does not rest on the ability for someone to actually do these calculations in a suitable timespan. The specifics of the algorithm have no relevance.
If you did see the process running, when asked would you say it understands Chinese?
You can do a thought experiment about an invisible pink dragon, that doesn't mean I have to take a position on it. "Suppose" is doing all the lifting. My position is that experiment can't happen as described.
There is no algorithm for manipulating abstract symbols in a manner that "speaks chinese" without "understanding" it. The experiment bakes in the conclusion from the beginning.
> My position is that experiment can't happen as described.
Say you are the room and are passed symbols on paper, like the suits of playing cards. You use a book (lookup table) to transform series of symbols into a new symbol, and pass it out of the room to the observer.
You get passed ♠ + ♣ and you return ♢. Do you have an understanding of the underlying concept? If so, reply and tell me what it is! But if you don't know what the underlying concept is, how could you argue that the person in the room does?
What does "understand" even mean here? So many people arguing about this seem to assume they can just use words and everyone must accept that because the words have a certain connotation, their argument must be true.
I have no idea how Magnus Carlsen "understands" chess. Neither does anyone else. His brain is giant neural net, taking inputs, sending signals around, and coming out with an output. We think we understand the mechanics of this, but we do not understand exactly why or how sending these signals around produces such good outputs.
So to argue you know for certain that an LLM is not intelligent because it is "just" a next token predictor, without knowing if that is how the human brain operates, is thinking too highly of yourself.
I don't have to try and imagine how Magnus Carlsen understands chess, since I also understand chess, and I operate with the assumption that other people are not zombies and possess a similar form of consciousness. My comment works regardless of the skill of the player.
Imagine you have never played chess, you have no concept of the rules or how the game is played, yet you've learned the entirety of Stockfish's algorithms and can dutifully run them step by step on a piece of paper when you look at a chess position. You would be the strongest chess player ever, and yet you would have less understanding of the game than even a beginner. Just because you can take an input and produce an intelligent output does not mean there is any sort of underlying understanding. This is really just a modification of Searle's Chinese Room Argument, and one of the most famous refutations of functionalism.
"In almost any other application, the biggest Achilles heel of AI is that it makes unverifiable mistakes. But in mathematics, almost uniquely, you can automatically check the output — at least if the output is supposed to be the proof of a theorem, although that is not the only thing mathematicians do. So, AI companies have recognized that their most unambiguous successes — if they’re going to have any — are going to come from mathematics.
In my opinion, there are many use cases of AI that are risky and controversial. In mathematics, the downsides are much more limited"
AI successes in mathematics don't generalize to successes in other fields as the AI promoters want to suggest.
That only explains why the post training is much more efficient - and thus where we have seen the most gains. It says nothing to support the notion that a stochastic parrot has “predicted” an original result.
I knew how LLMs work since 2019 and I've been testing their capabilities. I believe they actually are smart in every meaningful way.
"Next word prediction" just means that answer is generated through computation. I don't think computation can't be smart.
If you believe that LLMs are probabilitic and humans aren't, how do you explain randomness in human behavior? E.g. people making random typos. Have you ever tried to analyze your own behavior, understand how you function? Or do you just inherently believe you're smarter than any computation?
What's the difference between "smart" and "next word prediction", at this point? Back when they first came out, sure, but now they can write code and create art.
What would it take for you to concede a future model was smart?
My personal take would always be that it produces something that isn't in the training set, ie: Demonstrable Creativity, or innovation.
For example, it's training set it purely engineering and code with general language data set, would be "aware" what art is, but has never seen an artistic image, aware what colours are and able to create something it never saw before.
Like a child with a paintbrush, there is an intuitive behavior that happens.
Can you name any examples of a human doing this? I learned about colors, color theory, and so forth in school. I've definitely seen artistic images before.
They can already create something they've never seen - you can prompt ChatGPT to generate images, and there's a few dedicated models for it: https://chatgpt.com/images/
That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is just a next word prediction. Gives the impression it's actually thinking, and this is by design. Personally I think it's dangerous in the sense it gives users a false sense of confidence in the LLM and so a LOT of people will blindly trust it. This isn't a good thing.