Introduction

There is no credibility to the idea of an evolutionary route from generative AI to artifical general intelligence (AGI). The mainstream has seemingly picked up on this a bit since the latest Apple drop, spicily titled The Illusion of Thinking, but I think it was already pretty clear to anyone who wasn't stoned on booster farts and ignorance.

We can certainly look to other approaches. None of which, so far as I'm aware, are delivering results at anything like an AGI-compatible level or getting anywhere near as much investor attention. There's this feeling that, because of what transformer architecture has delivered, we're on the precipice of AGI. It must be so close. But we are not - it remains just as much a prospect for the far future as it was before all this hype.

Before proceeding, a disclaimer. I don't have a particularly good technical understanding of any of this stuff. I have probably overreached significantly and unknowingly here. My argument is fundamentally vibe-based, and I'm sure that I could have found plenty of convinving-sounding evidence for an opposing argument. As ever, do your own due dilligence and critical thinking.

Also I should probably define AGI. Let's say uh, it can perform any cognitive task that a human can, with equal or greater ability. It can reason, think abstractly, deal with novel problems and a reasonable amount of complexity, acquire new knowledge and skills, stuff like that.

The Illusion of Thinking

An Apple research paper attempting to fundamentally evaluate the capabilities of Logical Reasoning Models (LRMs). Most evaluations of this are not fundamental. First because they focus on outputs rather than internal processes, and second because the benchmarks used are potentially contaminated (in my understanding, by problem answers being contained in training data and thus muddying determination of reasoned vs. regurgitated answers). So, the paper is trying to evaluate how much reasoning LRMs are actually capable of.

They did this by having LRMs try to solve puzzles. For every model tested, accuracy progressively declined as complexity increased, before collapsing entirely (to zero). LLMs (surprisingly) performed best in low-complexity tasks, LRMs in medium-complexity, and both collapsed in high-complexity tasks.

What this means, in my intepretation, is that LRMs are not a stepping stone towards AGI. This also lends credence to the argument Gary Marcus (I'm going to be mentioning this guy a lot) has been making since 1998 - that neural nets suck at novel (outside of training data) problems. For the runaway self-improvement path to AGI everyone seems to keep talking about, we are going to need AIs which are good, or at least fairly capable, in dealing with novel problems.

Addressing criticisms of the paper

Most booster criticism seem to be along the lines of "Well, most humans couldn't solve the Tower of Hanoi problem at the level LRMs break down at either." Which is true, but is not the point.

Gary Marcus responds to this criticism with a quote from the paper's co-lead:

Really, you should probably just read Gary's blog. Unlike me he actually knows what he's talking about, and he responds to more criticisms of the paper here.

So, this is a straw man. The claim being attacked by the Apple paper is not "LRMs are better at reasoning than humans", but "LRMs have legitimate reasoning ability". Which they do not - they can simulate reasoning, which is a crucial distinction because simulated reasoning will always collapse in the face of nontrivial complexity and/or novelty - the point at which the reasoning required cannot be interpolated from that which is already encoded in latent space. Again, this is a limitation of neural nets that has been known of for decades, and persists in a major way.

Why can't AGI evolve from generative models?

Because generative models are incapable of understanding, which makes them incapable of reasoning.

There is stuff happening in LLMs that, following limited, functional definitions, can be described as "thinking" and "reasoning"^[1]. But those would be unconventional and very limited definitions.

If you were to ask someone whether they'd expect a "thinking, reasoning" artifical intelligence to be able to solve the Tower of Hanoi for n disks, they would probably say "yes". Especially if the intelligence in question were supplied with an algorithmic solution (which was done in The Illusion of Thinking). It should be as simple as understanding what the algorithm is, reasoning that it could be applied to the problem, and applying it. But LRMs cannot do this, because they cannot actually reason.

There's a whole lot more to what people tend to think of when prompted with words like "thinking" and "reasoning" than what's happening in language models. General intelligence is far more than probabilistic computation based on the statistical relationships between symbols.

LLM performance has hit diminishing returns

No need for me to comment.

Symbolic grounding problem

It's cool that deep patterns emerge when a text corpus as large as LLM training data is mined. For example, a shared symbolic topology which underlies separate languages^[1:1]. Very cool, albeit unsurprising (how else would this work?) Language models then use that stuff to generate sensible-seeming outputs.

But this symbolic topology doesn't mean anything to the model. It's just symbols, and their relationships to other symbols. Nothing is known about their real-world referents. This is the symbolic grounding problem.

Consider what the word "apple" means to you. It's symbolically understood as a fruit, which comes from an apple tree, is coloured red and/or green, can taste sweet, tart, looks a certain way, and so on. This is all referential to your embodied, phenomenological experience. That an apple tastes a certain way, or is a certain colour, means something to you.

A language model has access to far more symbolic knowledge about apples than you. It can identify an apple from a photograph, tell you all about apple varieties and cultivation, depictions of apples in noteworthy artworks, whatever. But the knowledge it has access to is exclusively symbolic - it is not grounded in any phenomenological experience of apples and apple-ness.

Language is not as powerful as we think it is. The map is not the territory, but we increasingly (since the Enlightenment, probably) forget that, especially in the field of AI research. It's like we're giving a hypothetical model which has legitimate reasoning ability ever-increasing volumes of geographical maps, and wondering "Why does it still not get what a mountain is? They're right there, on the maps!" But a load of squiggly lines on a map is not experience of a mountain, in the same way that no volume of symbolic representations of an apple is experience of an apple.

It thus seems very likely that AGI will require the symbolic grounding problem to be resolved, and we seem to be quite some way from achieving that.

Have yet another Gary Marcus quote:

Which I... think is what I'm saying here? A general intelligence must have grounding for its symbols and a means of actually understanding/reasoning about things.

LRMs are not what they claim to be

LRMS are just fancy prompting (and some tools/extensions, like scratchpads). They eke out a bit more performance, but cannot address the fundamental shortcomings of LLMs. Like the appearance of thought is not thought, the appearance of reasoning is not reasoning.

Emergent LLM abilities actually existing is dubious

The pro-scaling current has relied on emergent abilities to justify their position, and there is a huge amount of research purporting to show emergent ability. The problem with this is that measuring things well - especially new, complex things - is hard. Whether the dominant measures of LLM performance actually measure what they intend to is not clear.

If emergent abilities aren't real, that would seem to be a further indictment of the scaling paradigm.

Why would this surprise anyone?

Wizard of Oz type shit, technical ignorance

I figure this is mainly due to the Wizard of Oz type shit going on here. You show most people a LLM, and they figure that it walks like an intelligence, and quacks like an intelligence, so it probably is one. And there are all these smart-sounding people with ostensibly, or even actually impressive credentials saying it's an intelligence, or is very close to becoming one. They probably know what they're talking about?

But in reality, when you look behind the curtain there's not even a Chinese Room there. If there were, I'd be proved wrong because the "thinking" and "reasoning" language models do would be functionally equivalent to humanlike verions of them. There's just an illusion of intelligence. Which is useful, but should not be confused for what it appears to be.

Bubbles and hype, boosters and doomers

In short, I think people would do well to consider the enormous agency problems involved here. There are many ways in which people in the AI space are incentivised to be dishonest.

For example, we've got Bob McGrew, OpenAI's former research head (and Palantir alumni, so good confidence he's entirely amoral), spouting insane takes about LLMs scaling to hallucination-free infinity and AGI-enabling tech already existing... on Sequoia Capital's YouTube channel. Even in 2023, Sequoia had eleven billion USD invested in OpenAI. Their AUM in late 2024 was $56.3bn USD, so unless things have shifted significantly, OpenAI investment is a full ~20% of their AUM. I'd imagine, if anything, it has grown. Sequoia also backed FTX (yes, the Sam Bankman-Fried FTX) with $225m, and published a blog hyping SBF up. Gee, I wonder if there might be an incentive for some bias here? And a precedent for it? It's almost like venture capitalists can somehow benefit from creating and pumping speculative bubbles and hype.

So yeah, we're in an enormous speculative/hype bubble. This doesn't incentivise sober analysis. Airing a moderate view like "I think it'll be a big deal, but we're really not on the path to building God here" doesn't tend to attract media coverage, investment, or returns on investment.

Whereas takes like "We are going to achieve superhuman AI soon, and it's going to be bigger than the industrial revolution and very scary" are much more enaging. It doesn't matter if your "predictions" are really speculative fiction - nobody who attention would be paid to is likely to call you out on it, because the bubble is getting everyone paid and many don't know better anyway. The amount of hand-wringing I've seen over AI 2027 is straight up embarrassing, Roko's Basilisk levels of silliness. It has been misunderstood and misrepresented to an absolutely incredible degree. I'll link to yet another Gary Marcus post wherein he dissects the whole thing and reflects on its potential impacts - stoking the AI arms race by speculating on AI's intersection with the Chinese threat to Western hegemony, and making Sam Altman stacks of money. Which brings us back to - ah, lovely - agency problems.

Read with the understanding of it actually being speculative fiction^[2], AI-2027 is quite fun, interesting and potentially valuable as a thought experiment for policymakers. I certainly agree that they should be doing more in anticipating AI risks. What we have today is already incredibly risky. But so far as I can tell, the authors are effective altruism/LessWrong^[3] rationalist types - i.e. so smart they've looped round and become total idiots. If you're to read it as a legitimate prediction, you may as well huff some glue and watch a Terminator movie.

Consider what happened to past tech hypes. The Dot-com bubble has been the largest thus far, but there have been many underwhelming allegedly big deals more recently. Big data, crypto, blockchain, Web3, augmented reality, virtual reality, IoT, wearables, 3D TV, 3D printing. All of these things were purportedly going to be revolutionary. Most of them are still around, but their impact has been small. My point is not that this is always the case (phone, for example, has proven to be a pretty big deal), but that tech has a very established track record of overpromising/overhyping and underdelivering, and this should factor into how we critically engage with the AI media climate.

What do directly involved parties get out of hype/bubbles?

Part of it is staggering amounts of money. OpenAI is now valued at $300bn USD. But keeping a hype bubble going requires escalation. Each funding round requires more grandiose claims about how close we are to AGI, how much better the next model will be, who's going to be put out of work by it, and so on. The DeepSeek fallout seems telling here. The challenge to the scaling paradigm couldn't be outright ignored, but was interpreted as "No, it's still good - we just need to pivot to scaling smarter." Which would be a rather convenient narrative for rejustifying the bubble, buying time to try and deliver on all these outlandish promises.

The inventive structure here is very broken. Everyone needs to buy into, and stick to, a story. VCs to justify their investments, executives to justify their company's valuation, researchers to justify bubble-sustaining research directions, and employees in general to preserve the value of their equity. There's huge social and financial incentive towards not dissenting, and for dissent to be silenced - because if people stop believing the story, the bubble bursts.

This may explain why Sam Altman's ousting (which the actual reasons for were seemingly continually lying, a huge conflict of interest in his personally owning OpenAI's VC fund, and his concealing that ownership) incited an overwhelming employee revolt. The ousting occurred just before a planned $1bn USD employee stock sale - that is, just before employees got some big cheques. "The share sale was now in the balance, with the weekend’s drama representing a material change in circumstances, but could yet go ahead should Altman return, said one person with knowledge of the situation." Perhaps this guy who's widely reported to be a real prick is actually just that great of a boss and leader, but great enough for 95% of employees to threaten to walk if he wasn't reinstated? Hm.

There's also the fact that AI companies are incentivised to hype AI risks not only to seek growth, investment and market share, but in order to achieve regulatory capture. These "Please, our product is so dangerous, regulate us!" appeals aim to both allow them an inappropriate level of influence over the nature of regulation, and get regulation in place soon, erecting high entry barriers in the AI market. Which is dominated by huge players, so - another agency problem - of course they have an interest in keeping new, potentially disruptive entrants out. DeepSeek was embarrassing. Please, no more of that. We do need these incredible amounts of money and compute scale, actually!

And, of course, hype/doom can get people using products. Particularly when it involves threats that they're going to be out of a job very shortly, that they can save a lot of money on expensive developer salaries, or that their laggard company is going to be hopelessly outcompeted by early AI adopters.

This has been a lot of agency problem stuff. Hopefully you can see how strongly incentivised industry insiders are towards being less than honest.

Tech hubris

Big tech is a very hubristic industry. I don't think this has ever been more evident. I'm sure there are industry boosters who don't really believe what they're saying, but are saying it due to the aforementioned agency problems (I'll discount that possibility for this section). But there are also true believers, and they literally think that building God is imminently possible. Perhaps they've drawn reasonable conclusions from information I don't have, but I think it's far more likely that it's hubris and groupthink.

Plenty of Sam Altman examples here. This is a good one. In 2023, he said that "It’s totally hopeless to compete with us on training foundation models." In 2025, DeepSeek dropped. Here he is in June 2025 saying they've "cracked reasoning". And here, he responds in a very Elon-coded way to Gary Marcus mentioning how much he keeps doubling down.

In 2016, Geoffrey Hinton said it was "completely obvious" that radiologists would be replaced by AI.

SoftBank's legendarily incisive (sarcastic, derogatory) Masayoshi Son has revised his estimates on AGI realisation from "in the next ten years" (2023), to two to three years (late 2024), to "much earlier" than he previously thought (early 2025).

These are clearly intelligent people, but intelligence is not wisdom.

People are prone to being idiots

There seems to be something about AI which particularly encourages people to reach strongly-held beliefs based on little to no knowledge or understanding. Perhaps it's just the incredible volume and strength of bubble hype. I would not claim to have a good understanding of this stuff, but I'm confident that it's significantly better than average... and I'm fairly careful when forming opinions (or not) about things I'm ignorant of, at risk of being an idiot. Generally, I like to think of myself as someone who's quite aware of the Dunning-Kruger effect (yes, I know - but it doesn't feel like I need a really strong technical understanding of this stuff to write this). And I would hazard that most people are not.

It seems like most of these casual (no financial incentive) boosters do not work in tech, and that feels deeply telling. They simply do not have good opinions on this stuff.

Online boosters

Feels like I shouldn't be commenting on this as it's mostly because I wanna dunk, but a lot of the boosters someone is likely to encounter are online so it's probably warranted.

Going back to The Illusion of Thinking, a response appeared quite quickly and was widely shared and perceived as a legitimate rebuttal. Unfortunately it turned out that this response was literally a joke. This illustrates just how hyped people are, and how cult-y it has gotten.

The vibe reminds me of Bitcoin/crypto and Gamestop (stock) online spaces. A lot of people are very excited about the technological singularity, and probably a lot are excited about robot waifus. They've been dreaming about AGI and its implications for humanity for years, are very emotionally invested, and now something that's ostensibly a path to that has appeared - so of course their reasoning about it isn't, uh, very reasonable. So dissenting opinions tend to be answered by dogpiling instead of legitimate engagement. Being part of something world-changing is an exciting prospect, and it's presumably quite threatening when people call the reality of it into doubt.

I also have the impression that there's been quite a lot of goalpost-moving. Weren't we hearing that hallucinations would be quickly solved a few years ago? And that models would be able to perform better than people by now?

There really is a lot going on in humanlike cognition

We have many cognitive faculties besides thinking and reasoning. But people tend to use words like "thinking" and "reasoning" as shorthand which includes a whole mess of other, even more nebulously defined and understood, things. Even in neuroscience this stuff isn't really robustly defined, but it's particularly loose in the AI space.

A big part of the gap might fall under the heading "world modelling". We can create models of the world, and use them to make predictions. We have the ability to model ourselves (this is what the ego is), and place these self-models into our world models as a sort of subjective avatar, projected into a model of experience. The world models we create blend phenomenological experience with symbols.

We also do a lot of causal reasoning with these models. This does involve seeing patterns, which language models are very good at. But it also involves modelling interventions and counterfactuals, which they are very bad at.

Going back to the apple thing, if I ask you what will happen if I throw an apple at your head, you can reason about it using the parts of your world model relating to apples and human heads, and you can communicate that to me in symbols. You can do this even if you've never read anything about apple/human head interfacing, or experienced an apple being thrown at you, because apples and human heads are both part of your world model - you understand things about apples and human heads. Not merely in relation to other symbols, but in relation to your experience of apples and human heads.

So to answer, you'd probably use integrated perceptive, memory, emotional and physical understanding, along with your agentic self-awareness/self-modelling, to imagine the event. And what you'd be doing here by "imagining" is running a wildly sophisticated simulation. This is rich, sensoriomtor imagining - you might feel your anxiety, yourself tensing up in preparation to try and dodge, the sound of the apple whistling through the air, the slightly wet impact of it hitting your head, and the stinging it might cause. Any counterfactual that arises can be entertained - what if you dodge, or the apple is rotten, or unripe, or it's hollow plastic and I'm just pranking you, or you know I can't throw for shit, so I probably won't hit you. Because your symbols are grounded, you can then translate all this - which was both symbolic and experiential - into symbols (language), and communicate it to me.

While language models can simulate this sort of reasoning to a degree, they fundamentally cannot actually, really do it. You might still have the exchange:

User: If someone were to throw an apple at my head, how would it feel?
Language model: It'd hurt, but apples don't tend to be very hard, so you'd probably be okay.

But this would be a result of pattern recognition and interpolation, not modelling or reasoning. The model is just drawing probabilities from a deeply mined symbolic corpus in liminal space. What it would do is much less complex and rich than what you would, and as a result it is not anywhere near as useful.

What I'm trying to get at is that "meaning" is not something LLMs can apprehend. Training data and outputs are both meaningful when interpreted by us, and it's easy to assume that the chain of meaningfulness was not broken in the middle, in latent space. But this is a fundamental misapprehension - latent space representations are just mathematical objects.

I'd entertain an argument that the meaning in training data is encoded into latent space, and so can be reproduced from there even if the model doesn't "get it", but that is regurgitation and interpolation - not reasoning. Even if we accept that meaning is encoded and reproduced, it cannot be usefully transformed because that requires actual understanding - explicit representation of facts, the context of some kind of world model, and all that. So, claiming that LLMs deal with meaning in a legitimate way is like claiming that the words in a book have meaning, to the book.

I think this is why we see only limited, brittle (simulated) reasoning in LLMs, and why it collapses quickly when LLMs are asked to generalise much beyond the reasoning represented in their training data.

To summarise, we exist primarily in the world of direct experience. Our world of symbols is like a conceptual augmented reality overlay and communication protocol - a veneer which sits on top of experience, to some degree aids our modelling of it, and allows us to communicate about it. While we often consciously "forget" that it was experience which came first, our world models and symbols are nevertheless grounded.

But LLMs don't have access to anything other than the world of symbols. They can't world model, because they're missing the foundation of experience and the cognitive faculties required to do it. It's not that phenomenology or humanlike reasoning are necessarily required for AGI, but something like them is. AGI cannot happen without the ability to world model, and to do something like reasoning in a robust, strongly generalisable manner.

Alternative approaches

There is a lot of work going on in embodied robotics and world modelling, but it's very early days. Research has been simmering for years, but only with the AI bubble has it started to draw the attention of investors. Who are highly fixated on short-term returns, so... is their attention going to be held when the bubble bursts? Probably not, and only deep-pocketed tech companies will be left doing such research with an eye towards AGI.

Meta's chief AI scientist Yann LeCun, one of the more sober voices this directly involved in AI, recognises the need for world modelling. Perhaps this sobriety can be attributed to Meta having one of the sicklier dogs in this race - so, he may have the levity to say more or less what he actually thinks.

But, for whatever his estimates are worth, he puts AGI-compatible world modelling at at least a decade away. I'm not aware of anyone offering a more optimistic estimate, and suspect his is overly optimistic anyway. But this stuff working well is crucial for AGI.

As an example of the current state of the art, Meta just dropped V-JEPA 2 - and it's impressive, but also very rudimentary. The only realistic applications in the nearish future are things like industrial robotics.

So, what will actually happen?

This isn't to say that I don't think AI will be deeply transformative. Speculating on how is happily out of scope. Though I do expect that large parts of the internet will become unusably sloppy, accurate and trustworthy information will become increasingly hard to find and identify^[4], and that this will have a huge contribution to the death of consensus reality^[5].

But for AGI, it's full self-driving all over again. Consistent messaging of "It's getting closer, we're almost there!", and not much else. Maybe it will arrive, one day, but not any time soon. I expect the bubble to have burst in spectacular fashion long before then.