Ya getting costs down will be the big one, i imagine quantization, distillation and lots and lots of improvements on the compute side both hardware and software wise.

Well right now, running this model is really expensive, but we should prepare a new cope for when equivalent models no longer are, ahead of time.

Let&#x27;s just define AI as &quot;whatever computers still can&#x27;t do.&quot; That&#x27;ll show those dumb statistical parrots!

It&#x27;s not silly. The computer that could beat Kasparov couldn&#x27;t do anything else so of course it wasn&#x27;t Artificial General Intelligence.o3 can do much much more. There is nothing narrow about SOTA LLMs. They are already General. It doesn&#x27;t matter what ARC Maintainers have said. There is no common definition of General that LLMs fail to meet. It&#x27;s not a binary thing.By the time a single machine covers every little test humanity can devise, what comes out of that is not &#x27;AGI&#x27; as the words themselves mean but a General Super Intelligence.

This is just as silly as claiming that people &quot;moved the goalposts&quot; when a computer beat Kasparov at chess to claim that it wasn&#x27;t AGI: it wasn&#x27;t a good test and some people only realize this after the computer beat Kasparov but couldn&#x27;t do much else. In this case the ARC maintainers specifically have stated that this is a necessary but not sufficient test of AGI (I personally think it is neither).

Great. Now we have to think of a new way to move the goalposts.

You could make an equivalently dismissive comment about the hypesters.

The pundits response to the (alleged) plateau was proportional to the certainty with which CEOs of frontier labs discussed pre-training scaling. The o3 result is from scaling test time compute, which represents a meaningful change in how you would build out compute for scaling (single supercluster --&gt; presence in regions close to users). Thus it is important to discuss.

&gt; So much for a plateau lol.It’s been really interesting to watch all the internet pundits’ takes on the plateau… as if the two years since the release of GPT3.5 is somehow enough data for an armchair ponce to predict the performance characteristics of an entirely novel technology that no one understands.

I legit see that if there is not even a new breakthrough just one week, people start shouting plateau plateau.. Our rate of progress is extraordinary and any downplay of it seems like stupid

Nope, makes sense to me. Seems unreasonable to conclude the dataset is not compromised now.

viewed from a skeptical lens of incentives:openai and epochai are both startups with every incentive to peddle this narrative. when no one else can independently verify.

&gt;Frontier Math (25% on high compute, previous 2%)This is so insane that I can&#x27;t help but be skeptical. I know FM answer key is private, but they have to send the questions to OpenAI in order to score the models. And a significant jump on this benchmark sure would increase a company&#x27;s valuation...Happy to be wrong on this.

It&#x27;s not 6,670$&#x2F;task. That was the high efficiency cost for 400 questions.

At 6,670$&#x2F;task? I hope there&#x27;s a jump

&quot;New&quot; reasoning models are plain LLMs with clever reinforcement learning. o1 is itself reinforcement learning on top GPT-4o.They found a way to make test time compute a lot more effective and that is an advance but the idea is not new, the architecture is not new.And the vast majority of people convinced LLMs plateaued did so regardless of test time compute.

You&#x27;re talking apples and oranges. The plateau the frontier models have hit is the limited further gains to be had from dataset (+ corresponding model&#x2F;compute) scaling.These new reasoning models are taking things in a new direction basically by adding search (inference time compute) on top of the basic LLM. So, the capabilities of the models are still improving, but the new variable is how deep of a search you want to do (how much compute to throw at it at inference time). Do you want your chess engine to do a 10 ply search or 20 ply? What kind of real world business problems will benefit from this?

This is also wildly ahead in SWE-bench (71.7%, previous 48%) and Frontier Math (25% on high compute, previous 2%).So much for a plateau lol.

Where does this golf-course phenomenon come from? It doesn&#x27;t really match the real world or how golfing works.

Often what happens is the golf-course phenomenon. As golfing gets less popular, low and mid tier golf courses go out of business as they simply aren&#x27;t needed. But at the same time demand for high end golf courses actually skyrockets because people who want to golf either can give it up or go higher end.This I think will happen with programmers. Rote programming will slowly die out, while demand for super high end will go dramatically up in price.

This assumes that software products in the future will remain at the same complexity as they are today, just with AI building them out.But they won’t. AI will enable building even more complex software which counter intuitively will result in need even more human jobs to deal with this added complexity.Think about how despite an increasing amount of free open source libraries over time enabling some powerful stuff easily, developer jobs have only increased, not decreased.

The question is: How to become a senior when there is no place to be a junior? Will future SWE need to do the 10k hours as a hobby? Will AI speed up or slow down learning?

Frontier expert specialist programmers will always be in demand.Generalist junior and senior engineers will need to think of a different career path in less than 5 years as more layoffs will reduce the software engineering workforce.It looks like it may be the way things are if progress in the o1, o3, oN models and other LLMs continues on.

The absolutist type comments are such a wild take given how often they are so wrong.

Nah, it will arrive. And regardless, this sort of AI reduces the skill level required to make the app. It reduces the amount of people required and thus reduces the demand for engineers. So, even though AI is not CLOSE to what you are suggesting, it can significantly reduce the salaries of those that ARE required. So maybe fewer $150K programmers will be hired with the same revenue for even higher profits.The most bizarre thing is that programmers are literally writing code to replace themselves because once this AI started, it was a race to the bottom and nobody wants to be last.

I’ll believe the models can take the jobs of programmers when they can generate a sophisticated iOS app based on some simple prompts, ready for building and publication in the app store. That is nowhere near the horizon no matter how much things are hyped up, and it may well never arrive.

SaaS works for general problems faced by many businesses.

There’s a very good chance that if a company can replace its programmers with pure AI then it means whatever they’re doing is probably already being offered as a SaaS product so why not just skip the AI and buy that? Much cheaper and you don’t have to worry about dealing with bugs.

Which implies that a few years later they will not become senior SWEs either.

3 to 5 years, max. Traditional coding is going to be dead in the water. Optimistically, the junior SWE job will evolve but more realistically dedicated AI-based programming agents will end demand for Junior SWEs

Well, considering they floated the $2000 subscription idea, and they still haven&#x27;t revealed everything, they could still introduce the $2k sub with o3+agents&#x2F;tool use, which means, till about next week.

Unless the LLMs see multiple leaps in capability, probably indefinitely. The Malthusians in this thread seem to think that LLMs are going to fix the human problems involved in executing these businesses - they won&#x27;t. They make good programmers more productive and will cost some jobs at the margins, but it will be the low-level programming work that was previously outsourced to Asia and South America for cost-arbitrage.

I think they will have to figure out how to get around context limits before that happens. I also wouldn&#x27;t be surprised if the future models that can actually replace workers are sold at such an exorbitant price that only larger companies will be able to afford it. Everyone else gets access to less capable models that still require someone with knowledge to get to an end result.

If it’s any consolation, Agile priests and middle managers will be the first to go

You&#x27;re not being paid $150K to &quot;write code&quot;. You&#x27;re being paid that to deliver solutions - to be a corporate cog than can ingest business requirements and emit (and maintain) business solutions.If there are jobs paying $150K just to code (someone else tells you what to code, and you just code it up), then please share!

How much longer can I get paid $150k to write code ?

Its funny when they say this, as if all humans can solve basic ass question&#x2F;answer combos, people seem to forget theirs a percentage of the population that honestly believe the world is flat along with other hallucinations at the human level

From the post&gt; Passing ARC-AGI does not equate achieving AGI, and, as a matter of fact, I don&#x27;t think o3 is AGI yet. o3 still fails on some very easy tasks, indicating fundamental differences with human intelligence.

If people constantly have to ask if your test is a measure of AGI, maybe it should be renamed to something else.

<pre><code> | Name | Semi-private eval | Public eval |
 |--------------------------------------|-------------------|-------------|
 | Jeremy Berman | 53.6% | 58.5% |
 | Akyürek et al. | 47.5% | 62.8% |
 | Ryan Greenblatt | 43% | 42% |
 | OpenAI o1-preview (pass@1) | 18% | 21% |
 | Anthropic Claude 3.5 Sonnet (pass@1) | 14% | 21% |
 | OpenAI GPT-4o (pass@1) | 5% | 9% |
 | Google Gemini 1.5 (pass@1) | 4.5% | 8% |

</code></pre>
<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2412.04604" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2412.04604</a>

Here are the results for base models[1]:<pre><code> o3 (coming soon) 75.7% 82.8%
 o1-preview 18% 21%
 Claude 3.5 Sonnet 14% 21%
 GPT-4o 5% 9%
 Gemini 1.5 4.5% 8%
</code></pre>
Score (semi-private eval) &#x2F; Score (public eval)[1]: <a href="https:&#x2F;&#x2F;arcprize.org&#x2F;2024-results" rel="nofollow">https:&#x2F;&#x2F;arcprize.org&#x2F;2024-results</a>

Are there any single-step non-reasoner models that do well on this benchmark?I wonder how well the latest Claude 3.5 Sonnet does on this benchmark and if it&#x27;s near o1.

Solving ARC doesn&#x27;t mean we have AGI. Also o3 presumably isn&#x27;t doing program synthesis, seemingly proving Francois wrong on that front. (Not sure I believe the speculation about o3&#x27;s internals in the link.)What I&#x27;m saying is the fact that as models are getting better at reasoning they are also scoring better on ARC proves that it is measuring something relating to reasoning. And nobody else has come up with a comparable benchmark that is so easy for humans and so hard for LLMs. Even today, let alone five years ago when ARC was released. ARC was visionary.

I won&#x27;t be as brutal in my wording, but I agree with the sentiment. This was something drilled into me as someone with a hobby in PC Gaming and Photography: benchmarks, while handy measures of potential capabilities, are not guarantees of real world performance. Very few PC gamers completely reinstall the OS before benchmarking to remove all potential cruft or performance impacts, just as very few photographers exclusively take photos of test materials.While I appreciate the benchmark and its goals (not to mention the puzzles - I quite enjoy figuring them out), successfully passing this benchmark does not demonstrate or guarantee real world capabilities or performance. This is why I increasingly side-eye this field and its obsession with constantly passing benchmarks and then moving the goal posts to a newer, harder benchmark that claims to be a better simulation of human capabilities than the last one: it reeks of squandered capital and a lack of a viable&#x2F;profitable product, at least to my sniff test. Rather than simply capitalize on their actual accomplishments (which LLMs are - natural language interaction is huge!), they&#x27;re trying to prove to Capital that with a few (hundred) billion more in investments, they can make AGI out of this and replace all those expensive humans.They&#x27;ve built the most advanced prediction engines ever conceived, and insist they&#x27;re best used to replace labor. I&#x27;m not sure how they reached that conclusion, but considering even their own models refute this use case for LLMs, I doubt their execution ability on that lofty promise.

100%. The hype is misguided. I doubt half the people excited about the result have even looked at what the benchmark is.

This emphasizes persons and a self-conceived victory narrative over the ground truth.Models have regularly made progress on it, this is not new with the o-series.Doing astoundingly well on it, and having a mutually shared PR interest with OpenAI in this instance, doesn&#x27;t mean a pile of visual puzzles is actually AGI or some well thought out and designed benchmark of True Intelligence(tm). It&#x27;s one type of visual puzzle.I don&#x27;t mean to be negative, but to inject a memento mori. Real story is some guys get together and ride off Chollet&#x27;s name with some visual puzzles from ye olde IQ test, and the deal was Chollet then gets to show up and say it proves program synthesis is required for True Intelligence.Getting this score is extremely impressive but I don&#x27;t assign more signal to it than any other benchmark with some thought to it.

Apparently o3 scored about 25%<a href="https:&#x2F;&#x2F;youtu.be&#x2F;SKBG1sqdyIU?t=4m40s" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;SKBG1sqdyIU?t=4m40s</a>

You&#x27;re right, I was wrong to say &quot;most challenging&quot; as there have been harder ones coming out recently. I think the correct statement would be &quot;most challenging long-standing benchmark&quot; as I don&#x27;t believe any other test designed in 2019 has resisted progress for so long. FrontierMath is only a month old. And of course the real key feature of ARC is that it is easy for humans. FrontierMath is (intentionally) not.

&gt; making the most interesting and challenging LLM benchmark so far.This[1] is currently the most challenging benchmark. I would like to see how O3 handles it, as O1 solved only 1%.1. <a href="https:&#x2F;&#x2F;epoch.ai&#x2F;frontiermath&#x2F;the-benchmark" rel="nofollow">https:&#x2F;&#x2F;epoch.ai&#x2F;frontiermath&#x2F;the-benchmark</a>

It shows objectively that the models are getting better at some form of reasoning, which is at least worth noting. Whether that improved reasoning is relevant for the real world is a different question.

It doesn&#x27;t follow, faulty logic. The two are probably correlated though.

&quot;The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning.&quot;Not sure I understand how this follows. The fact that a certain type of model does well on a certain benchmark means that the benchmark is relevant for a real-world reasoning? That doesn&#x27;t make sense.

I liked the SimpleQA benchmark that measures hallucinations. OpenAI models did surprisingly poorly, even o1. In fact, it looks like OpenAI often does well on benchmarks by taking the shortcut to be more risk prone than both Anthropic and Google.

ARC wasn&#x27;t designed as a benchmark for LLMs, and it doesn&#x27;t make much sense to compare them on it since it&#x27;s the wrong modality. Even a MLM with image inputs can&#x27;t be expected to do well, since they&#x27;re nothing like 99.999% of the training data. The fact that even a text-only LLM can solve ARC problems with the proper framework is important, however.

It&#x27;s the least interesting benchmark for language models among all they&#x27;ve released, especially now that we already had a large jump in its best scores this year. It might be more useful as a multimodal reasoning task since it clearly involves visual elements, but with o3 already performing so well, this has proven unnecessary. ARC-AGI served a very specific purpose well: showcasing tasks where humans easily outperformed language models, so these simple puzzles had their uses. But tasks like proving math theorems or programming are far more impactful.

The benchmark is designed to test for AGI and intelligence, specifically the ability to solve novel problems.If the hypothesis is that LLMs are the “computer” that drives the AGI then of course the benchmark is relevant in testing for AGI.I don’t think you understand the benchmark and its motivation. ARC AGI benchmark problems are extremely easy and simple for humans. But LLMs fail spectacularly at them. Why they fail is irrelevant, the fact they fail though means that we don’t have AGI.

Highly challenging for LLMs because it has nothing to do with language. LLMs and their training processes have all kinds of optimizations for language and how it&#x27;s presented.This benchmark has done a wonderful job with marketing by picking a great name. It&#x27;s largely irrelevant for LLMs despite the fact it&#x27;s difficult.Consider how much of the model is just noise for a task like this given the low amount of information in each token and the high embedding dimensions used in LLMs.

Does it work on short stories, but not novels?  If so, then that&#x27;s just a minor question of context length that should self-resolve over time.

Looks like it&#x27;s not updated for nearly a year and I&#x27;m guessing Gemini 2.0 Flash with 2m context will simply crush it

Benchmark how? Is it good if the LLM can or can&#x27;t solve it?

NovelQA is a great one! I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions, and sampling them repeatedly, varying things like which proper nouns are used, what order relevant details appear, how many irrelevant details (GSM-NoOp) and where they are in the question, things like that.LLMs are far, _far_ below human on elementary problems, once you allow any variation and stop spoonfeeding perfectly phrased word problems. :)<a href="https:&#x2F;&#x2F;machinelearning.apple.com&#x2F;research&#x2F;gsm-symbolic" rel="nofollow">https:&#x2F;&#x2F;machinelearning.apple.com&#x2F;research&#x2F;gsm-symbolic</a><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2410.05229" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2410.05229</a>Paper came out in October, I don&#x27;t think many have fully absorbed the implications.It&#x27;s hard to take any of the claims of &quot;LLMs can do reasoning!&quot; seriously, once you understand that simply changing what names are used in a 8th grade math word problem can have dramatic impact on the accuracy.

&gt; I&#x27;d like to see one that is solving the mystery in a mystery book right before it&#x27;s revealed.I would think this is a not so good bench. Author does not write logically, they write for entertainment.

That&#x27;s an old leaderboard -- has no one checked any SOTA LLM in the last 8 months?

There is a benchmark, NovelQA, that LLMs don&#x27;t dominate when it feels like they should. The benchmark is to read a novel and answer questions about it.LLMs are below human evaluation, as I last looked, but it doesn&#x27;t get much attention.Once it is passed, I&#x27;d like to see one that is solving the mystery in a mystery book right before it&#x27;s revealed.We&#x27;d need unpublished mystery novels to use for that benchmark, but I think it gets at what I think of as reasoning.<a href="https:&#x2F;&#x2F;novelqa.github.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;novelqa.github.io&#x2F;</a>

Because LLMs are on an off-ramp path towards AGI. A generally intelligent system can brute force its way with just memory.Once a model recognizes a weakness through reasoning with CoT when posed to a certain problem and gets the agency to adapt to solve that problem that&#x27;s a precursor towards real AGI capability!

i am confused cause this dataset is visual-based, and yet being used to measure &#x27;LLM&#x27;.  I feel like the visual nature of it was really the biggest hurdle to solving it.

&gt; The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning.One might also interpret that as &quot;the fact that models which are studying to the test are getting better at the test&quot; (Goodhart&#x27;s law), not that they&#x27;re actually reasoning.

Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far.A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning.It&#x27;s obvious to everyone that these models can&#x27;t perform as well as humans on everyday tasks despite blowout scores on the hardest tests we give to humans. Yet nobody could quantify exactly the ways the models were deficient. ARC is the best effort in that direction so far.We don&#x27;t need more &quot;hard&quot; benchmarks. What we need right now are &quot;easy&quot; benchmarks that these models nevertheless fail. I hope Francois has something good cooked up for ARC 2!

fun! the benchmarks are so interesting because real world use is so variable. sometimes 4o will nail a pretty difficult problem, other times o1 pro mode will fail 10 times on what i would think is a pretty easy programming problem and i waste more time trying to do it with ai

If the models run on OpenAIs servers then surely they could still see the questions being put into it if they wanted to cheat? That could only be prevented by making the evaluation a one-time deal that can&#x27;t be repeated, or by having OpenAI distribute their models for evaluators to run themselves, which I doubt they&#x27;re inclined to do.

The private evaluation set is private from the public&#x2F;OpenAI so companies can&#x27;t train on those problems and cheat their way to a high score by overfitting.

the evals are the question&#x2F;answers, ARC-AGI doesn&#x27;t share the questions and answers for a portion so that models can&#x27;t be trained on them, the public ones... the public knows the questions so theres a chance they could have been at least partially been trained on the question (if not the actual answer).Thats how i understand it

So now not only are the models closed, but so are their evals?!  This is a &quot;semi-private&quot; eval. WTH is that supposed to mean? I&#x27;m sure the model is great but I refuse to take their word for it.

The withheld part is really a red flag for me. Why do you want to withhold a compute number?

Why would they give a cost estimate per task on their low compute mode but not their high mode?&quot;low compute&quot; mode: Uses 6 samples per task, Uses 33M tokens for the semi-private eval set, Costs $17-20 per task, Achieves 75.7% accuracy on semi-private evalThe &quot;high compute&quot; mode: Uses 1024 samples per task (172x more compute), Cost data was withheld at OpenAI&#x27;s request, Achieves 87.5% accuracy on semi-private evalCan we just extrapolate $3kish per task on high compute? (wondering if they&#x27;re withheld because this isn&#x27;t the case?)

It&#x27;s a test on which (apparently until now) the vast majority of humans have far outperformed all machine systems.

Yes, it&#x27;s pretty similar to Raven&#x27;s. The reason it is an interesting benchmark is because humans, even very young humans, &quot;get&quot; the test in the sense of understanding what it&#x27;s asking and being able to do pretty well on it - but LLMs have really struggled with the benchmark in the past.Chollett (one of the creators of the ARC benchmark) has been saying it proves LLMs can&#x27;t reason. The test questions are supposed to be unique and not in the model&#x27;s training set. The fact that LLMs struggled with the ARC challenge suggested (to Chollett and others) that models weren&#x27;t &quot;Truly reasoning&quot; but rather just completing based on things they&#x27;d seen before - when the models were confronted with things they hadn&#x27;t seen before, the novel visual patterns, they really struggled.

ML is quite good at understanding and forecasting patterns when you train on the data you want to forecast. LLMs manage to do so much because we just decided to train on everything on the internet and hope that it included everything we ever wanted to know.This tries to create patterns that are intentionally not in the data and see if a system can generalize to them, which o3 super impressively does!

I just googled arc agi questions, and it looks like it is similar to an iq test with raven matrix. Similar as in you have some examples of images before and after, then an image before and you have to guess the after.Could anyone confirm if this is the only kind of questions in the benchmark? If yes, how come there is such a direct connection to &quot;oh this performs better than humans&quot; when llm can be quite better than us in understanding and forecasting patterns? I&#x27;m just curious, not trying to stir up controversies

I agree. If the LLMs we have today never got any smarter, the world would still be transformed over the next ten years.

&gt; Reading constant comments about LLMs can&#x27;t be general intelligence or lack reasoning etc, to me seems like people witnessing the airplane and complaining that it isn&#x27;t &quot;real flying&quot; because it isn&#x27;t a bird flapping its wings (a large portion of the population held that point of view back then).That is a natural reaction to the incessant techbro, AIbro, marketing, and corporate lies that &quot;AI&quot; (or worse AGI) is a real thing, and can be directly compared to real humans.There are people on this very thread saying it&#x27;s better at reasoning than real humans (LOL) because it scored higher on some benchmark than humans... Yet this technology still can&#x27;t reliably determine what number is circled, if two lines intersect, or count the letters in a word. (That said behaviour may have been somewhat finetuned out of newer models only reinforces the fact that the technology inherently not capable of understanding anything.)

People aren’t responding to their own assumption that AGI is necessary, they’re responding to OpenAI and the chorus constantly and loudly singing hymns to AGI.

&gt; to me seems like people witnessing the airplane and complaining that it isn&#x27;t &quot;real flying&quot; because it isn&#x27;t a bird flapping its wingsTo me it is more like there is someone jumping on a pogo ball while flapping their arms and saying that they are flying whenever they hop off the ground.Skeptics say that they are not really flying, while adherents say that &quot;with current pogo ball advancements, they will be flying any day now&quot;

&gt; It doesn&#x27;t need to be general intelligence or perfectly map to human intelligence.&gt; All it needs to be is useful.Computers were already useful.The only definition we have for &quot;intelligence&quot; is human (or, generally, animal) intelligence. If LLMs aren&#x27;t that, let&#x27;s call it something else.

And look at the airplanes, they really can’t just land on a mountain slope or a tree without heavy maintenance afterwards. Those people weren’t all stupid, they questioned the promise of flying servicemen delivering mail or milk to their window and flying on a personal aircar to their workplace. Just like todays promises about whatever the CEOs telltales are. Imagining bullshit isn’t unique to this century.Aerospace is still a highly regulated area that requires training and responsibility. If parallels can be drawn here, they don’t look so cool for a regular guy.

If I could put it into Tesla style robot and it could do dishes and help me figure out tech stuff, it would be more than enough.

On the contrary, the pushback is critical because many employers are buying the hype from AI companies that AGI is imminent, that LLMs can replace professional humans, and that computers are about to eliminate all work (except VCs and CEOs apparently).Every person that believes that LLMs are near sentient or actually do a good job at reasoning is one more person handing over their responsibilities to a zero-accountability highly flawed robot. We&#x27;ve already seen LLMs generate bad legal documents, bad academic papers, and extremely bad code. Similar technology is making bad decisions about who to arrest, who to give loans to, who to hire, who to bomb, and who to refuse heart surgery for. Overconfident humans employing this tech for these purposes have been bamboozled by the lies from OpenAI, Microsoft, Google, et al. It&#x27;s crucial to call out overstatement and overhype about this tech wherever it crops up.

I don&#x27;t think many informed people doubt the utility of LLMs at this point. The potential of human-like AGI has profound implications far beyond utility models, which is why people are so eager to bring it up. A true human-like AGI basically means that most intellectual&#x2F;white collar work will not be needed, and probably manual labor before too long as well. Huge huge implications for humanity, e.g. how does an economy and society even work without workers?

It doesn&#x27;t need to be general intelligence or perfectly map to human intelligence.All it needs to be is useful. Reading constant comments about LLMs can&#x27;t be general intelligence or lack reasoning etc, to me seems like people witnessing the airplane and complaining that it isn&#x27;t &quot;real flying&quot; because it isn&#x27;t a bird flapping its wings (a large portion of the population held that point of view back then).It doesn&#x27;t need to be general intelligence for the rapid advancement of LLM capabilities to be the most societal shifting development in the past decades.

What kind of preparation are you suggesting?

You should look up the terms necessary and sufficient.

Failing the test may prove the AI is not intelligent. Passing the test doesn&#x27;t necessarily prove it is.

&quot;it will take over the world&quot;Calibrating to the current hype cycle has been challenging with AI pronouncements.

I agree, it&#x27;s like watching a meadow ablaze and dismissing it because it&#x27;s not a &#x27;real forest fire&#x27; yet. No it&#x27;s not &#x27;real AGI&#x27; yet, but *this is how we get there* and the pace is relentless, incredible and wholly overwhelming.I&#x27;ve been blessed with grandchildren recently, a little boy that&#x27;s 2 1&#x2F;2 and just this past Saturday a granddaughter. Major events notwithstanding, the world will largely resemble today when they are teenagers, but the future is going to look very very very different. I can&#x27;t even imagine what the capability and pervasiveness of it all will be like in ten years, when they are still just kids. For me as someone that&#x27;s invested in their future I&#x27;m interested in all of the educational opportunities (technical, philosphical and self-awareness) but obviously am concerned about the potential for pernicious side effects.

If AI takes over white collar work that&#x27;s still half of the world&#x27;s labor needs untouched. There are some promising early demos of robotics plus AI. I also saw some promising demos of robotics 10 and 20 years that didn&#x27;t reach mass adoption. I&#x27;d like to believe that by the time I reach old age the robots will be fully qualified replacements for plumbers and home health aides. Nothing I&#x27;ve seen so far makes me think that&#x27;s especially likely.I&#x27;d love more progress on tasks in the physical world, though. There are only a few paths for countries to deal with a growing ratio of old retired people to young workers:1) Prioritize the young people at the expense of the old by e.g. cutting old age benefits (not especially likely since older voters have greater numbers and higher participation rates in elections)2) Prioritize the old people at the expense of the young by raising the demands placed on young people (either directly as labor, e.g. nurses and aides, or indirectly through higher taxation)3) Rapidly increase the population of young people through high fertility or immigration (the historically favored path, but eventually turns back into case 1 or 2 with an even larger numerical burden of older people)4) Increase the health span of older people, so that they are more capable of independent self-care (a good idea, but difficult to achieve at scale, since most effective approaches require behavioral changes)5) Decouple goods and services from labor, so that old people with diminished capabilities can get everything they need without forcing young people to labor for them

&gt; It&#x27;s time people woke up and realised that the old age of AI is over. This new kind is here to stay and it will take over the world. And you better guess it&#x27;ll be sooner rather than later and start to prepare.I was just thinking about how 3D game engines were perceived in the 90s. Every six months some new engine came out, blew people&#x27;s minds, was declared photorealistic, and was forgotten a year later. The best of those engines kept improving and are still here, and kinda did change the world in their own way.Software development seemed rapid and exciting until about Halo or Half Life 2, then it was shallow but shiny press releases for 15 years, and only became so again when OpenAI&#x27;s InstructGPT was demonstrated.While I&#x27;m really impressed with current AI, and value the best models greatly, and agree that they will change (and have already changed) the world… I can&#x27;t help but think of the Next Generation front cover, February 1997 when considering how much further we may be from what we want: <a href="https:&#x2F;&#x2F;www.giantbomb.com&#x2F;pc&#x2F;3045-94&#x2F;forums&#x2F;unreal-yes-this-is-an-actual-pc-screenshot-419874&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.giantbomb.com&#x2F;pc&#x2F;3045-94&#x2F;forums&#x2F;unreal-yes-this-...</a>

I&#x27;m a little torn. ARC is really hard, and Francois is extremely smart and thoughtful about what intelligence means (the original &quot;On the Measure of Intelligence&quot; heavily influenced my ideas on how to think about AI).On the other hand, there is a long, long history of AI achieving X but not being what we would casually refer to as &quot;generally intelligent,&quot; then people deciding X isn&#x27;t really intelligence; only when AI achieves Y will it be intelligence. Then AI achieves Y and...

You are telling a bunch of high earning individuals ($150k+) that they may be dramatically less valuable in the  eat future. Of course the goal posts will keep being pushed back and the acknowledgements will never come.

&gt; These comments are getting ridiculous.Not really. Francois (co-creator of the ARC Prize) has this to say:<pre><code> The v1 version of the benchmark is starting to saturate. There were already signs of this in the Kaggle competition this year: an ensemble of all submissions would score 81%

 Early indications are that ARC-AGI-v2 will represent a complete reset of the state-of-the-art, and it will remain extremely difficult for o3. Meanwhile, a smart human or a small panel of average humans would still be able to score &gt;95% ... This shows that it&#x27;s still feasible to create unsaturated, interesting benchmarks that are easy for humans, yet impossible for AI, without involving specialist knowledge. We will have AGI when creating such evals becomes outright impossible.

 For me, the main open question is where the scaling bottlenecks for the techniques behind o3 are going to be. If human-annotated CoT data is a major bottleneck, for instance, capabilities would start to plateau quickly like they did for LLMs (until the next architecture). If the only bottleneck is test-time search, we will see continued scaling in the future.
</code></pre>
<a href="https:&#x2F;&#x2F;x.com&#x2F;fchollet&#x2F;status&#x2F;1870169764762710376" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;fchollet&#x2F;status&#x2F;1870169764762710376</a> &#x2F; <a href="https:&#x2F;&#x2F;ghostarchive.org&#x2F;archive&#x2F;Sqjbf" rel="nofollow">https:&#x2F;&#x2F;ghostarchive.org&#x2F;archive&#x2F;Sqjbf</a>

The goalposts have moved, again and again.It&#x27;s gone from &quot;well the output is incoherent&quot; to &quot;well it&#x27;s just spitting out stuff it&#x27;s already seen online&quot; to &quot;WELL...uhh IT CAN&#x27;T CREATE NEW&#x2F;NOVEL KNOWLEDGE&quot; in the space of 3-4 years.It&#x27;s incredible.We already have AGI.

These comments are getting ridiculous. I remember when this test was first discussed here on HN and everyone agreed that it clearly proves current AI models are not &quot;intelligent&quot; (whatever that means). And people tried to talk me down when I theorised this test will get nuked soon - like all the ones before. It&#x27;s time people woke up and realised that the old age of AI is over. This new kind is here to stay and it will take over the world. And you better guess it&#x27;ll be sooner rather than later and start to prepare.

I&#x27;m not sure it&#x27;d help what they are talking about much.E.g. go back in time and imagine you didn&#x27;t know there are ways for computers to be really good at performing integration yet as nobody had tried to make them. If someone asked you how to tell if something is intelligent &quot;the ability to easily reason integrations or calculate extremely large multiplications in mathematics&quot; might seem like a great test to make.Skip forward to the modern era and it&#x27;s blatantly obvious CASes like Mathematica on a modern computer range between &quot;ridiculously better than the average person&quot; to &quot;impossibly better than the best person&quot; depending on the test. At the same time, it becomes painfully obvious a CAS is wholly unrelated to general intelligence and just because your test might have been solvable by an AGI doesn&#x27;t mean solving it proves something must have been an AGI.So you come up with a new test... but you have the same problem as originally, it seems like anything non-human completely bombs and an AGI would do well... but how do you know the thing that solves it will have been an AGI for sure and not just another system clearly unrelated?Short of a more clever way what GP is saying is the goalposts must keep being moved until it&#x27;s not so obvious the thing isn&#x27;t AGI, not that the average human gets a certain score which is worse..All that aside, to answer your original question, in the presentation it was said the average human gets 85% and this was the first model to beat that. It was also said a second version is being worked on. They have some papers on their site about clear examples of why the current test clearly has a lot of testing unrelated to whether something is really AGI (a brute force method was shown to get &gt;50% in 2020) so their aim is to create a new goalpost test and see how things shake out this time.

&quot; it&#x27;s complete hubris to conflate ARC or any benchmark with truly general intelligence.&quot;Maybe it would help to include some human results in the AI ranking.I think we&#x27;d find that Humans score lower?

Obviously, the multi billion dollar companies will try to satisfy the benchmarks they are not yet good in, as has always been the case.

From the statement where - this is a pretty tough test where AI scores low vs humans just last year, and AI can do it as good as humans may not be AGI which I agree, but it means something with all caps

It&#x27;s certainly interesting. I&#x27;m just not convinced it&#x27;s a test of general intelligence, and I don&#x27;t think we&#x27;ll know whether or not it is until it&#x27;s been able to operate in the real world to the same degree that our general intelligence does.

&gt; My skeptical impression: it&#x27;s complete hubris to conflate ARC or any benchmark with truly general intelligence.But isn’t it interesting to have several benchmarks? Even if it’s not about passing the Turing test, benchmarks serve a purpose—similar to how we measure microprocessors or other devices. Intelligence may be more elusive, but even if we had an oracle delivering the ultimate intelligence benchmark, we&#x27;d still argue about its limitations. Perhaps we&#x27;d claim it doesn&#x27;t measure creativity well, and we&#x27;d find ourselves revisiting the same debates about different kinds of intelligences.

&gt; I&#x27;m curious what would happen in your eyes if we misattributed general intelligence to an AI model? What are the consequences of a false positive and how would they affect your life?Rich people will think they can use the AI model instead of paying other people to do certain tasks.The consequences could range from brilliant to utterly catastrophic, depending on the context and precise way in which this is done. But I&#x27;d lean toward the catastrophic.

&gt; truly general intelligenceIndistinguishable from goalpost moving like you said, but also no true Scotsman.I&#x27;m curious what would happen in your eyes if we misattributed general intelligence to an AI model? What are the consequences of a false positive and how would they affect your life?It&#x27;s really clear to me how intelligence fits into our reality as part of our social ontology. The attributes and their expression that each of us uses to ground our concept of the intelligent predicate differs wildly.My personal theory is that we tend to have an exemplar-based dataset of intelligence, and each of us attempts to construct a parsimonious model of intelligence, but like all (mental) models, they can be useful but wrong. These models operate in a space where the trade off is completeness or consistency, and most folks, uncomfortable saying &quot;I don&#x27;t know&quot; lean toward being complete in their specification rather than consistent. The unfortunate side-effect is that we&#x27;re able to easily generate test data that highlights our model inconsistency - AI being a case in point.

My guess as an amateur neuroscientist is that what we call intelligence is just a &#x27;measurement&#x27; of problem solving ability in different domains. Can be emotional, spatial, motor, reasoning, etc etc.There is no special sauce in our brain. And we know how much compute there is in our brain– So we can roughly estimate when we&#x27;ll hit that with these &#x27;LLMs&#x27;.Language is important in a human brain development as well. Kids who grow up deaf grow up vastly less intelligent unless they learn sign language. Language allow us to process complex concepts that our brain can learn to solve, without having to be in those complex environments.So in hindsight, it&#x27;s easy to see why it took a language model to be able to solve general tasks and other types deep learning networks couldn&#x27;t.I don&#x27;t really see any limits on these models.

how about a extra large dose of your skepticism. is true intelligence really a thing and not just a vague human construct that tries to point out the mysterious unquantifiable combination of human behaviors?humans clearly dont know what intelligence is unambiguously. theres also no divinely ordained objective dictionary that one can point at to reference what true intelligence is. a deep reflection of trying to pattern associate different human cognitive abilities indicates human cognitive capabilities arent that spectacular really.

I disagree. It’s vastly inefficient, but it is managing to actually solve these problems with a vast search space. If we extrapolate this approach into the future and assume that the search becomes better as the underlying model improves, and assume that the architecture grows more efficient, and assume that the type of parallel computing used here grows cheaper, isn’t it possible that this is a lot more than brute-forcing in terms of what it will achieve? In other words, is it maybe just a really ugly way of doing something functionally equivalent to reasoning?

I think it&#x27;s still an interesting way to measure general intellience, it&#x27;s just that o3 has demonstrated that you can actually achieve human performance on it by training it on the public training set and giving it ridiculous amounts of compute, which I imagine equates to ludicrously long chains-of-thought, and if I understand correctly more than one chain-of-thought per task (they mention sample sizes in the blog post, with o3-low using 6 and o3-high using 1024. Not sure if these are chains-of-thought per task or what).Once you look at it that way it the approach really doesn&#x27;t look like intelligence that&#x27;s able to generalize to novel domains. It doesn&#x27;t pass the sniff test. It looks a lot more like brute-forcing.Which is probably why, in order to actually qualify for the leaderboard, they stipulate that you can&#x27;t use more than $10k more of compute. Otherwise, it just sounds like brute-forcing.

My initial impression: it&#x27;s very impressive and very exciting.My skeptical impression: it&#x27;s complete hubris to conflate ARC or any benchmark with truly general intelligence.I know my skepticism here is identical to moving goalposts. More and more I am shifting my personal understanding of general intelligence as a phenomenon we will only ever be able to identify with the benefit of substantial retrospect.As it is with any sufficiently complex program, if you could discern the result beforehand, you wouldn&#x27;t have had to execute the program in the first place.I&#x27;m not trying to be a downer on the 12th day of Christmas. Perhaps because my first instinct is childlike excitement, I&#x27;m trying to temper it with a little reason.

may be for some sub-domains like math and code it can do that since the verification process can be done &#x2F; relatively tractable

I don’t believe so. If it’s at the point where you could just plug it into a bunch of camera feeds around the world and it could only filter out a useful training set for itself out of that data then we truly would have AGI. I don’t think it’s there yet.

Isn&#x27;t this at the level now where it can sort of self improve. My guess is that they will just use it to improve the model and the cost they are showing per evaluation will go down drastically.So, next step in reasoning is open world reasoning now?

I remember hearing Tesla trying to automate all of production but some things just couldn’t , like the wiring which humans still had to do.

Isn&#x27;t that generally what ... all jobs are? Automation Cost vs Longterm Human cost... its why amazon did the weird &quot;our stores are AI driven&quot; but in reality was cheaper to higher a bunch of guys in a sweat shop to look at the cameras and write things down lol.The thing is given what we&#x27;ve seen from distillation and tech, even if its 6,000&#x2F;task... that will come down drastically over time through optimization and just... faster more efficient processing hardware and software.

Is it? The moore’s law is dead dead, I don’t think this is a given.

Compute can get optimized and cheap quickly.

Yeah, but next year they&#x27;ll come out with a faster GPU, and the year after that another still faster one, and so on. Compute costs are a temporary problem.

They&#x27;re gonna figure it out. Something is being missed somewhere, as human brains can do all this computation on 20 watts. Maybe it will be a hardware shift or maybe just a software one, but I strongly suspect that modern transformers are grossly inefficient.

That&#x27;s the elephant in the room with the reasoning&#x2F;COT approach, it shifts what was previously a scaling of training costs into scaling of training and inference costs. The promise of doing expensive training once and then running the model cheaply forever falls apart once you&#x27;re burning tens, hundreds or thousands of dollars worth of compute every time you run a query.

Time and availability would also be factors.

And human costs have been increasing a few percent per year for a few centuries!

Compute costs on AI with the same roughly the same capabilities have been halving every ~7 months.That makes something like this competitive in ~3 years

This makes me think and speculate if the solution comprises of a &quot;solver&quot; trying semi-random or more targeted things and a &quot;checker&quot; checking these? Usually checking a solution is cognitively (and computationally) easier than coming up with it. Else I cannot think what sort of compute would burn 6000$ per task, unless you are going through a lot of loops and you have somehow solved the part of the problem that can figure out if a solution is correct or not, while coming up with the actual correct solution is not as solved yet to the same degree. Or maybe I am just naive and these prices are just like breakfast for companies like that.

From reading the blog post and Twitter, and cost of other models, I think it&#x27;s evident that it IS actually cost per task, see this tweet: <a href="https:&#x2F;&#x2F;files.catbox.moe&#x2F;z1n8dc.jpg" rel="nofollow">https:&#x2F;&#x2F;files.catbox.moe&#x2F;z1n8dc.jpg</a>And o1 cost $15&#x2F;$60 for 1M in&#x2F;out, so the estimated costs on the graph would match for a single task, not the whole benchmark.

It&#x27;s not 6000&#x2F;task (i.e per question). 6000 is about the retail cost for evaluating the entire benchmark on high efficiency (about 400 questions)

Well they got 75.7% at $17&#x2F;task. Did you see that?

What if we use those humans to generate energy for the tasks?

O3 High (tuned) model scored an 88% at what looks like $6,000&#x2F;task hahaI think soon we&#x27;ll be pricing any kind of tasks by their compute costs. So basically, human = $50&#x2F;task, AI = $6,000&#x2F;task, use human. If AI beats human, use AI? Ofc that&#x27;s considering both get 100% scores on the task

Actually prefer new geminis too. 2.0 experimental especially.

The new ai studio from Google is fantastic

The new gemini&#x27;s are pretty good too

To be fair, until the last checkpoint released 2 days ago, o1 didn&#x27;t really beat sonnet (and if so, barely) in most non-competitive coding benchmarks

Claude has a habit of sometimes just getting “lost”Like I’ll have it a project in Cursor and it will spin up ready to use components that use my site style, reference existing components, and follow all existing patternsThen on some days, it will even forget what language the project is in and start giving me python code for a react project

Yeah it&#x27;s almost like system 1 vs system 2 thinking

To fill this out, I find o1-pro (and -preview when it was live) to be pretty good at filling in blindspots&#x2F;spotting holistic bugs. I use Claude for day to day, and when Claude is spinning, o1 often can point out why. It&#x27;s too slow for AI coding, and I agree that at default its responses aren&#x27;t always satisfying.That said, I think its code style is arguably better, more concise and has better patterns -- Claude needs a fair amount of prompting and oversight to not put out semi-shitty code in terms of structure and architecture.In my mind: going from Slowest to Fastest, and Best Holistically to Worst, the list is:1. o1-pro
2. Claude 3.5
3. Gemini 2 FlashFlash is so fast, that it&#x27;s tempting to use more, but it really needs to be kept to specific work on strong codebases without complex interactions.

o1 is pretty good at spotting OWASP defects, compared to most other models.<a href="https:&#x2F;&#x2F;myswamp.substack.com&#x2F;p&#x2F;benchmarking-llms-against-common" rel="nofollow">https:&#x2F;&#x2F;myswamp.substack.com&#x2F;p&#x2F;benchmarking-llms-against-com...</a>

I just asked o1 a simple yes or no question about x86 atomics and it did one of those A or B replies. The first answer was yes, the second answer was no.

o1 is when all else fails, sometimes it does the same mistakes as weaker models if you give it simple tasks with very little context, but when a good precise context is given it usually outperforms other 
Models

Yeah I feel for chat use case, o1 is just too slow for me, and my queries aren’t that complicated.For coding, o1 is marvelous at Leetcode question I think it is the best teacher I would ever afford to teach me leetcoding, but I don’t find myself have a lot of other use cases for o1 that is complex and requires really long reasoning chain

Can you solve this by putting your lazy prompt through GPT-4o or Sonnet 3.6 and asking it to expand the prompt to a full prompt for o1?

Have you found any tool or guide for writing better o1 prompts? This isn’t the first time I’ve heard this about o1 but no one seems to know how to prompt it

I find myself hoping between o1 and Sonnet pretty frequently these days, and my personal observation is that the quality of output from o1 scales more directly to the quality of the prompting you&#x27;re giving it.In a way it almost feels like it&#x27;s become too good at following instructions and simply just takes your direction more literally. It doesn&#x27;t seem to take the initiative of going the extra mile of filling in the blanks from your lazy input (note: many would see this as a good thing). Claude on the other hand feels more intuitive in discerning intent from a lazy prompt, which I may be prone to offering it at times when I&#x27;m simply trying out ideas.However, if I take the time to write up a well thought out prompt detailing my expectations, I find I much prefer the code o1 creates. It&#x27;s smarter in its approach, offers clever ideas I wouldn&#x27;t have thought of, and generally cleaner.Or put another way, I can give Sonnet a lazy or detailed prompt and get a good result, while o1 will give me an excellent result with a well thought out prompt.What this boils down to is I find myself using Sonnet while brainstorming ideas, or when I simply don&#x27;t know how I want to approach a problem. I can pitch it a feature idea the same way a product owner might pitch an idea to an engineer, and then iterate through sensible and intuitive ways of looking at the problem. Once I get a handle on how I&#x27;d like to implement a solution, I type up a spec and hand it off to o1 to crank out the code I&#x27;d intend to implement.

I&#x27;ve found gemini-1206 to be best. and we can use it free (for now), in google&#x27;s aistudio. It&#x27;s number 1 on lmarena.ai for coding, and generally, and number 1 on bigcodebench.

Which o1? A new version was released a few days ago and beats Sonnet 3.5 on Livebench

Just as an aside, I&#x27;ve personally found o1 to be completely useless for coding.Sonnet 3.5 remains the king of the hill by quite some margin

It seems O3 following trend of Chess engine that you can cut your search depth depends on state.It&#x27;s good for games with clear signal of success (Win&#x2F;Lose for Chess, tests for programming). One of the blocker for AGI is we don&#x27;t have clear evaluation for most of our tasks and we cannot verify them fast enough.

The cost axis is interesting.  O3 Low is $10+ per task and 03 High is over $1000 (it&#x27;s logarithmic graph so it&#x27;s like $50 and $5000 respectively?)

Which parts of reasoning do you think is missing? I do feel like it covers a lot of &#x27;reasoning&#x27; ground despite its on the surface simplicity

I&#x27;d like to see this o3 thing play 5d chess with multiverse time travel or baba is you.The only effect smarter models will have is that intelligent people will have to use less of their brain to do their work. As has always been the case, the medium is the message, and climate change is one of the most difficult and worst problems of our time.If this gets software people to quit en-masse and start working in energy, biology, ecology and preservation? Then it has succeeded.

As excited as I am by this, I still feel like this is still just a small approximation of a small chunk of human reasoning ability at large. o3 (and whatever comes next) feels to me like it will head down the path of being a reasoning coprocessor for various tasks.But, still, this is incredibly impressive.

It&#x27;s not saturated.  85% is average human performance, not &quot;best human&quot; performance.  There is still room for the model to go up to 100% on this eval.

There are things Chimps do easily that humans fail at, and vice&#x2F;versa of course.There are blind spots, doesn&#x27;t take away from &#x27;general&#x27;.

You&#x27;d be surprised what the AVERAGE human fails to do that you think is easy, my mom can&#x27;t fucking send an email without downloading a virus, i have a coworker that believes beyond a shadow of a doubt the world is flat.The Average human is a lot dumber than people on hackernews and reddit seem to realize, shit the people on mturk are likely smarter than the AVERAGE person

Still it&#x27;s comparing average human level performance with best AI performance. Examples of things o3 failed at are insanely easy for humans.

In that vein, perhaps the delta between o3 @ 87.5% and Human @ 85% represents a deficit in the ability of text to communicate human reasoning.In other words, it&#x27;s possible humans can reason better than o3, but cannot articulate that reasoning as well through text - only in our heads, or through some alternative medium.

Agreed. I think what really makes them alien is everything else about them besides intelligence. Namely, no emotional&#x2F;physiological grounding in empathy, shame, pride, and love (on the positive side) or hatred (negative side).

What’s interesting is it might be very close to human intelligence than some “alien” intelligence, because after all it is a LLM and trained on human made text, which kind of represents human intelligence.

An algorithm is “a process or set of rules to be followed in calculations or other problem-solving operations, especially by a computer”How does a giant pile of linear algebra not meet that definition?

Deterministic (ieee 754 floats), terminates on all inputs, correctness (produces loss &lt; X on N training&#x2F;test inputs)At most you can argue that there isn&#x27;t a useful bounded loss on every possible input, but it turns out that humans don&#x27;t achieve useful bounded loss on identifying arbitrary sets of pixels as a cat or whatever, either. Most problems NNs are aimed at are qualitative or probabilistic where provable bounds are less useful than Nth-percentile performance on real-world data.

Running inference on a model certainly is a algorithm.

How do you define &quot;algorithm&quot;? I suspect it is a definition I would find somewhat unusual. Not to say that I strictly disagree, but only because to my mind &quot;neural net&quot; suggests something a bit more concrete than &quot;algorithm&quot;, so I might instead say that an artificial neural net is an implementation of an algorithm, rather than or something like that.But, to my mind, something of the form &quot;Train a neural network with an architecture generally like [blah], with a training method+data like [bleh], and save the result. Then, when inputs are received, run them through the NN in such-and-such way.&quot; would constitute an algorithm.

NN is a very wide term applied in different contexts.When a NN is trained, it produces a set of parameters that basically define an algorithm to do inference with: it&#x27;s a very big one though.We also call that a NN (the joy of natural language).

A fair comparison might be average human. The average human isn&#x27;t a STEM grad. It seems STEM grad approximately equals an IQ of 130. <a href="https:&#x2F;&#x2F;www.accommodationforstudents.com&#x2F;student-blog&#x2F;the-subjects-with-the-highest-iqs" rel="nofollow">https:&#x2F;&#x2F;www.accommodationforstudents.com&#x2F;student-blog&#x2F;the-su...</a>From a post elsewhere the scores on ARC-AGI-PUB are approx average human 64%, o3 87%. <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42474659">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42474659</a>Though also elsewhere, o3 seems very expensive to operate. You could probably hire a PhD researcher for cheaper.

Human performance is much closer to 100% on this, depending on your human. It&#x27;s easy to miss the dot in the corner of the headline graph in TFA that says &quot;STEM grad.&quot;

Super human isn&#x27;t beating rando mech turk.Their post has stem grad at nearly 100%

It actually beats the human average by a wide margin:- 64.2% for humans vs. 82.8%+ for o3....Private Eval:- 85%: threshold for winning the prize [1]Semi-Private Eval:- 87.5%: o3 (unlimited compute) [2]- 75.7%: o3 (limited compute) [2]Public Eval:- 91.5%: o3 (unlimited compute) [2]- 82.8%: o3 (limited compute) [2]- 64.2%: human average (Mechanical Turk) [1] [3]Public Training:- 76.2%: human average (Mechanical Turk) [1] [3]...References:[1] <a href="https:&#x2F;&#x2F;arcprize.org&#x2F;guide" rel="nofollow">https:&#x2F;&#x2F;arcprize.org&#x2F;guide</a>[2] <a href="https:&#x2F;&#x2F;arcprize.org&#x2F;blog&#x2F;oai-o3-pub-breakthrough" rel="nofollow">https:&#x2F;&#x2F;arcprize.org&#x2F;blog&#x2F;oai-o3-pub-breakthrough</a>[3] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2409.01374" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2409.01374</a>

You don&#x27;t think there are already plenty of attempts out there?When someone is &quot;disinterested enough&quot; to publish though, note the obvious way to launch a new fund or advisor with a good track record: crank out a pile of them, run them one or two years, discard the many losers and publish the one or two top winners. I.E. first you should be suspicious of why it&#x27;s being published, then of how selected that result is.

I’ll believe it when the AI can earn money on its own. I obviously don’t mean someone paying a subscription to use the AI I mean, letting the AI lose on the Internet with only the goal of making money and putting it into a bank account.

Curious about how many tests were performed. Did it consistently manage to successfully solve many of these types of problems?

This is so strange. people think that an llm trained on programming questions and docs can do mundane tasks like this means intelligent? Come on.It really calls into question two things.1. You don&#x27;t know what you&#x27;re talking about about.2. You have a perverse incentive to believe this such that you will preach it to others and elevate some job salary range or stock.Either way, not a good look.

Human performance is 85% [1]. o3 high gets 87.5%.This means we have an algorithm to get to human level performance on this task.If you think this task is an eval of general reasoning ability, we have an algorithm for that now.There&#x27;s a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works.Congrats to both Francois Chollet for developing this compelling eval, and to the researchers who saturated it![1] <a href="https:&#x2F;&#x2F;x.com&#x2F;SmokeAwayyy&#x2F;status&#x2F;1870171624403808366" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;SmokeAwayyy&#x2F;status&#x2F;1870171624403808366</a>, <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;html&#x2F;2409.01374v1" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;html&#x2F;2409.01374v1</a>

&gt; We are bags of tricks.We are trick generators, that is what it means to be a general intelligence. Adding another trick in the bag doesn&#x27;t make you a general intelligence, being able to discover and add new tricks yourself makes you a general intelligence.

&gt; the other is that there was an approach that made the task easier than we expected.from reading Dennett&#x27;s philosophy, I&#x27;m convinced that that&#x27;s how human intelligence works - for each task that &quot;only a human could do that&quot;, there&#x27;s a trick that makes it easier than it seems. We are bags of tricks.

Id agree with you if there hasn’t been very deliberate work towards solving ARC for years, and if thr conceit of the benchmark wasn’t specifically based on a conception of human intuition being, put simply, learning and applying out of distribution rules on the fly. ARC wasn’t some arbitrary inverse set, it was designed to benchmark a fundamental capability of general intelligence

Or the test wasn&#x27;t testing anything meaningful, which IMO is what happened here. I think ARC was basically looking at the distribution of what AI is capable of, picked an area that it was bad at and no one had cared enough to go solve, and put together a benchmark. And then we got good at it because someone cared and we had a measurement. Which is essentially the goal of ARC.But I don&#x27;t much agree that it is any meaningful step towards AGI. Maybe it&#x27;s a nice proofpoint that that AI can solve simple problems presented in intentionally opaque ways.

Whenever a benchmark that was thought to be extremely difficult is (nearly) solved, it&#x27;s a mix of two causes. One is that progress on AI capabilities was faster than we expected, and the other is that there was an approach that made the task easier than we expected. I feel like the there&#x27;s a lot of the former here, but the compute cost per task (thousands of dollars to solve one little color grid puzzle??) suggests to me that there&#x27;s some amount of the latter. Chollet also mentions ARC-AGI-2 might be more resistant to this approach.Of course, o3 looks strong on other benchmarks as well, and sometimes &quot;spend a huge amount of compute for one problem&quot; is a great feature to have available if it gets you the answer you needed. So even if there&#x27;s some amount of &quot;ARC-AGI wasn&#x27;t quite as robust as we thought&quot;, o3 is clearly a very powerful model.

&gt; We’ve seemingly hit a ceiling on the base LLM forward-pass capability so any further wins are going to be in how we juggle multiple inferences to solve the problem spaceNot so sure - but we might need to figure out the inference&#x2F;search&#x2F;evaluation strategy in order to provide the data we need to distill to the single forward-pass data fitting.

The general message here seems to be that inference-time brute-forcing works as long as you have a good search and evaluation strategy. We’ve seemingly hit a ceiling on the base LLM forward-pass capability so any further wins are going to be in how we juggle multiple inferences to solve the problem space. It feels like a scripting problem now. Which is cool! A fun space for hacker-engineers. Also:&gt; My mental model for LLMs is that they work as a repository of vector programs. When prompted, they will fetch the program that your prompt maps to and &quot;execute&quot; it on the input at hand. LLMs are a way to store and operationalize millions of useful mini-programs via passive exposure to human-generated content.I found this such an intriguing way of thinking about it.

Looks pretty obvious to me, although, of course, it took me a few moments to understand what&#x27;s expected as a solution.c6e1b8da is moving rectangular figures by a given vector, 0d87d2a6 is drawing horizontal and&#x2F;or vertical lines (connecting dots at the edges) and filling figures they touch, b457fec5 is filling gray figures with a given repeating color pattern.This is pretty straightforward stuff that doesn&#x27;t require much spatial thinking or keeping multiple things&#x2F;aspects in memory - visual puzzles from various &quot;IQ&quot; tests are way harder.This said, now I&#x27;m curious how SoTA LLMs would do on something like WAIS-IV.

I&#x27;ll sound like a total douche bag - but I thought they were incredibly obvious - which I think is the point of them.What took me longer was figuring out how the question was arranged, i.e. left input, right output, 3 examples each

Is it just me or does looking at the ARC-AGI example questions at the bottom... make your brain hurt?

I think it was just announced today! You&#x27;re fine!

Uhh...some of us are apparently living under a rock, as this is the first time I hear about o3 and I&#x27;m on HN far too much every day

Besides higher scores - is there any improvements for a general use? Like asking to help setup home assistant etc etc?

Do you follow Jack Clark? I noticed he&#x27;s been on the road a lot talking to governments and policy makers, and not just in the &quot;AI is coming&quot; way he used to talk.

This picture doesn&#x27;t make sense. If most don&#x27;t have any money to buy products, just invent some other money and start paying one of the other people who doesn&#x27;t have any money to start making the products for you.In reality, if there really is mass unemployment, AI driven automation will make consumables so cheap that anyone will be able to buy it.

&gt; When none of us have jobs or income, there will be no ability for us to buy products. And then no reason for companies to buy ads to sell products to people who don’t have money. Without ad money (or the potential of future ad money), the people pushing the bounds of AGI into work replacement will lose the very income streams powering this research and their valuations.I don&#x27;t think so. I agree the push for AGI will kill the modern consumer product economy, but I think it&#x27;s quite possible for the economy to evolve into a new form (that will probably be terrible for most humans) that keep pushes &quot;work replacement.&quot;Imagine, an AGI billionare buying up land, mines, and power plants as the consumer economy dies, then shifting those resources away from the consumer economy into self-aggrandizing pet projects (e.g. ziggurats, penthouses on Mars, space yachts, life extension, and stuff like that). He might still employ a small community of servants, AGI researchers, and other specialists; but all the rest of the population will be irrelevant to him.And individual autarky probably isn&#x27;t necessary, consumption will be redirected towards the massive pet production I mentioned, with vestigial markets for power, minerals, etc.

When none of us have jobs or income, there will be no ability for us to buy products. And then no reason for companies to buy ads to sell products to people who don’t have money. Without ad money (or the potential of future ad money), the people pushing the bounds of AGI into work replacement will lose the very income streams powering this research and their valuations.Ford didn’t support a 40 hour work week out of the kindness of his heart. He wanted his workers to have time off for buying things (like his cars).I wonder if our AGI industrialist overlords will do something similar for revenue sharing or UBI.

&gt; DoctorsMany replaceable&gt; Police officersMany replaceable (desk officers)

&gt;Doctors capped per year by law.&gt;Trades people only work after having something to do. If you don&#x27;t have sufficient demand for builders, electricians, plumbers, etc... No one can afford to become one. Nevermind the fact that not everyone should be any of those things. Economics fails when the loop fails to close.

The economic theory answer is that people simply switch to jobs that are not yet replaceable by AI. Doctors, nurses, electricians, construction workers, police officers, etc. People in aggregate will produce more, consume more and work less.

&gt; It has well replaced journalists, artists and on its way to replace nearly both junior and senior engineers.Did it, really? Or did it just provide automation for routine no-thinking-necessary text-writing tasks, but is still ultimately completely bound by the level of human operator&#x27;s intelligence? I strongly suspect it&#x27;s the latter. If it had actually replaced journalists it must be junk outlets, where readers&#x27; intelligence is negligible and anything goes.Just yesterday I&#x27;ve used o1 and Claude 3.5 to debug a Linux kernel issue (ultimately, a bad DSDT table causing TPM2 driver unable to reserve memory region for command response buffer, the solution was to use memmap to remove NVS flag from the relevant regions) and confirmed once again LLMs still don&#x27;t reason at all - just spew out plausible-looking chains of words. The models were good listeners, and a mostly-helpful code generators (when they didn&#x27;t make silliest mistakes), but they gave no traces of understanding and no attention for any nuances (e.g. LLM used `IS_ERR` to check `__request_resource` result, despite me giving it full source code for that function and there&#x27;s even a comment that makes it obvious it returns a pointer or NULL, not an error code - misguided attention kind of mistake).So, in my opinion, LLMs (as currently available to broad public, like myself) are useful for automating away some routine stuff, but their usefulness is bounded by the operator&#x27;s knowledge and intelligence. And that means that the actual jobs (if they require thinking and not just writing words) are safe.When asked about what I do at work, I used to joke that I just press buttons on my keyboard in fancy patterns. Ultimately, LLMs seem to suggest that it&#x27;s not what I really do.

Great results. However, let&#x27;s all just admit it.It has well replaced journalists, artists and on its way to replace nearly both junior and senior engineers. The ultimate intention of &quot;AGI&quot; is that it is going to replace tens of millions of jobs. That is it and you know it.It will only accelerate and we need to stop pretending and coping. Instead lets discuss solutions for those lost jobs.So what is the replacement for these lost jobs? (It is not UBI or &quot;better jobs&quot; without defining them.)

There’s this <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2312.05481v9" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2312.05481v9</a>

Some thoughts I put together on all this circa 2010: <a href="https:&#x2F;&#x2F;pdfernhout.net&#x2F;beyond-a-jobless-recovery-knol.html" rel="nofollow">https:&#x2F;&#x2F;pdfernhout.net&#x2F;beyond-a-jobless-recovery-knol.html</a> &quot;This article explores the issue of a &quot;Jobless Recovery&quot; mainly from a heterodox economic perspective. It emphasizes the implications of ideas by Marshall Brain and others that improvements in robotics, automation, design, and voluntary social networks are fundamentally changing the structure of the economic landscape. It outlines towards the end four major alternatives to mainstream economic practice (a basic income, a gift economy, stronger local subsistence economies, and resource-based planning). These alternatives could be used in combination to address what, even as far back as 1964, has been described as a breaking &quot;income-through-jobs link&quot;. This link between jobs and income is breaking because of the declining value of most paid human labor relative to capital investments in automation and better design. Or, as is now the case, the value of paid human labor like at some newspapers or universities is also declining relative to the output of voluntary social networks such as for digital content production (like represented by this document). It is suggested that we will need to fundamentally reevaluate our economic theories and practices to adjust to these new realities emerging from exponential trends in technology and society.&quot;

Is anyone here aware of the latest research that tries to predict the outcome? Please share - super curious as well

I’m super curious as to whether this technology completely destroys the middle class, or if everyone becomes better off because productivity is going to skyrocket.

OpenAI O3 breakthrough high score on ARC-AGI-PUB