Yeah... probably right. I do hold out hope that this is mostly a timeframe thing. Like, the library, printing press, etc. all had their moments of centralization. But eventually they federated.

I like the analogy of streaming services vs local media streaming, although I don&#x27;t think it holds up when looking at history.Streaming Services are getting worse and more expensive. I don&#x27;t see a single report suggesting piracy is decreasing, it seemingly is only increasing now.When costs increase, quality decreases people look for alternatives. The advent of faster broadband enabled Napster and MP3 sharing. I think this could have a resurgence if the peices align correctly (a new bitorrent client, a new torrent site, something to break the status quo).How this related to AI, I don&#x27;t know, although I wouldn&#x27;t be set on the idea that we will never have local AI as the norm. There is a lot more movement in this space then there is for local streaming imo.

That&#x27;s like wondering if enough people discovering local media streaming will disrupt commercial streaming services. It&#x27;s not going to happen. Most people are not ambitious and will let themselves be controlled by the services of least resistance.And you can&#x27;t take comfort in knowing that you, personally, will remain in control of your own computing. The majority will let the range and direction of their thoughts and output be determined by the will of the tech giant whose AI they adopt. And that will shape society.

I wonder if a popularization moment for local AI will ultimately be the pin-prick that pops the AI bubble. Like the deepseek or openclaw moments but bigger&#x2F;next.

This is a bit naive IMHO...I have tried quite a bunch of local models, and the reality is that it&#x27;s not just a matter of of &quot;it&#x27;s a small model that should be hostable easily&quot;. Its also a matter of whats your acceptable prefill TTFT and decode t&#x2F;s.All the local models I used, on a _consumer grade_ server (32GB DDR5, AMD Ryzen) have been mostly unusable interactively (no use as coding agent decently possible), and even for things like classification, context size is immediatly an issue.I say that with 6m experience running various local models for classifying and summarizing my RSS feeds. Just offline summarizing ans tagging HN articles published on the front page barely make the queue sustainable and not growing continuously.

One of my hobbyist workflows involved transcribing ETF prospecti into yaml for an optimizer to optimize over.Used to take me maybe 10-20 minutes per sheet.Then I got codex to whip up a script that sends each sheet to a fairly low parameter locally running LLM and I have the yaml in a couple seconds.My dream is to bootstrap myself to local productivity with providers… I know I’ll never get there because hedonic treadmill etc, but I do feel there’s lots more juice to squeeze. I just need to invest more time into AI engineering…

You&#x27;re maybe missing the article&#x27;s point, which is to use local models appropriately:&gt; “But Local Models Aren’t As Smart”&gt; Correct.&gt; But also so what?&gt; Most app features don’t need a model that can write Shakespeare, explain quantum mechanics, and pass the bar exam. They need a model that can do one of these reliably: summarize, classify, extract, rewrite, or normalize.&gt; And for those tasks, local models can be truly excellent.

I would love for local inference to be possible, but from my experience, Kimi 2.6 is the only model that would be worth it, and its a $10k (M3 Ultra max spec&#x27;d - 30s TTFT so kind of slowish) to $30k (RTX6000&#x2F;700GB+ DDR5) upfront, noise &#x2F; power consumption aside.

Magic math and computer science that allows us to get the same quality response for a fraction of the GPU.

I&#x27;d assume its a totally different architecture that isn&#x27;t based on storing a compressed dataset of all digital human text.

oof, this bubble popping is gonna be brutal.

It seems pretty clearly inline with the dotcom bubble to me. Every company claims to be a leading AI company, those building infrastructure are promising the moon and getting 1&#x2F;3 of the way there, and no one knows how to monetize it justify the hype or expense.

The cynical take is getting more and more to be the only rational one:The promised mega-data center deals are meant to boost valuations today, not serve tons of customers three years from now.

It took us only, what 70-ish years of computer and AI research to get to this point, so yeah, probably just one little thing and then we&#x27;ll have it &lt;&#x2F;sarcasm&gt;Seriously. I have never ever seen so many people so willingly drink the marketing kool-aid from companies selling their product before. It&#x27;s scarier to me than any threats of AI actually disrupting society (because it is so far from being capable of doing that).

It feels like we&#x27;re one technological breakthrough away from all of these data centers going up to be deemed irrelevant.

That sounds like government. So your problem is mostly that you expect to have a collective social effort, but not enough to pay for it as a public good.

Depends on the domain. There are plenty of different use cases where the data needed for training is available for personal, or non-commercial, use. At that point, it does come down to compute&#x2F;time to do the training, which if you are willing to wait, consumer grade hardware is perfectly capable of developing useful models.

You&#x27;re also embodied and experiencing the world around you with more senses than only the ability to read text.

What does this even have to do with the parent?  Your capabilities have nothing to do with LLM capabilities.  The two work in completely different ways.  The reason LLMs work is because they are huge and have been trained on vast amounts of data, full stop.  Sure, there&#x27;s potential someday to get something useful using less data, but we aren&#x27;t there.

This seems overly pessimistic.I may personally be of modest intelligence, but to acquire the intelligence that I do have, I did not need to train on every book ever written, every Wikipedia article ever written, every blog post ever written, every reference manual ever written, every line of code ever written, and so on. In fact, I didn&#x27;t train on even 1% of those materials, or even 0.00000000001% of those. The texts themselves were demonstrably not a prerequisite for intelligence.At minimum, given that it only took me about 20 years of casual observation of my surroundings to approximate intelligence, this is proof positive that the only &quot;dataset&quot; you need is a bunch of sensors and the world around you.And yes, of course, the human brain does not start from zero; it had a few million years of evolution to produce a fertile plot for intelligence to take root. But that fundamental architecture is fairly generic, and does not at all seem predicated on any sort of specific training set. You could feasibly evolve it artificially.

Not the whole thing, at least with current technology, but LoRAs are really good at fine tuning, and can be generated in a few hours on high-end gaming computers, so as long as the base model is in your language, you likely have enough spate computing power, in whatever electronics you own, to train a few LoRAs a month.In the future, when regular home computers have the capabilities of modern servers, we&#x27;ll be able to train the entire LLM at home.

There is so much technology that we are unable to reproduce locally, I don&#x27;t think LLMs are in any way different. There will be large LLM manufacturers, small LLM manufacturers, LLM artisanals, LLM enthusiasts and of course LLM consumers, just like with everything.

If you don&#x27;t like the model you have, ditch it and try another one.  There are already dozens.

And this is important because even though you are running a model locally, it&#x27;s still a proprietary model.  You have no say in what it was trained on, how that training data is labeled, what the guardrails are, what biases it might have, none of that.

That&#x27;s a fair concern, but I&#x27;d separate training from inference here

My problem with LLMs (apart from philosophical aspects and economical impact) is that it would be unlikely for any of us to be able to train something functional locally (toy-like LLMs -- sure, but something really useful -- no). Apart from that it requires immense computing power, it also requires a dataset which is for the most part is obtained illegally.

&gt; and even then the inference is ungodly slow.This is the wrong way of putting it. Local inference with SOTA models is all about slowing down compute for the sake of fitting on bespoke repurposed hardware. You don&#x27;t need to go fast if you have the whole machine to yourself 24&#x2F;7. Cloud AI vendors can&#x27;t match that kind of economics.

A local Answer Machine is the dream, especially when the internet is decaying and generally on its last legs, but the hardware requirements seem like a huge mountain to climb. Things are progressing tremendously - deepseek v4 flash is very good for what it is - but even that goes beyond any reasonable local setup, which imo is 128 GB ram + 16 GB vram. 4 ram slots on a consumer board craters ram speed, 256 gb macs are too expensive, and even then the inference is ungodly slow.On the other hand… v4 flash model is actual magic compared to what was available 2 years ago. If the rate of improvement stays as is, we’ll get a similar performance in a ~120B model in a year, which is viable (if expensive) for everyman hardware. Possibly you’ll be able to run its equivalent on a ~$1200 laptop by 2028, which for me-in-2020 would sound straight out of a scifi movie. A good harness that lets the model fetch data from other sources like a local wikipedia copy from kiwix could do a lot for factual knowledge, too; there’s only so much you can encode in the model itself, but even a cheapish (pre-curent prices) 2TB drive can hold an immense amount of LLM-accessible data.Big caveat: I don’t see local models for programming or generally demanding agentic tasks being worth it anytime soon. You likely want bleeding edge models for it, and speed is far more important. Chat at 20tok&#x2F;s is fine; working on even a small codebase at 20tok&#x2F;s, especially on a noticeably weaker model, is just a waste of time. Maybe it’s a PEBKAC but I have no idea how people make any meaningful use out of qwen 3.6.

Until the hardware is economical and powerful enough, local AI that can compete with frontier models today is still far off.If we could even get something like GPT 5.5 running locally that would be quite useful.

&gt;&gt; years ago I launched &quot;The Brutalist Report&quot;proceeds to brutalise the reader with an 88-point headline font.

Same as local compute.Welcome back to 2014. Let us now continue yelling at the cloud.

I&#x27;m someone who is trying to build a subscription-based business to cover underlying LLM costs, and very hopeful I can one day just sell a permanent license to the software instead with customers using local LLMs to power it.

It should be feasible to crowd fund training runs right?

Perhaps you can create a compelling UX around it and sell it as a subscription. &quot;Normies&quot; will not be able&#x2F;willing to build it. You can then patch the model&#x2F;ship new features around it as it evolves. For example I have built an ambient todo list &#x2F; health data extractor using Gemma 4 2EB and Whisper. Nothing to brag about but it does fairly decent job even in foreign languages.

&gt; What is the business model of open weight AI?This is what I do not understand as well and advertising the knowledge and more advanced model is also the only thing that comes to my mind.Since a month I am using gemma4 locally successfully on a MBP M2 for many search queries (wikipedia style questions) and it is really good, fast enough (30-40t&#x2F;s) and feels nice as it keeps these queries private. But I don&#x27;t understand why Google does this and so I think &quot;we&quot; need to find a better solution where the entire pipeline is open and the compute somehow crowdfunded. Because there will be a time when these local models will get more closed like Android is closing down. One restriction they might enforce in the future could be that they cripple the models down for &quot;sensitive&quot; topics like cybersecurity or health topics. Or the government could even feel the need to force them to do so.

Meta released Llama just when OpenAI was so hot and its valuation was going through the roof. Speculating, but Meta probably thought the model not competitive enough to keep as a secret weapon but well good enough to commercially damage OpenAI who were a sudden competitor for most-valued-company?In the same way you can imagine the Chinese government pushing the release of deepseek etc to make sure no one thinks the US has “won” and to keep everyone aware that a foreign model might leapfrog in the short term future etc.At some point though if OpenAI&#x2F;Antropic&#x2F;Google plateau or go bust then the open source sponsorship becomes less likely, as making it open source was a weapon not a principle.

&gt; What is the business model of open weight AI? I don&#x27;t think there is any. At best it can serve as an advertisement for the more advanced models you sell.I don&#x27;t think local will necessarily be open-weight. And then it&#x27;s not that different from personal computing: you&#x27;re giving up the big lucrative corporate mainframe, thin-client model for &quot;sell copies to a ton of individuals.&quot;So it&#x27;d be someone else (an Apple, or the next-year equivalent of 1976 Apple) who&#x27;d start eating into that. There are a few on-device things today, but not for much heavy lifting. At first it&#x27;s a toy, could maybe become more realized in a still-toy-like basis like a fully-local Alexa; in the future it grows until it eats 80-90% of the OpenAI&#x2F;Anthropic use cases.Incumbents would always rather you pay a subscription or per-use forever, but if the market looks big enough, someone will try to disrupt it.

The business model is the total lack of attention to Qwen and Kimi that would happen if their models weren&#x27;t downloadable. Before releasing the weights, there was basically zero attention paid in the western hemisphere to them, for whatever reason. By releasing the weights, they&#x27;re relevant in the western world. The business model is to get people in the West to pay to use their platform hosting their AI, that otherwise would never have heard of them. As you said, advertising&#x2F;marketing, essentially.

What is the business model of Wikipedia? I don&#x27;t think there is any.Not everything good in our society needs to have a &quot;business model&quot;. People still work on it. It&#x27;s FINE.

This is where government funding can play a role.Sometimes there are things where the public good is best served with public expenditure.

If a local model hits critical mass the business model is to use it to shape opinions in a way that is advantageous for the company&#x2F;owners.Much like the current Twitter model, being able to put your thumb on the scale of &quot;truth&quot;. Bake a stronger bias towards their preferred narrative directly into the model. Could be as &quot;benign&quot; as training it to prefer Azure over AWS. Could be much worse.

Open sourcing models is a marketing strategy. Chinese labs and small international labs have no awareness or distribution, so unless they become a hot topic for a while, nobody is going to bother trying out their models. Open source gets them that, and is essentially a tax on newcomers. When you start out you simply have no other option but to open source your models.So, the business model of open models is the same as closed models: Sell inference. Open source is marketing for that inference.<a href="https:&#x2F;&#x2F;try.works&#x2F;#why-chinese-ai-labs-went-open-and-will-remain-open" rel="nofollow">https:&#x2F;&#x2F;try.works&#x2F;#why-chinese-ai-labs-went-open-and-will-re...</a>

Cloud providers have incentives to release open source models but for some reasons this happens only in China. Amazon, Azure, Google benefit from open source models because people run them on their hardware.

Hardware sales would be an excellent business model for open weights. Nvidia is already on it with their Nemotron models. Any new LLM&#x2F;NPU hardware companies would want to so the same, if noone else does it for them (Chinese labs currently do).Selling managed self-hosting solutions would be another. That is the business of that recent American company.Selling fine-tuning services or similar adaptations is another. That is what Unsloth is going for, I believe.Most likely any sound business strategy is going to be of &quot;commoditize your compliments&quot; type. There are many complementary products to open-weight - some probably not invented&#x2F;discovered yet.

What is the business model of open weight AI? I don&#x27;t think there is any. At best it can serve as an advertisement for the more advanced models you sell.The huge difference to open source is that you can&#x27;t just train an LLM with free time and motivation. You need lots of data and a lot of compute.I sure want to be wrong on that, I definitely like the open-weight version of the future more

Disagree with this. When cost becomes an important factor or the free but worse option becomes compelling and accessible (i.e. on device agent via apple style UX), there has been significant user behavior towards local. Think about stuff like removing backgrounds from photos, OCR on PDFs, who uses paid services for casual usage of these things?

Just as their counterparts in 1975 had no idea that &quot;personal computers&quot; were even a thing.Read through a 1970s-era issue of Popular Electronics or Byte, and then spend some time surfing &#x2F;r&#x2F;LocalLlama. You&#x27;ll get a sense of real-time deja vu, like you&#x27;re watching history unfold again.

The mainstream audience does not have the faintest idea that &quot;local AI&quot; is even a thing.

I’m just waiting for the US Government to implement their own local AI. Which will eventually lead to them open sourcing it because it’s tax payer funded and being that the NSA has decades worth of internet data they can train on; open weights would be just as good as any companies…

&gt; they don&#x27;t have a used by dateFor quite a lot of use cases, the current systems arguably do get worse over time if not continually updated. The knowledge cutoff date will start to hurt more and more as the weights age in a hypothetical scenario where you are stuck with them forever.Coding, one of the most popular usescases today, would not be great if it say only understood java to a version from years ago etc.<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Knowledge_cutoff" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Knowledge_cutoff</a>

The use cases in the future will be nothing like the use cases from today.

FOMO. A new model comes out weekly and the HN crowd debates over the minutia of changes.Pockets are too deep, it will only change once everyone is out of money.

Hardware.  Frontier labs are driving up demand so much that it&#x27;s priced significantly above cost making it far less affordable.  Just look at Nvidia&#x27;s profit margins.

They’re really not good enough, unless you consider 64 GB of memory or more consumer grade.

&gt; What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time?Uh… the hardware requirements? And stop acting like some dog shit 8B model the average Joe can run on a laptop is even close to being comparable to what Claude or even Codex can currently do.I have pretty good hardware and I’ve tinkered with the best sub-150B models you can use and they are awful compared to Anthropic&#x2F;OAI&#x2F;Grok.

&gt; They&#x27;re good enough for 95% of use casesThey&#x27;re not at all, not even close. Especially when you consider the use cases for people who are paying for LLM services today.

&gt; It&#x27;s a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control.What stops you from running the best open weighted LLMs currently available on consumer grade hardware for the rest of time? They&#x27;re good enough for 95% of use cases, and they don&#x27;t have a used by date. From what I can see, the &quot;danger&quot; is not having the next tier that comes out, but the impact of that is very low.

Bandwidth is the killer, in distributed LLM training.

I own 2 5070TI cards in a rig I would gladly donate time to for a distributed training model effort. The kicker is the training data. I would want to gate the data to anything before 2022. I don’t know how to coordinate that, but I would really like to be involved in something like this. SETI, for LLMs.

What&#x27;s the gamble here exactly? What agency do we have in it right now?

I don&#x27;t think it should be local vs cloud AI. I think local AI should be treated as a separate product. local ai should do things that really don&#x27;t need cloud AI, then cloud AI should be used as a fallback. That would reduce a lot of costs

Exactly this. The assumption that your access will last is very risky. Or that Chinese companies will keep trying to erode the economic viability of American models by open sourcing the reversed engineered models for ever is naive.

Keep the Silicon Valley pattern in mind:1. Innovate, create, and offer it all at sweetheart prices to the public while you rack up debt.2. Shovel in more money and either buy out or outlast the competition. Become dominant. Lock in your users any which way you can.3. Enshittify and cash in.The deals Anthropic, OpenAI, etc. offer won&#x27;t stay this good much longer. Don&#x27;t let them lock you in. Failing that, you should budget more for the same service. You&#x27;re going to need it. Having an open alternative running on your own hardware offers non-negligible peace of mind.

For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it&#x27;s obviously wrong for most of us.The dependency we have with anthropic and openai for coding for instance is insane. Most accept it because either they don&#x27;t care, or they just hope chinese will never stop open weights. The business model of open weights is very new, include some power play between countries and labs, and move an absurd amount of money without any concrete oversight from most people.It&#x27;s a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control.

Depending on some remote AI provider is a major lock-in pitfall. But it&#x27;s exactly what those AI providers want you to do.

Local AI is a bit like wind parks. Everyone is in favor, except if they are in your own backyard. There was recently a huge outcry when Chrome shipped a local 4 GB AI model:
<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=48019219">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=48019219</a>I have to conclude that people would like to have powerful local AI but it should at the same time only be a tiny model. In which case it wouldn&#x27;t be powerful.

Personally I wouldn&#x27;t want a couple dozen apps installed all with their own model.It seems easier to have industry specs that define a common interface for local models.I also assume the OS can, or would need to, be involved in proving the models. That may not be a good thing depending on your views of OS vendors, but sharing a single local model does seem more like an OS concern.

There is no other way than shipping your own model, because you will want an abstracted API over the inference, and you don&#x27;t know what the user has installed. Also you can ship 9b fp4 model but it all just depends

&gt; And if you have a local app, how do you take a dependency on whatever random model is installed?Why not ship your own model? In the age of Electron apps, 10GB+ apps are not unheard of.

&gt; And if you have a local app, how do you take a dependency on whatever random model is installed?Reading the tea leaves here, it will probably be common for OS’s to have built in models that can be accessed via API. Apple already does this.

Downvotes: IKR? It&#x27;s a signal that there&#x27;s a lot of motivated reasoning.I don&#x27;t think that many people have built apps against these models.I mean, I use a heavily quantized version of qwen3 for image classification, caption generation, prompt expansion etc. for image generation, instruction-driven edits, and so on. You can go a long way when you don&#x27;t need a lot.A model that can do tool calls - any tool calls at all - can look reasonably cool once you put it in a harness where there&#x27;s enough immediate context to take action. You can get carried away by anything happening at all. But golly gosh it&#x27;s a long way short of intelligence available in the bigger models.And the lighter you make your harness, giving the model more free reign, more autonomy, you get a big jump in capability combined with a big jump in failure modes when the model is dumb.

I don&#x27;t know why you are being downloaded. These are precisely the facts that advocates for local models completely ignore.Local models are absolutely going to be the future for things like simple automation and classification tasks that run occasionally and don&#x27;t need to rely on internet access.But for all of the serious stuff where you are doing knowledge work, the models will simply continue to be too big, and too slow to run locally.The article says:&gt; Use cloud models only when they’re genuinely necessary.But at least for me, they&#x27;re genuinely necessary for 99+% of my LLM usage.At the end of the day, the constraint here really is efficiency and cost.Privacy can be ensured with the legal system, the same way that businesses that compete with Google still have no problem storing their data in Google Workspace and Google Cloud. The contractual guarantees of privacy are ironclad, and Google would lose its entire cloud business overnight as its customers fled if it ever violated those contractual agreements (on top of whatever penalties they allow for).

Local models are extraordinarily expensive if you&#x27;re not maximizing throughput, and you&#x27;re not going to be maximizing it.Local models need to be resident in expensive RAM, the kind that has fat pipes to compute. And if you have a local app, how do you take a dependency on whatever random model is installed? Does it support your tool calling complexity? Does it have multimodal input? Does it support system messages in the middle of the conversation or not? Is it dumb enough to need reminders all the time?Spend enough time building against local models and you&#x27;ll see they&#x27;re jagged in performance. You need to tune context size, trade off system message complexity with progressive disclosure. You simply can&#x27;t rely on intelligence. A bunch of work goes into the harness.Meanwhile, third party inference is getting the benefits of scale. You only need to rent a timeslice of memory and compute. It&#x27;s consistent and everybody gets the same experience. And yes, it needs paying for, but the economics are just better.

It depends a lot on how you run those models. I think a lot of disagreement is because of that. A lot of people run local models with incredibly small context windows (makes an agentic LLM circle in loops), use very small quants (like 4 bit =&gt; huge degradation), don&#x27;t set the recommended parameters (like top-p&#x2F;temperature), or download GGUFs with broken chat templates. And then they claim model X is bad :)I&#x27;m currently running both Sonnet 4.6 and Qwen 3.6-27b on the same codebase (via OpenCode, the parameters were carefully tuned to have a good quality&#x2F;context size ratio), and on this project, they both struggle with complex non-trivial tasks, and both work flawlessly otherwise. Sonnet 4.6 understands the intent better if my task is ambiguously formulated, but otherwise the gap is pretty small for coding under a harness.

This.I’ve begun to suspect that most people are probably running different hardware. Sure, you run the latest deep flash on your brand new M5 128G maybe you get acceptable performance?But honestly, how many people have an extra $9000 laying around these days?Right now, running with acceptable performance is kind of a luxury. I wish the people who always say - “This is great!” - would realize that not everyone has their hardware.

&gt; Every time a thread about LLMs comes up, there are tons of people in the comments insisting that they&#x27;re getting just as good results from the latest DeepSeek&#x2F;qwen&#x2F;whatever as with Opus, and that just hasn&#x27;t been my experience at all: open-source models just fall over completely compared to Claude when asked to do anything remotely complicated.Different usage patterns - you want to issue a single spec then walk away and come back later (when it has consumed $10k worth of API tokens inside your $200&#x2F;m subscription) to a finished product.Many people issue a spec for a single function, a single class or similar. When you break it down like that, the advantages of SOTA models shrinks.

I&#x27;m also just not seeing good performance from local models. Every time a thread about LLMs comes up, there are tons of people in the comments insisting that they&#x27;re getting just as good results from the latest DeepSeek&#x2F;qwen&#x2F;whatever as with Opus, and that just hasn&#x27;t been my experience at all: open-source models just fall over completely compared to Claude when asked to do anything remotely complicated.I have a sneaking suspicion this is kinda like the situation with Linux in the 90s, where it kinda worked but it reeeeeally wasn&#x27;t ready for the home user, but you had a lot of people who would insist to your face everything was fine, mostly for ideological reasons.

We are speaking about local AI, and having all this SOTA models basically for free is blocking the progress of local or independent third party setups.

&gt; The problem is that it&#x27;s much easier to use the SOTA models (especially if they are subsidized) instead of spending time fixing the knobs with the local one.That&#x27;s not a problem, that&#x27;s a feature; I have something like 8 tabs open to different free-tier providers. ChatGPT, Claude and Gemini are the SOTA ones.I have no problem maxing one out, then moving to the next. I can do this all day, have them implement specific functions (or classes) in my code. The things is, because I actually know how to write and design software, I don&#x27;t need to run an agent in a loop to produce everything in a day, I can use the web chatbots with copy&#x2F;paste to literally generate thousands of lines of code per hour while still having a strong mental model of the code that I can go in and change whatever I need to.[1]---------------------[1] Just did that this morning on a Python project: because I designed what I needed, each generation was me prompting for a single function. So when I needed to add something this morning I didn&#x27;t even bother asking an chatbot to do it, I just went ahead directly to the correct place and did it.You can&#x27;t do that if you generate the entire thing from specs.

The path of least resistance usually wins, especially when the pricing hides the real cost

&gt; Use cloud models only when they’re genuinely necessary.The problem is that it&#x27;s much easier to use the SOTA models (especially if they are subsidized) instead of spending time fixing the knobs with the local one.I just realized this with coding agents, yeah, you probably shouldn&#x27;t always use latest version at xhigh, but you will end doing it because you do the job in less time, with less &quot;effort&quot; and basically at the same price.I guess we&#x27;ll see a real effort for local AI only when major vendors will start billing based on actual token usage.

I wish I could upvote this twice. We (devs) really REALLY need to consider on-device compute before going to the cloud for LLM inference.

Maybe don&#x27;t use gigabytes of bandwidth and storage space, without asking.

Although you can opt out and even disable the download feature when you build them in some cases, most of the local LLM tools are too download–happy by default.

This is a weird take. If its not opt in or you’re shoe horning it into a browser, then that sucks. Nobody is getting enraged that an app for running local LLMs downloads data to do so.

Did you opt into WebGPU? QUIC? Canvas 2D? Brotli? Browsers don&#x27;t work that way.

You don&#x27;t understand the difference between &quot;I run a local LLM because I chose to&quot; vs &quot;The browser chose to run a local LLM and I have no say&quot;? You don&#x27;t understand?Not to mention that the LLM that I choose to run requires a monster machine and is infinitely more capable than whatever google chose to put on their browser?I mean, none of this affects me because I don&#x27;t use chrome, obviously, but you don&#x27;t see the difference? Bewildering.

I&#x27;m guessing you immediately close the What&#x27;s New Chrome tab when you update?<a href="https:&#x2F;&#x2F;developer.chrome.com&#x2F;blog&#x2F;new-in-chrome-148#prompt-api" rel="nofollow">https:&#x2F;&#x2F;developer.chrome.com&#x2F;blog&#x2F;new-in-chrome-148#prompt-a...</a><a href="https:&#x2F;&#x2F;www.google.com&#x2F;chrome&#x2F;ai-innovations&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.google.com&#x2F;chrome&#x2F;ai-innovations&#x2F;</a>They have absolutely not been shy about any of this.

If it was such a good and laudable idea why didn&#x27;t they tell me about it before they activated it? It seems to me like they avoided it in the hopes that I wouldn&#x27;t notice, because, presumably if I had, I would have IMMEDIATELY disabled it.Also why doesn&#x27;t their task manager show that it&#x27;s actually the one downloading? Why does it go out of it&#x27;s way to hide this activity?Since I have conky on my desktop I could catch this immediately, and take the action I preferred with my own computer, which was to _immediately_ disable it.

AFAIK Apple and MS auto-download local models.

This is a bit disingenuous. People aren&#x27;t losing their shit about a local model being installed. It&#x27;s the lack of user autonomy. Just give the option to download a model instead of a silent install. It&#x27;s not that hard. This is how every other local option works.

You might want to read the comments to understand what people are actually complaining about.This comment is quite dishonest about the nature of the discussion.

If I want a model I&#x27;ll go download one. (And I did, not long ago, to play around with image generation.)

Typical HN arguing they need a button to opt in. In reality 99% people don&#x27;t care, if it works they&#x27;re fine with that.

Yet there is another post a few rows down where people are losing their shit that Chrome has a local LLM model that uses a couple of GB of space for local-inference.Damned if they do, damned if they don&#x27;t.

&quot;NO AI&quot; needs to be the norm, we should be working on better ways of sharing information and better documentation instead of fighting with computers for substandard results.

I think using the best model for every tasks makes sense when these models are subsidised. when the prices go up (assuming they do) this could trigger a more varied approach. assuming the model doesn&#x27;t self select for you.

Two issues -1. Local models are likely to be more power-expensive to run (per-&quot;unit-of-intelligence&quot;) than remote models, due to datacenter economies of scale. People do not like to engage with this point, but if you have environmental concerns about AI, this is a pretty important one.2. Using dumb models for simple tasks seems like a good idea, but it ends up being pretty clear pretty quick that you just want the smartest model you can afford for absolutely every task.

&gt; One of the current trends in modern software is for developers to slap an API call to OpenAI or Anthropic for features within their app.Well there’s your problem, control needs to go the other way. If you want your app to be AI-enabled, you need to make it easy for AI to control your app. Have you used OpenClaw? It’s awesome!

The ones that are good for more than elaborate auto-complete are pretty hefty, but it can be done. They’re still not Opus behind claude code.

This is a good snapshot of things:<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=48050751">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=48050751</a>A specialist handrolls a cut-down framework to power a 1 or 2 bit quantised version of a cut-down sort-of-frontier model.It can be yours if you have 128GB or 256GB of RAM.

Question: for software development, how much of an AI do you need for local development? Can it be run locally? Can someone train something that knows a lot about software but lacks comprehensive coverage of history, politics, and popular culture?

We need computers with 128gb or maybe even 192gb of memory before local use make sense. From my own experience 32b LLMs are the absolute minimum for proper tool use and decent output quality. But for local ai you want also vision models and maybe even various LLMs. Plus some memory for the system of course. 
On my 36gb M3 the 24b Gemma model is nice. But the entire system gets allocated for that thing.

Posts like this are so funny to me. I&#x27;m staring at a mountain of old hardware right now that cost about $20k ten years ago. I have to pay someone now to come haul it away. What makes you think the current new hardware won&#x27;t end up with the same fate.&gt; Just write your own fkin code peopleBro is nostalgic for googling random stack overflow threads for 10 days to figure out a bug the agent fixes in an hour.

No one runs SOTA models 24&#x2F;7 for individual use or even for a single household or small business, whereas you can run your own hardware basically 24&#x2F;7 for AI inference.With the new DeepSeek V4 series and its uniquely memory-light KV cache you can even extend this to parallel inference in order to hide memory bandwidth bottlenecks and increase compute intensity.This is perhaps not so useful on a 128GB or 96GB RAM Apple Silicon device (I&#x27;ve seen recent reports of DS4 runs with even one agent flow hitting serious thermal and power limits on these devices, so increasing compute intensity will probably not be helpful there) but it will become useful with 64GB devices or lower that have to stream from a slow disk, or with things like the DGX Spark or to a lesser extent Strix Halo, that greatly overprovision compute while being bottlenecked on memory bandwidth.

&gt; two 4090s is not consumer gradeI think that is a very narrow perspective. Enormous numbers of consumers own $50,000 cars, but a pair of $2000 GPUs is &quot;not consumer&quot;?I agree with your view that cheap tokens on SOTA are a trap-- people should use local AI or no AI.

It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm.Not if you&#x27;re OK with 4-bit quantization. More like $30K-$50K one time.Spring for 8 RTX6000s instead of 4, and you can use the full-precision K2.6 weights ( <a href="https:&#x2F;&#x2F;github.com&#x2F;local-inference-lab&#x2F;rtx6kpro&#x2F;blob&#x2F;master&#x2F;models&#x2F;kimi-k26.md" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;local-inference-lab&#x2F;rtx6kpro&#x2F;blob&#x2F;master&#x2F;...</a> ).

API prices are most likely not subsidised. A brief look at openrouter can tell you that. There are plenty of providers that have 0 reason to subsidise that sell models at roughly the same average price. So the model works for them (or they wouldn&#x27;t do it otherwise).

This is simply delusional, It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm.To sell tokens profitably you&#x27;d need to be able to run inference at 150 tokens per second for less than $1,000 USD a month.I don&#x27;t think people realize how expensive it is to host decently capable models and how much their use of capable models is subsidized.You can only squeeze so many parameters on consumer grade hardware(that&#x27;s actually affordable, two 4090s is not consumer grade and neither is 128gb macbooks, this is incredibly expensive for the average person, and the models you can still run are not &quot;good enough&quot; they are still essentially useless).People are betting their competency on a future where billionaires are forever generous, subsidizing inference at a 10-1 20-1 loss ratio. Guess what, that WILL end and probably soon. This idea that companies can afford to give you access to 2mm in GPUs for 5 hours a day at a rate of $200.00 a month is simply unsustainable.Right now they are trying to get you hooked, DON&#x27;T FALL FOR IT. Study, work hard, sweat and you&#x27;ll reap the benefits. The guy making handmade watches, one a month in Switzerland makes a whole lot more than the guy running a manufacturing line make 50k in China. Just write your own fkin code people.Don&#x27;t bet your future on having access to some billionaire&#x27;s thinking machine. Intelligence, knowledge and competency isn&#x27;t fungible, the llm hype is a lie to convince you that it is.

The biggest impact of local models may simply be that they prevent remote inference from becoming the only game in town

In my experience once you get to ~30 gigs of ram for a model like Gemma4, the rest of the 128g of memory is simply nice to have.  The speed and costs are what make it tough though, because its slower and more expensive than the same model served on a big accelerator card, and is going to be worse than a frontier model.

Perhaps I am the odd one out here, but a small part of me wants to see what happens when you run a proprietary SOTA model on a laptop.

I built my own IDE and run my own model specifically to have private agentic coding. I can still access model APIs but I can be purely local if I want too. It’s amazing.

This is my exact setup as well and dear lord gemma is absolutely batshit insane. I&#x27;m trying to get a self-reflection and confidence loop going now, but it does feel like it&#x27;s not the local resources, it&#x27;s the limits of the training. Dedicated coding or dedicated real-world task models would be a good optimisation.

I need to see these proper harnessesI tried oMLX and OpenCode a few weeks ago and the 65k context window was useless, it tried to analyze a very small codebase before going full on agentic and ran out of context window immediatelyI don&#x27;t have time to tweak 1,000 permutations of settings just re-prove that its not as smart as Opus 4.6I need out the box multimodal behavior as similar as typing claude in the command line and its so not there yetbut I&#x27;m open to seeing what people&#x27;s workflows are

Multiple gazillion dollar companies each seem to be spending to ensure that they alone pretty much dominate all knowledge work, with customers eating up their tokens like Cookie Monster. I wonder if the any of them could survive as LLM providers if they not only failed to do that, but the entire industry ended up selling what the current Cookie Monster would call a “sometimes snack,” for very special occasions?

Can you share how you use it to categorize trip photos!

I&#x27;m sorry to spoil it for you, but Perl script was able to do all of that like ... 10 years ago? The out-of-the-box Shotwell manages photos quite well without any intelligence. The problem, as people mentioned above, is SOTA models cognitive and tooling abilities. Also, have you noticed as top-end Mac Studios got downgraded recently? They don&#x27;t want you to have access to frontier models. And you will not have it. See Mythos as Exibit A.

&gt;It&#x27;s here, right now.I mean I&#x27;ve been forcing my good old 1080ti to run local models since a short while after llama was first leaked.But I wouldn&#x27;t say &quot;local models are here&quot; in the same way as &quot;year of the Linux desktop!111&quot;Until someone can just go out and buy some sort of &quot;AI pod&quot; that they can take home, plug in and hit one button on a mobile app to select a model (or even just hide models behind various personas) then I wouldn&#x27;t say it&#x27;s quite there yet.It&#x27;s important that the average consumer can do it, I think the limitations for that are: things are changing too quickly, ram+compute components are exceedingly expensive now, we&#x27;re still waiting on better controls&#x2F;harnesses for this stuff to stop consumers not just from shooting themselves in the foot, but blowing their foot clean off.Would be interesting to see a Taalas-like chip in a product, albeit there&#x27;s so many changes going on atm with diffusion based models, Google&#x27;s Turboquant (which as someone who has had to almost always run quantized models, makes a lot of sense to me).

Are there any harnesses that are attempting to optimize for using local models like this? Unsurprisingly, my naive attempts to integrate with harnesses designed for frontier models have gone poorly. But it seems like a harness that understands the capabilities and limitations better could perform significantly better.

I run Gemma locally on a 3090, it&#x27;s amazing how useful it is to be able to call out to ollama in a bash script or cron job.

Has anyone tried to calculate the break even cost of buying a PC to run an LLM locally, vs the amount of tokens you could get from an AI provider?

&gt; They will be, and that moment is not that far off.It&#x27;s here, right now. I&#x27;m running quantized Qwen and Gemma on a decent, but three years old gaming rig (think RTX 3080 12GB and 32 GB RAM). Yes, it&#x27;s slow, it has a small context window. But it can (given a proper harness) run through my trip photos and categorize them. It can OCR receipts and summarize spendings. It can answer simple questions, analyze code and even write code when little context is required. Probably I could get a half-decent autocomplete out of it, if I bother with VS Code integration. &quot;128 GB VRAM on a MacBook Pro or a Strix Halo&quot; is already a minimum viable setup for agentic coding, I think.&gt; And then we&#x27;ll have the equilibrium we already have with the &quot;classic cloud&quot;: you either self-host or pay for flexibility and speed.Currently, it works exactly the other way. The cloud versions are orders of magnitude cheaper than self hosting, because sharing can utilize servers much more efficiently. Company can spend half a million bucks on a rig running GLM 5.1, and get data security, flexibility and lack of censorship, but oh it&#x27;s so expensive compared to Anthropic per-seat plans.

Other than this recent spike, it&#x27;s been trending cheaper continuously for decades. In a few years 128GB will be as affordable as 12GB (what flagship phones have now) is today.

Their prices are currently so unreachable because of the big players hoarding every chip they can get their hands on, but if&#x2F;when the market realizes that locally deployed LLMs are the way to go, maybe (hopefully?) then more chips will be available to the consumers for lower prices.

How fast do you reckon most people will be able to afford 128-256GB of RAM?

I think the proportion is small because someone has to pay for the cloud services. When phones, PCs and Desktops ship with NPUs whole new markets open up for all that stuff people want but not enough to pay for.

&gt; The question will be: how much of the current compute capacity craze will local hosting give the kiss of death to and what that means for the market.This will depend on how much inference happens for consumer (desktop, local) vs enterprise (&quot;cloud&quot;), vs consumer mobile (probably also cloud).I would assume that the proportion of &quot;consumer, local&quot; is small relative to enterprise and mobile.

i would argue we don&#x27;t need anything near Opus to be productive. Sonnet is plenty productive enough

Won’t these H100s drop in price in a few years? With the data center build out surely these will become 1&#x2F;10th the price and you’ll be able to set up a local LLM as good as opus 4.7. Even if the frontier model become more advanced, and memory hungry, you could use the same power usage as your oven to run a current day frontier model as needed? If I could drop $10,000 to have an effectively permanent opus 4.7 subscription today, I would.

Parameter size gets you world knowledge and better persistence of behavior as context grows. Both of those things can be engineered around to a large degree, and the latest Qwen models show that small models can be quite smart in narrow domains and short time windows.

Jokes on you.  We are already running Deepseekv4Flash, Mimo2.5, MiniMax2.7, Qwen3-397B locally in very affordable hardware.  These models are in the real of Opus4.6.   For those of us a bit crazy, we are running KimiK2.6, GLM5.1 and more ...

It depends on what you mean for &#x27;productive&#x27;. Article mainly seems to be about targeting consumer level hardware, such as the Neural Processing Unit you need for a &#x27;Copilot PC&#x27;. Windows Recall is (was?) one such local AI application. If Microsoft get their way and my next PC has one, I look forward to using it for &#x27;productive&#x27; purposes such as playing games, handling natural language stuff and leaving my GPU free for GPUing.

Flat wrong. Q6 Gemma 31b feels a lot like opus 4.5 to me when run in a harness so it can retrieve information and ground itself. The gap is not that big for a lot of usecases. Qwen MoE is fast as fuck locally for things that are oneshottable. I have subscriptions to all the major providers right now and since Gemma 4 and Qwen 3.6 came out I haven&#x27;t hit limits a single time. I&#x27;m actually super surprised by the number of things I try with Gemma 4 with the intent of seeing how it fails and then having Claude do it only to come away with something perfectly usable from the local model.

&gt; You are greatly underestimating the current hardware requirements for productive local LLMs.Fixed that for you. Right now most models produced are based on floating point maths and probabilities, which is &quot;expensive&quot; to do math on.Microsoft has researched 1-bit LLMs which can run much more efficiently, and on much cheaper hardware[1].If this research is reproducable and reusable outside their research models, this means the cost of running self-hosted LLMs will be reduced by an order of magnitude once this hits mainstream.[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;BitNet" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;BitNet</a>

Yes and no.The best analogy is the difference between having N senior level engineers working for you, versus having N entry level engineers.With frontier cloud models, you can give a single invocation one task, and it can figure everything out.With local models, you have to manage the inputs and outputs quite a bit more, but you can achieve similar results for tasks you set up harnesses for. They are not as a good at finding the right answer internally from their own weights, but they are very capable of ingesting context and reformatting text - for example, for debugging, local models can debug issues quite well if you give them the error and documentation for a particular feature you are trying to implement.

You are greatly underestimating the hardware requirements for productive local LLMs. Research consistently shows that parameter count sets the practical ceiling for a model&#x27;s reliability. Quantized models with double digit param counts will never be reliable enough to achieve results in the realm of something like Opus 4.6.

Certainly, I don&#x27;t think Data centers are the way here.I guess, it&#x27;ll most likely be an AI processing and everything else becoming API.In case of GPTs and Claudes of the world. They&#x27;ll be just using an Indexing APIs and KB on top of their LLMs.

Except you will want the frontier to compete. Local models are useful but you will always need $$$ to be in the same order of magintude as frontier. And also $$$ for same token speed.The question is would you choose to save $10 a day if it causes your inference to slow down 10x and waste 2 hours a day waiting on stuff.

Depending on your laptop, if your laptop is a Strix Halo or a Macbook with a decent amount of ram, that day they arrived is about 6 months ago, and today if you can run Gemma 31b, you&#x27;re golden for your basic workslop code. You can do most of it with local models. Heck, for a lot of the tier of programming you might encounter in the average job Qwen 35b MoE is good enough and it can hit 100tok&#x2F;s on decent hardware.

For the same reason that a human who is fluent in five languages can probably express themselves better in either one compared to human that only speaks one, while also having a more nuanced understanding of general grammar. 
From what I know, learning on a more diverse set makes a model better overall.

Do you think small models will arrive? I mean if I need to write a web application in typescript why should I use a model that knows all the programming languages and it is able to reply to any questions about almost everything? I just a need a small performant model that knows how to write web applications in typescript. That could be very helpful and easy to run on my laptop.

I think it&#x27;s inevitable that access to good enough LLM models will be democratised.However that&#x27;s not the real battle here. The real battle is control of information to operate over.While I might have access to a decent model - I don&#x27;t have the huge integrated databases of everything that companies like Google have, and increasingly governments will accumulate.As a citizen AI operating of these large datasets is where the concern should be.

&gt; how much of the current compute capacity craze will local hosting give the kiss of death to and what that means for the market.Nvidia and other hardware sellers would love if they could sell a bunch of chips to individual consumers that would sit idle for 95% of its life.

Even on a 5090 qwen is really impressive. Felt as good as Claude for little projects.

They will be, and that moment is not that far off. We&#x27;ve got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in &quot;a bunch of servers with a couple of H100s each&quot; territory, slowly going into &quot;128 GB VRAM on a MacBook Pro or a Strix Halo&quot;. Within the next year, the pattern of &quot;expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution&quot; will become the norm for companies, slowly moving to &quot;using local LLM for everything is good enough&quot;. And then we&#x27;ll have the equilibrium we already have with the &quot;classic cloud&quot;: you either self-host or pay for flexibility and speed. The question will be: how much of the current compute capacity craze will local hosting give the kiss of death to and what that means for the market.

It&#x27;s based on a Gemma 3n model, and yeah it&#x27;s not the best. But if you have a use case that needs constrained JSON output for example, it&#x27;s pretty neat.Maybe it would do better with the new Gemma 4 models, which the Chrome devs have been hinting at moving to. And why the API doesn&#x27;t let you introspect &#x2F; pick the model, I&#x27;m still not sure.

Yea I do not recommend treating chromes prompt API as a good example of local LLMs. It&#x27;s fine and stuff but it&#x27;s really weak. 8b models from a year ago are better in some ways. And a lot of the recent model drops are meaningfully better.

So you&#x27;re running an llm to do data transformation that deterministic processes would be much better suited for and running 1,000 watt power supply to do so. Wild.

&gt; I&#x27;ve got some demos of what the new Prompt API can do: 
&gt; Use surrounding context to rewrite your ad copy:Yup, that&#x27;s the plan. No local model, no webpage; more, better and cheaper adtech extortion&#x2F;surveillance for vendors while everyone else pays for the juice and hardware degradation.

I&#x27;ve got some demos of what the new Prompt API in Chrome that uses a local model can do: <a href="https:&#x2F;&#x2F;adsm.dev&#x2F;posts&#x2F;prompt-api&#x2F;#what-could-you-build-with-it" rel="nofollow">https:&#x2F;&#x2F;adsm.dev&#x2F;posts&#x2F;prompt-api&#x2F;#what-could-you-build-with...</a>As OP says, it shines in constrained environments where the model is transforming user-owned data. Definitely less useful for anything more open-ended.

To be fair the author of the post is using the model Apple provides with the OS so it doesn&#x27;t have any extra binary size

The shitty thing here is, either everyone&#x27;s shipping 800 MB at least with their binary, or, you have to rely on the platform vendor anyway. I&#x27;m hoping there&#x27;s enough external pressure that the OS vendors turn it more into a repository than a blessed-model-garden.

Furthermore, for the example given, it would have made a lot of sense to me to generate those article summaries on the backend. Once and for all, no need to burden each client device (which are going to need to download the content anyway), no need to tie yourself to a specific provider (Apple in this case), can have the same experience everywhere. Of course, the backend could use a local (to itself) model.Not saying it’s _wrong_ either – maybe it doesn’t use a backend of its own (the client downloads content directly from some predefined set of sites), maybe there is functionality to adjust how the summaries work that benefit from doing it on device, etc. Just doesn’t convince me that ”local AI should be the norm”.

&gt; We are building applications that stop working the moment the server crashes or a credit card expiresIsn’t this true of any application that accesses anything not running on your computer? This is just describing what it means to add an API call to your app. Nothing to do with AI (?)

agree with the article but the limitation for local llm usefulness is the limited scope from my experiments. eventually context heavy data pipelines require larger models which consumer hardware can&#x27;t deal with yet. the local model for summary on a page like you describe could be done via code as well, i&#x27;ve found using an llm isn&#x27;t always the right choice. for example i use ner tagging in my md docs for better indexing and llm search capabilities. this is purely code based and not via an llm. tried with an llm and the results were a lot worse. augmenting tools to make the llm produce better outputs gives better results.

i researched that question for apfel <a href="https:&#x2F;&#x2F;github.com&#x2F;Arthur-Ficial&#x2F;apfel" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Arthur-Ficial&#x2F;apfel</a> and standardized API is openai api so thats what i went with

&gt; What is the answer to local AI for Linux?run an ai api endpoint on a unix domain socket

I would like a standardized API for local AI to exist outside of the Apple ecosystem. The Prompt API is Chrome is halfway there.* What is the answer to local AI for native apps on Windows?* What is the answer to local AI for Linux?This is a big opportunity for Linux, given the high quality of open-weight models. I hope some answer emerges before designs fracture and we get a dozen mutually incompatible answers.

Yes. For essentially any problem where a complete solution exists that doesn&#x27;t use an LLM, it will beat any solution that does in size, speed, energy use, reliability and everything else.Naturally, it&#x27;s actually complicated. But LLM is a considerable weight and risk. Maybe it&#x27;s worth involving and maybe not.

If you don&#x27;t need a lot of smarts, do you even need an LLM?  Aren&#x27;t older machine learning techniques just as good, or like, you know, old-school algorithms?

Here I was hoping that this was some plea for us to get away from proprietary solutions that we have no control over and go back to open source, but no, not that at all.

Will there even be a web to search in the future? These days public access blogs are dying and being replaced with hallucinated AI websites. Sites with original research like Reddit and YouTube are being locked up to prevent 3rd party indexing.Knowledge and clean data sets are becoming increasingly valuable, and free community knowledge is drying up. The next big programming language won’t have years of Stack Overflow posts to train on.Maybe we will see some kind of licensing deals where owners of good datasets charge you a fee to let your AI search them.

The example in the post confirms my theory that for local models to succeed they need to be &quot;good enough&quot;, not big enough that they can compete with frontier models.They need to be able to do a small task well and they need to be able to run reasonably on consumer-class devices. Even better if they can run on mobile phones.In my experiments with local LLMs I noticed that while increasing the size of the model is nice the real thing that turns a barely useless model into something useful is the ability to use tools.
Giving my models the ability to search the web and fetch web pages did way more to solve hallucinations than getting a bigger model. And it doesn&#x27;t have a training cutoff.
Sure, the bigger model is probably better at using tools but I often find the smaller models to be good enough.

I mostly agree, though I think local AI will need better UX around failure modes. Cloud models are often used not just because developers are lazy, but because they are more capable and easier to support consistently across devices.

Another option is verifiably private inference with open source models running inside secure enclaves on the cloud (using NVIDIA confidential computing), and the enclave code is open source and verified via remote attestation upon connection, cryptographically proving that the inference provider cannot see any data. Tinfoil: <a href="https:&#x2F;&#x2F;tinfoil.sh&#x2F;">https:&#x2F;&#x2F;tinfoil.sh&#x2F;</a> is a good example of this (disclaimer: i&#x27;m the cofounder). You can read more about how this works here: <a href="https:&#x2F;&#x2F;docs.tinfoil.sh&#x2F;verification&#x2F;verification-in-tinfoil">https:&#x2F;&#x2F;docs.tinfoil.sh&#x2F;verification&#x2F;verification-in-tinfoil</a>&gt;that open models are in the ballpark of the best commercial modelsThis is basically true for certain tasks. As an example, chat interfaces are not well poised to take advantage of higher model intelligence than what the best open source models already provide. But coding harnesses still benefit from greater model intelligence and even more so, the reinforcement learning that tightly interlinks the provider&#x27;s coding harness (claude-code, codex) with the model&#x27;s tool calling interfaces is another reason for discrepancy in effectiveness even when controlled for model intelligence. The opencode founder (open source coding harness that supports different model providers) was recently complaining about the challenges making the harness work well with different providers: <a href="https:&#x2F;&#x2F;x.com&#x2F;thdxr&#x2F;status&#x2F;2053290393727324313" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;thdxr&#x2F;status&#x2F;2053290393727324313</a>

I think we should separate the private AI discussion from the local AI discussion. 
The pragmatic choice to run big LLMs is one&#x2F;several big servers online, but that doesn&#x27;t mean private companies should be the only ones to run them.A self hosted inference solution that offer good tenant isolation guarantees (ideally zero trust) and is easy enough to deploy and maintain (think Plex for AI) would be my choice for privacy. Now to be honest I have done zero research about this and have zero idea how feasible that is, maybe it already exists and there&#x27;s some discord servers I should join?Edit: I don&#x27;t need to mention it here but what&#x27;s incredible is that open models are in the ballpark of the best commercial models so supposedly, the hardest part by far is already solved.

While I agree that would be the goal, we are too early for that. Just like how speech recognition used to require many server in a Datacenter to process and you send your data over. It is now completely on devices.We are at least 5 years away from that. And DRAM needs a substantial breakthrough in cost reduction.

Consumer&#x2F;private needs to be local.Work? I don&#x27;t want it local at all. I want it all cloud agent.

Entrenched interests are going to do everything to stop local, but there&#x27;s at least a few technical reasons to believe small and specialized models could be the norm eventually. If that does happen, local will follow.TFA is focused on whether big models are necessary for what users want. There&#x27;s some evidence they may never actually be reliable enough unless a) mechanistic interpretation matures far enough or b) our multi-agent systems all become multi-model.For (a), advancement in MI might fix problems with big models, but would also mean we can maybe get unified representations, and just slice and dice the useful stuff out of huge models, getting only what we need without the junk. Ability to isolate problems won&#x27;t really come without bringing the ability to isolate functional subsystems. Only want logic? Only vision? Just cut it out of the big monster and enjoy reduced costs and surface area for problems.For (b), just look at stuff like the evil vector, or the category of hallucinations specific to tool-use. Without a complete solution for helpful&#x2F;honest&#x2F;harmless alignment, it seems likely that creativity and rigor (and many other things) are fundamentally at odds. If you start to need many models for everything anyway, why do we need the huge expensive do-everything ones? So specialization also becomes a pressure to shrink everything towards minimal reliable experts

&gt; So the DGX Spark is a joke, really.Not at all sure about that. They have really good compute, and DeepSeek V4 (with antirez&#x27;s 2-bit expert layer quant) may be able to leverage that compute via parallel inference - the jury is still out on that. Now if you had said Strix Halo&#x2F;Strix Point or perhaps the Intel close equivalents, that would&#x27;ve been a slightly stronger case.

&gt; So many expect an M5 Max Mac Studio with 1TB&#x2F;s+ bandwidth and specs up to 256GB or 512GB, probably for ~$10k later this year.This is what I&#x27;m really waiting for. It will enable models comparable to current SOTA at the enthusiast price range.

I&#x27;ve been looking into options for this and we are getting close. There are two main constraints: memory and memory bandwidth.NVidia segments the market by limiting the amount of memory on GPUs. It currently tops out at 32GB (on a 5090) but it has excellent memory bandwidth (~1.8TB&#x2F;s). If you want more than the you need to buy an RTX Pro (eg RTX 6000 Pro w&#x2F; 96GB for ~$10K) or you get into high high end solutions like H100, H200, etc that have significantly more memory and even higher bandwidth on HBM memory (eg 3.2TB&#x2F;s+).NVidia has released the DGX Spark w&#x2F; 128GB of memory for ~$4k. The problem is the memory bandwidth. It&#x27;s only 273GB&#x2F;s, which is less than the M5 Pro (307GB&#x2F;s) but more than the M5. You can buy a 16&quot; Macbook Pro with an M5 Max and 128GB of memory for $6k and it has a bandwidth of 614GB&#x2F;s. So the DGX Spark is a joke, really.In case it wasn&#x27;t clear, Apple is interesting in this space because it has a shared memory architecture so the GPU can use all the memory.Many, myself include, expect there to be no refresh to the 5000 series consumer GPUs this year, which would otherwise happen based on product cycles. So no 5080 Super, for example. And I wouldn&#x27;t expect a 6090 before 2028 realistically.One thing Apple hasn&#x27;t done yet is release the M5 Mac Studios, which are widely expected in Q3 this year. They are interesting because, for example, the M3 Ultra has a memory bandwidth of 819GB&#x2F;s and previously had a max spec of 512GB but that got discontinued (and the 256GB version also got discontinued more recently).So many expect an M5 Max Mac Studio with 1TB&#x2F;s+ bandwidth and specs up to 256GB or 512GB, probably for ~$10k later this year.You really have to use this hardware almost 24x7 for it to be economical because otherwise H100 computer hours are probably cheaper.But what happens when the next generation of GPUs comes out to the trillions in AI DC investment? It&#x27;s going to halve its value. That&#x27;s over $1 trillion in capex that will disappear overnight, effectively.I think Apple is the dark horse here because they have no interest in NVidia&#x27;s psuedo-monopoly. I&#x27;m just waiting for them to realize it.Now CUDA is an issue here still but I think as time goes on it&#x27;s going to be less of an issue. Memory is still a huge constraint both in terms of price and just general supply because NVidia can justify paying way more for it than you can, probably.It&#x27;s still sad to see that 128GB (2x64GB) DDR5 kits are almost $2k now and werre $400 a year ago. Expect that to continue until this bubble pops (which IMHO it will) and we&#x27;re likely in a global recession.So the other issue is models. OpenAI and Anthropic are built on proprietary models. Their entire valuation depends on this moat. I don&#x27;t think this last so both companies are doomed because open source models are going to be sufficiently good.We can already do some reasonably cool stuff on local hardware that isn&#x27;t that expensive and even more so once you get to $5-10k hardware. That&#x27;s going to be so much better in 2 years that I&#x27;m hesitant to spend any amount of money now.Plus the code for running these things is getting better. Just in the last month there have been huge speed ups in local LLMs with MTP.

Local LLMs is the only thing viable and probably the only thing it will remain once the hype dies down.A smaller cheaper local model can delivery most the value for coding, while we still use some services for code review and security compliance.Once the VC money runs out and they start to charge the real price, the C-level will have to impose budges or limits. The current pissing contest over who can expend the most tokens is both ridiculous and shortsighted

Agreed, but the way ram prices are going, I don&#x27;t think we would be able to afford hardware that can run any useful model.

Who can afford to backup their own photos?who can afford a house?

It&#x27;s a good question, although I think hard to quantify.If you are simply measuring Watt Cost per Token, you are missing the mark drastically. You have to measure quality output per Watt.It sounds reasonably difficult to benchmark this, maybe I&#x27;m wrong though.

Local models are much less energy efficient right?

Overall I&#x27;m bullish on standardized local APIs that ship with the browser or platform. Far more tractable than expecting end users to stand up their own local model instances, though r&#x2F;LocalLLaMA is a fantastic community to follow if you want to go that route.A useful framing over “local vs cloud AI” can be split along two axes: does the task touch private data, and does it need frontier intelligence? You can use frontier models for developing the software (doesn’t touch data), but open-source models running locally for ops: maintenance, debugging and monitoring (touches data). If you need to fall back to frontier intelligence at some point for a particularly hard to resolve problem, you can still rely on local models for pre-transforming and filtering input in a way that&#x27;s privacy-preserving or satisfies some constraint before it’s sent off to the cloud for processing. OpenAI&#x27;s privacy filter is a good example of a model that can be used to mask PII and secrets and that can run locally: <a href="https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;introducing-openai-privacy-filter&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;introducing-openai-privacy-filter&#x2F;</a>, before sending any data externally for processing.Another framing for local vs frontier closed which the article mentions is whether the task saturates model capability. With certain tasks like PDF processing or voice or summarization, adding more intelligence isn&#x27;t necessarily useful. Arguably we&#x27;ve approached that point for chat interfaces already with frontier open-source models. But for coding and ops through well structured tool use inside a coding capable harness, we&#x27;re still a ways away.Tangentially, a contrarian take here is that AI can actually enable more privacy preserving software if you’re so inclined. You can just build personalized software and it lowers the barrier to entry and the effort required to self host. SaaS complexity often comes from scaling and supporting features for all types of customers, and if you&#x27;re building software for personal use, you don&#x27;t need all that additional complexity. Additionally, foundational and infra software that is harder to vibecode with AI is often already open source.

There was never a better time to run LLMs locally. It&#x27;s just a few commands from zero till a fully working LLM homelab.```
harbor pull unsloth&#x2F;Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL# Open WebUI -&gt; llama.cpp + SearXNG for Web RAG + OpenTerminal as sandbox
harbor up searxng webui llamacpp openterminal
```That&#x27;s it, it&#x27;s already better than Claude&#x27;s or ChatGPT&#x27;s app.

&gt; Most people won’t settle for a product if it’s not the very best and incredibly convenient.... uh?

People are trying to “make the best software”, though.I think the Quixotic accelerationists of AI are more or less a vocal minority of the people who make software, and the choice of online APIs over local systems is largely a choice made for users, rather than developer’s laziness.You can do more and better with private AI today than with local models. There is no getting around that. Even if local AIs get better, being on the cutting edge of LLM performance is often a very worthy investment.Most people won’t settle for a product if it’s not the very best and incredibly convenient. That’s a high bar, and local AI often doesn’t meet those standards.HN’s insistence on treating all users like they are open-source, privacy-first, self-hosted Linux fanatics is painfully corny.

Local AI will catch up. Unless we can&#x27;t get our hands on hardware anymore, which is a legitimate concern I have.

Sounds great, but if you din&#x27;t cave to apple&#x2F;google (eg: graphene, lineage), models are not built-in. Every app needs to ship their own models, and they are not tiny.Is there a solution for this?
I&#x27;m currently just making users download onnx models if they want a feature, but it&#x27;s not smooth UX

Not your weights not your brain. Owning your own action and decision model is super important as these models emulate more of our decisions, thinking and learning. Built claudectl - a local brain for coding agents
<a href="https:&#x2F;&#x2F;github.com&#x2F;mercurialsolo&#x2F;claudectl" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;mercurialsolo&#x2F;claudectl</a>

I think with turbo quant forks eventually being merged, its becoming more feasible on mid tier consumer h&#x2F;wDont quite think its ready yet.

Could be an opportunity for a business? Except nobody ever wants to pay for software

The roadblock to this is you seem to have to build it yourself. I&#x27;ve noted that none of the current cloud models are very good at building a replacement for themselves, and there&#x27;s significant work that needs to be done to make a local LLM reliable in any way. I haven&#x27;t found a single standalone package that makes setting them up easy. Sure, I can run Hermes Agent and a model, but getting the self-reflection loop in and all of the other stuff the need to actually be good? I&#x27;m still at it, trying to get anything to work reliably and factually.

Tbf I&#x27;ve always hated that about HN formatting as it&#x27;s not very clear at all that that&#x27;s how it works.If there&#x27;s a newline in my comment, why not retain it? Whyyyyy?!

Please add double new lines as your formatting for the bullet point list makes it all one paragraph.

RAG on every machine, and the means for corporations &#x2F; shadowy powers to query it.

It depends a lot on the specific task and specific model (and version (and quant..)). For example, Phi 4 mini is good for math and logic, and the reasoning version is surprisingly good at tool calling&#x2F;RAG, but the family sucks at everything else. Gemma 4 and Qwen 3.5 are well known for having fantastic general-purpose models in the 4B-9B range, but at the lower end they actually suck again because it&#x27;s just scaled down, and so will probably loop 50% of the time at the smallest sizes. For the very small (350M-1.2B), LiquidAI&#x27;s LFM 2.5 uses novel techniques to eliminate doom loops, and they have a vision variant - but it&#x27;s still a tiny model, so don&#x27;t try to code with it. And when you just want basic tool calling, even &quot;old&quot; models like llama 3.2, gemma 2, qwen 2.5, are good, fast and low-memory. If you do more searching you can find specific models that are the best at specific tasks in a given size range.For any model where you notice looping, tune the LLM settings. Reduce temperature and top_p, increase presence&#x2F;frequency penalty, reduce context size. If you have a specific task to do, fine-tuning is the absolute best way to both reduce memory usage and boost performance and quality. Remember that tiny models are not designed for 0-shot&#x2F;1-shot, they need lots of specific instruction and context in the prompt, with multi-shot prompts having a dramatic effect on output quality. Try to keep your prompt to specific tasks. Think of small models as children, SOTA models as experienced professionals, and middle-of-the-road models as an average adult; you give the bigger ones more responsibility&#x2F;agency, but more rules and guardrails to the little ones.For coding you do want the biggest model you can fit, so this is where larger RAM shines (32GB+ iGPU). If you can fit a dense model, do that. MoE is ok but will perform better on narrower tasks. Use the bleeding edge forks of llamacpp for turboquant&#x2F;etc and Multi-Token Prediction.The last thing is quants. If you&#x27;re running something that isn&#x27;t the bare model (like an unsloth dynamic quant), model performance is gonna suffer the smaller you go, and smaller models will be much more affected. So try to max out the amount of memory you can dedicate to the model, and pick larger quants like Q6&#x2F;Q8. You can quant the k&#x2F;v cache but that also may have a negative effect. And again, if you can fine-tune for a task, you will gain much more performance and quality and reduce memory.

Would you share your experience of the used models? I have quite some experience with the larger models but the smaller ones tend to loop around or just fail on their tasks...

&gt;it really doesn&#x27;t need to be localI urge you to reconsider this attitude. If AI has a tenth the significance people claim, you&#x27;re signing away your life; your ideas, your privacy, your very sovereignty of mind, all under someone else&#x27;s control and revocable at any moment. Don&#x27;t move your brain to the cloud.Imagine an alternate timeline where we never had personal general purpose computers, only dumb terminals to access corporate servers on subscription. Don&#x27;t vote for that world with your wallet, today.Don&#x27;t be a cloudhead!

Here&#x27;s some things you can do right now with local models on a consumer device:- text-to-speech
 - speech-to-text
 - dictionary
 - encyclopedia
 - help troubleshooting errors
 - generate common recipes and nutritional facts
 - proofread emails, blog posts
 - search a large trove of documents, find information, summarize it (RAG)
 - manipulate your terminal&#x2F;browser&#x2F;etc
 - analyze a picture or video
 - generate a picture or video
 - generate PDFs, documents, etc (code exec)
 - simple programming
 - financial analysis&#x2F;planning
 - math and science analysis
 - find simple first aid&#x2F;medical information
 - &quot;rubber ducking&quot; but the duck talks backA quarter of those don&#x27;t need more than a gig of RAM, the rest benefit from more RAM. Technically you don&#x27;t even need a GPU, it just makes it faster. I do half that stuff on my laptop with local models every day.That said, it really doesn&#x27;t need to be local. I like the idea that I can do all that stuff offline if I&#x27;m traveling, but I usually have cell service, and the total tokens is pretty cheap (like $2&#x2F;month for all my non-coding AI use).

This article makes 0 sense. Its not up to billing or computer systems or ease of use or anything else that matters. The question is will the scaling laws, which in the asymptote are likely the laws of physics, hold up in converting energy to smarter models. Its not really up to anyone, the labs or developers, to choose if local or remote models will be the norm.

I&#x27;ve been fooling with the Apple Foundation model for AlliHat, so you can chat with it from a Safari sidebar instead of just Claude. It&#x27;s passible for some basic things like summarizing a page. But it really reminds me of Claude from like 3 years ago. I was trying to get it to generate synonyms for me and it would only generate about 10 with some duplicates. And when I asked for more, it said it would be a waste of resources to generate more. It has some kind of &quot;act responsible&quot; thing that Claude seemed to have. I also asked it to help me come up with synonyms for the game Pimantle, and it decided Pimantle was related to the adult industry and no matter how many times I said &quot;it&#x27;s just a game&quot; or &quot;I think you&#x27;ve misunderstood&quot;, it was stuck on not helping me with anything related to adult websites. And recommended I should play Wordle instead.All of this being said, it seems Claude gave up this &quot;constitution&quot; it used to train on? I remember trying to get it to help me code some video editing tools, and it was convinced I was pirating videos and so wouldn&#x27;t help me anymore in that session.

I&#x27;m building a protocol and router runtime for hybrid local&#x2F;cloud AI.The goal is that you would assign roles to models based on tasks, capabilities and observed performance. The router would then take care of model selection in the background.It&#x27;s tricky though. Probably have another two weeks before I can release the runtime.I have a preview up at <a href="https:&#x2F;&#x2F;role-model.dev&#x2F;" rel="nofollow">https:&#x2F;&#x2F;role-model.dev&#x2F;</a>You can follow me on Twitter if you want updates (see profile)

The way I interpret the drama over the Chrome model is that for a large chunk of users, perhaps the majority, Chrome is the OS, and this 4GB model will be their OS Level feature for local AI.

Interesting to see this after the recent post about Chrome’s on-device model using up 4gb of storage, which frustrated a lot of people [1].I agree local models are great, and it’s cool that Apple has models built in now. But I feel like it basically has to be an OS level feature or users are going to get upset. I’d certainly rather have a small utility call out to OpenAI than download its own model.[1]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=48019219">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=48019219</a>

From what I read, ds v4 is very close with opus 4.6 performance.

DeepSeek V4 with 1 million token context window is pretty powerful, although still not there. There&#x27;s hope that Opus 4.5 level performance locally is not that far away.

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I&#x27;m right there with you.Until then, I&#x27;m going to keep sending my JSON to the server farm in Virginia because it&#x27;s the only place that can serve me a model that actually works for my uses.

Local AI needs to be the norm