Everyone is getting disrupted by AI sooner or later.The trick is to use AI to do things it would take you five lifetimes to learn. It&#x27;s a tool to lower opportunity cost, financial capital, and human capital. That gives anyone leveraging it a much bigger platform, and the ability to dream big without resources.If you can become your own &quot;studio&quot;, you&#x27;re the indie artist of the future. You don&#x27;t need Disney or Universal Music backing.Anyone can step up and do this. The artists being threatened can use these tools to do more than they&#x27;ve ever done by themselves.

And the rest of the world was better off for it.If we&#x27;d prevented new technologies from influencing our artwork, our paintings would never have left the cave wall. I&#x27;m a musician with live published albums as well; if there comes a time when I think AI will help with my creative process, you can bet that I&#x27;ll be using it.

I&#x27;m already using AI to make my music production process more efficient! Namely, I&#x27;m using a program called Sononym which listens to my tens of thousands of audio samples and lets you search by audio similarity through the entire library, as well as sort by various sonic qualities.I think I&#x27;d still go for a human artist for a bigger release such as an album! It&#x27;s a lot less hassle than sorting through (often rubbish) AI output &amp; engineering your prompts, though it does cost £££ which is the main thing making it prohibitive for single releases.

&gt; artist who can&#x27;t afford to iterate his cover art ideas multiple times with a professional finds a creative solution

&gt; artist whose domain has not yet been disrupted by AI fires artist in favor of AI

I&#x27;ve been using Stable Diffusion to generate cover images for the music I release &amp; produce for others. It&#x27;s a massive time saver compared to comping together the release art using image editing software, and a lot cheaper than working with artists, which just doesn&#x27;t make sense financially as an independent musician.It&#x27;s a little bit difficult to get what you want out of the models, but I find them very useful! And while the output resolution might be quite low, things are improving &amp; AI upscaling also helps a lot.

This is such an interesting read. It makes a compelling case, though how the likes of Google should react feels less like an adjustment and more like a revolution.

Is there any reason to think that zero-shot learning and better models&#x2F;more effient AI won’t drastically reduce those costs over time?

Think a little more laterally.If we&#x27;re talking about doing everything well, I think that&#x27;s true. However, if I want to create my own personal &quot;word calculator,&quot; I could take, for example, my own work (or Hemingway, or a journalist) and feed an existing OSS model my of samples, and then take a set of sources (books, articles, etc), I might be able to build something that could take an outline and write extended passages for me, turning me into an editor.A company might feed its own help documents and guidance to create its own help chat bot that would be as good as what OpenAI could do and could take the customer&#x27;s context into the system without any privacy concerns.A model doesn&#x27;t have to be better at everything to be better at something.

Yes, price = quality because they require supercomputing resources to train. GPT-3 required hundreds of Tesla GPUs running for several weeks. That&#x27;s millions of dollars just for hardware, not including power (the GPUs cost $15k each)

&gt; Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train!Because surely price = quality. Solid argumentation there.

This is hugely misleading. If your bot just memorizes Shakespeare and output segments from memorization, of course nobody can tell the difference. But as soon as you start interacting with them the difference can&#x27;t be more pronounced.

Exciting to see open source catching up, but seems to still be a big difference in factuality and common sense between ChatGPT and Koala. Eg:Who built AlexNet on the ImageNet dataset?ChatGPT: AlexNet was built by a team of researchers led by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto. They entered the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 and won by a large margin, revolutionizing the field of computer vision with their deep convolutional neural network architecture. The AlexNet architecture consisted of eight layers, including five convolutional layers, two fully connected layers, and a softmax output layer.Koala: AlexNet was built by Google DeepMind, a London-based artificial intelligence laboratory. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an annual large-scale image and video dataset that was created for the ImageNet project by the ImageNet Large Scale Visual Recognition Challenge, which is an annual competition for image and video analysis.

From the article:&quot;April 3, 2023 - Real Humans Can’t Tell the Difference Between a 13B Open Model and ChatGPTBerkeley launches Koala, a dialogue model trained entirely using freely available data.They take the crucial step of measuring real human preferences between their model and ChatGPT. While ChatGPT still holds a slight edge, more than 50% of the time users either prefer Koala or have no preference. Training Cost: $100.&quot;

Once distributed training is solved, all those big LLMs will be left in the dust.

I&#x27;d agree they aren&#x27;t close, but they are way better than I expected to see in a short few months. At this rate they&#x27;ll be approaching &quot;good enough&quot; for me pretty soon. I don&#x27;t always need a dissertation out of it unless I&#x27;m fooling around. I want quick facts and explainers around difficult code and calculations. Been playing with Vicuna-7b on my iPhone through MLC Chat and it&#x27;s impressive.I use DDG over Google for similar reasons. It&#x27;s good enough, more &quot;free&quot; (less ads), and has better privacy.

All the things you mentioned make it a platform, but even as a model, none of the smaller open-source models come close to GPT 3.5 or 4 in my experience. You can test it by using the GPT3.5 or 4 with their API. They outputs are waaaay better than anything I get from the open source models.

Are you sure? I have yet to see any evidence that anyone at all (including Google) has built a model (or a &quot;platform&quot; as you prefer to refer to them) that can follow instructions as well as 50% of ChatGPT, let alone GPT-4. I don&#x27;t think any amount of work in LangChain and vector databases is enough to fix this: you really need a strong base model that is trained to align with human intentions well. Of course if you just want a bot that can answer free-form simple questions, then maybe people can&#x27;t tell the difference. Just give them some real work to do and it becomes glaringly obvious.

If all you’ve done is download the model and perform basic prompts then I understand why you think this. There is a lot more going on behind Bard and GPT than a chat window passing the inputs to the model.Edit for clarity: You’re comparing a platform (Bard, GPT) to a model (llama, etc). The majority of folks playing with local models are missing the platform.In order to close the gap, you need to hook up the local models to LangChain and build up different workflows for different use cases.Consequently, this is also when you start hitting the limits of consumer hardware. It’s easy to download a torrent, double click the binary and pass some simple prompts into the basic model.Once you add memory, agents, text splitters, loaders, vector db, etc, is when the value of a high end GPU paired with a capable CPU + tons of memory becomes evident.This still requires a lot of technical experience to put together a solution beyond running the examples in their docs.

Cringe, haven’t seen a single Open Source come even close to the ability of Bard, let alone ChatGPT. Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train!

I think that analogy is flawed to try to undercut OpenAI’s lead. The reason it’s flawed is that the search business is really lucrative and OpenAI is trying to completely disrupt Google’s business there. So while the AI isn’t a moat, establishing a lead in search is because you obviously will use that to inject ads in the commercial space and capture the market.

&gt; end up like vacuum cleanersThe term of art is Generic Trademark<a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Generic_trademark" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Generic_trademark</a>In US common law (and I&#x27;d imagine UK too), it&#x27;s usually something companies want to avoid if at all possible.Relevant case for Google itself: <a href="https:&#x2F;&#x2F;www.intepat.com&#x2F;blog&#x2F;is-google-a-generic-trademark&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.intepat.com&#x2F;blog&#x2F;is-google-a-generic-trademark&#x2F;</a>

Consumers recognizing ChatGPT might just end up like vacuum cleaners; at least in the UK, people will often just call it a &quot;hoover&quot; but the likelihood of it being a Hoover is low.It is difficult to see where the moat might exist if it&#x27;s not data and the majority of the workings are published &#x2F; discoverable. I don&#x27;t think the document identifies a readily working strategy to defend against the threats it recognises.

I&#x27;ll also mark myself as skeptical of the brand-as-moat. I think AskJeeves and especially Yahoo probably had more brand recognition just before Google took over than ChatGPT or openai has today.

&gt; in a context where the average consumer is starting to recognize ChatGPT as a brand name.That brand recognition could hurt them, though. If the widespread use of LLMs results in severe economic disruption due to unemployment, ChatGPT (and therefore OpenAI) will get the majority of the ire even for the effects of their competition.

&gt; ChatGPT as a brand nameYou&#x27;re forgetting the phenomenon of the fast follower or second to market effect. Hydrox and Oreos, Newton and Palm, MySpace and Facebook, etc. Just because you created the market doesn&#x27;t necessarily mean you will own it long term. Competitors often respond better to customer demand and are more willing to innovate since they have nothing to lose.

&gt; context where the average consumer is starting to recognize ChatGPT as a brand name.Zoom was once that brand name which was equated to a product. Now, people might say &quot;Zoom call&quot;, but may use Teams or Meet or whatever. Similarly, people call a lot of robot vacuum cleaners Roombas, even though they might be some other brand.Brand recognition is one thing, but the actual product used will always depend on what their employer uses, what their mobile OS might use, or what API their products might use.For businesses, a lot will be about the cost and performance vs &quot;the best available&quot;.

Great read, but I don&#x27;t agree with all of these points. OpenAI&#x27;s technological moat is not necessarily meaningful in a context where the average consumer is starting to recognize ChatGPT as a brand name.Furthermore, models which fine-tune LLMs are still dependent on the base model&#x27;s quality. Having a much higher quality base model is still a competitive advantage in scenarios where generalizability is an important aspect of the use case.Thus far, Google has failed to integrate LLMs into their products in a way that adds value. But they do have advantages which could be used to gain a competitive lead:
- Their crawling infrastructure could allow their to generate better training datasets, and update models more quickly.
- Their TPU hardware could allow them to train and fine-tune models more quickly.
- Their excellent research divisions could give them a head start with novel architectures.If Google utilizes those advantages, they could develop a moat in the future. OpenAI has access to great researchers, and good crawl data through Bing, but it seems plausible to me that 2 or 3 companies in this space could develop sizeable moats which smaller competitors can&#x27;t overcome.

Yes, but so far we&#x27;ve seen universities, venture-backed open source outfits, and massive collections of hobbyists train all sorts of large models.

It&#x27;s cheap to distill models, and trivial to scrape existing models. Anything anyone does rapidly gets replicated for 1&#x2F;500th the price.

Doesn&#x27;t the sheer cost of training create a moat on its own?

Agree, misleading title. The introduction makes the context clear, but probably too late to not call the article click-bait.&gt; [...] It originates from a researcher within Google. [...] The document is only the opinion of a Google employee, not the entire firm. [...]

Completely agree. It is interesting, but the gravitas of it seems lower than of course if an executive said this and corroborated it. I do feel that opensource for AI is going to be really interesting and shake things up.

99% of media coverage like “Tech employee&#x2F;company says &lt;provocative or controversial thing&gt;” are exactly like that.

And (probably) through no fault of their own they&#x27;ll get totally thrown under the bus for this--whether directly but when raises&#x2F;promotions come around or not.

This looks like a personal manifesto from an engineer who doesn&#x27;t even attempt to write it on behalf of Google? The title is significantly misleading.

I&#x27;m not sure? Placing ethics constraints on a company under a capitalist system is hard. Placing them on open source is impossible.

Seems to be the Open Source who is the real winner overall.. After OpenAI became basically ClosedAI it&#x27;s an excellent news

FWIW I posted Simon&#x27;s summary because it&#x27;s what I encountered first, but here&#x27;s the leaked document itself[0].Some snippets for folks who came just for the comments:&gt; While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.&gt; A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other.&gt; This recent progress has direct, immediate implications for our business strategy. Who would pay for a Google product with usage restrictions if there is a free, high quality alternative without them?&gt; Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.&gt; And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.[0]: <a href="https:&#x2F;&#x2F;www.semianalysis.com&#x2F;p&#x2F;google-we-have-no-moat-and-neither" rel="nofollow">https:&#x2F;&#x2F;www.semianalysis.com&#x2F;p&#x2F;google-we-have-no-moat-and-ne...</a>

Fortunately, what I said was that a single company becomes the sole source for the ML in education; not the same thing and thus I have no conflict with your statement.

ML dominating education seems pretty realistic to me. E.g. this series of prompts for example:&gt; &quot;Please design a syllabus for a course in Computer Architecture and Assembly language, to be taught at the undergraduate level, over a period of six weeks, from the perspective of an professor teaching the material to beginning students.&quot;&gt; &quot;Please redesign the course as an advanced undergraduate six-month Computer Architecture and Assembly program with a focus on the RISC-V ecosystem throughout, from the perspective of a professional software engineer working in the industry.&quot;&gt; &quot;Under the category of Module 1, please expand on &quot;Introduction to RISC-V ISA and its design principles&quot; and prepare an outline for a one-hour talk on this material&quot;You can do this with any course, any material, any level of depth - although as you go down into the details, hallucinations do become more frequent so blind faith is unwise, but it&#x27;s still pretty clear this has incredible educational potential.

Repeating myself from <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35164971" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35164971</a> :&gt; OpenAI can&#x27;t build a moat because OpenAI isn&#x27;t a new vertical, or even a complete product.&gt; Right now the magical demo is being paraded around, exploiting the same &quot;worse is better&quot; that toppled previous ivory towers of computing. It&#x27;s helpful while the real product development happens elsewhere, since it keeps investors hyped about something.&gt; The new verticals seem smaller than all of AI&#x2F;ML. One company dominating ML is about as likely as a single source owning the living room or the smartphones or the web. That&#x27;s a platitude for companies to woo their shareholders and for regulators to point at while doing their job. ML dominating the living room or smartphones or the web or education or professional work is equally unrealistic.

&gt; The premise of the paper is that while OpenAI and Google continue to race to build the most powerful language models, their efforts are rapidly being eclipsed by the work happening in the open source community.Another magnificent unsurprising set of correct prediction(s) [0] [1] [2] and as triumphantly admitted by Google themselves on open source LLMs eating both of their (Google) and OpenAI&#x27;s lunch.&quot;When it is the race to the bottom, AI LLM services, like ChatGPT, Claude (Anthropic), Cohere.ai, etc are winning the race. Open source LLMs are already at the finish line.&quot;[0] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=34201706" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=34201706</a>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35661548" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35661548</a>[2] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=34716545" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=34716545</a>

OpenAI may not have a moat, but Microsoft does with their Enterprise Agreements.

&gt; Larger and more complex systems can be built using better tooling.to what end, make rich people richer?

Kind of like every other improvement in technology? From interactive terminals, to compilers, to graphical debuggers?Nothing new there.What productivity improvements have opened up is more opportunities for developers. Larger and more complex systems can be built using better tooling.

In that situation it would be reasonable to expect to be paid twice as much while also being able to devote half the working day to personal&#x2F;open-source projects.

If they’re able to produce twice the work in half the time, wouldn’t it make sense to pay them less?

The nice thing about the new tools is that you can radicalize them by talking to them.

it&#x27;ll be fun to see the pikachu face when engineers are expected to do more, with the aid of these tools, but are not paid any more money.

No one has created even something closed-source that is equal to GPT4.

Vicuna-13b is based on LLAMA that was millions to train. $300 is just to finetuning.

ELI5 How is it too expensive? I know ChatGPT was expensive to train but Vicuna-13b is said to have cost $300 to train [<a href="https:&#x2F;&#x2F;lmsys.org&#x2F;blog&#x2F;2023-03-30-vicuna&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lmsys.org&#x2F;blog&#x2F;2023-03-30-vicuna&#x2F;</a>]

I disagree with this. It&#x27;s too expensive to train high quality models. For example I don&#x27;t see how anyone would make an open-source GPT4 unless OpenAI leaks their model to the public.

&gt; big models by OpenAI and Google that are much better (raw) than anything that was open source recently,When you say &quot;models&quot; do you mean TRAINED models?Wouldn&#x27;t the best training and supervised&#x2F;feedback learning still be in the hands of the big players?An open source &quot;model&quot; of all content in the open internet is great, but it has the garbage-in&#x2F;garbage-out problem.

Basically, if I understand correctly, the &quot;status quo&quot; was that the big models by OpenAI and Google that are much better (raw) than anything that was open source recently, would remain the greatest, and the moat would be the technical complexity of training and running those big models.However, the open sourcing led to tons of people exploring tons of avenues in an extremely quick fashion, leading to the development of models that are able to close in on that performance in a much smaller envelope, destroying the only moat and making it possible for people with limited resources to experiment and innovate.

Can someone dumb this down for me because I don’t understand why this is a surprise… people are getting excited and collaborating to improve and innovate the same models that these larger companies are milking to death

From one researcher, not a VP, director, etc.

Note that this is a personal manifesto, which doesn&#x27;t really represent Google&#x27;s official stance. Which is unfortunate because I&#x27;m largely aligned with this position.

I think OpenAI has a defensible moat by having the first movers&#x27; advantage. As the ease of producing written content declines, we can expect the amount of written content to increase in a consummate fashion. Due to OpenAI&#x27;s position, a vast majority of the newly generated data will come from OpenAI&#x27;s models. When it comes time to train new models with superior network structures or with new data, no one else is going to be able to differentiate human-generated text from LLM generated text. OpenAI&#x27;s training data should become far superior to others.

The non-public moat is big multiyear government contracts with dark money. And there is room there for both players :)

There is a huge (non-public) moat. It&#x27;s big multiyear government contracts with dark money. And there is room there for both players :)

There&#x27;s &quot;immediately profitable&quot; and &quot;eventually profitable&quot;. Vast compute scale allows collection of customer generated data so the latter is possible, AI as of yet is not the former.So GP point still stands. FAAMG can run much larger immediate deficits in order to corner the market on the eventual profitability of AI.

It seems the plan is to be a loss leader until scale is sufficient to reach near AGI levels of capability.

In other words, engage in anti-competitive behavior.

This is the timeline that&#x27;s scaring the shit out of them:Feb 24, 2023: Meta launches LLaMA, a relatively small, open-source AI model.March 3, 2023: LLaMA is leaked to the public, spurring rapid innovation.March 12, 2023: Artem Andreenko runs LLaMA on a Raspberry Pi, inspiring minification efforts.March 13, 2023: Stanford&#x27;s Alpaca adds instruction tuning to LLaMA, enabling low-budget fine-tuning.March 18, 2023: Georgi Gerganov&#x27;s 4-bit quantization enables LLaMA to run on a MacBook CPU.March 19, 2023: Vicuna, a 13B model, achieves &quot;parity&quot; with Bard at a $300 training cost.March 25, 2023: Nomic introduces GPT4All, an ecosystem gathering models like Vicuna at a $100 training cost.March 28, 2023: Cerebras trains an open-source GPT-3 architecture, making the community independent of LLaMA.March 28, 2023: LLaMA-Adapter achieves SOTA multimodal ScienceQA with 1.2M learnable parameters.April 3, 2023: Berkeley&#x27;s Koala dialogue model rivals ChatGPT in user preference at a $100 training cost.April 15, 2023: Open Assistant releases an open-source RLHF model and dataset, making alignment more accessible.

Charity is only a moat if it’s not profitable.

&gt; Having enough scale to perpetually offer free&#x2F;low-cost compute is a moat.Its a moat for services, not models, and its only a moat for AI services as long as that compute isn’t hobbled by being used for models which are so inefficient compared to SOTA as to waste the advantage, which underlines why leaning into open source the way this piece urges is in Google’s interests, the same way open source has worked to Google and Amazon’s benefits as service providers in other domains.(Not so much “the ability to offer free&#x2F;low-cost compute” but “the advantages of scale and existing need for widely geographically dispersed compute on the cost of both marginal compute and having marginal compute close to the customer where that is relevant”, but those are pretty close to differenly-focussed rephrasings of the same underlying reality.)

Yes there will, that&#x27;s the problem HN User Minimaxir is talking about.It will only get less and less expensive for Microsoft in terms of cost. And more and more effective for Microsoft in terms of results delivered.How do you compete with free? That&#x27;s the question. The previous internet experience has already shown us that &quot;also be free&quot; is not really a sustainable or even effective answer. You have to be better in some fundamental dimension.

What makes you think open ai won&#x27;t look at the FOSS improvements, include them in their tech, and make their GPU farm way cheaper, rendering their service even more competitive?Not to mention it&#x27;s easy to run stable diffusion, but midjourney is still a good business. I can run sd on my laptop, I still pay for midjourney because it&#x27;s convenient, the out of the box experience is better than any competition, and it keeps improving.

That&#x27;s what a lot of people think until they run Vicuna 13B or equivalent. We&#x27;re just 5 months in this, there will be many leaps.

And ChatGPT has a super low barrier to entry while open source alternatives have a high one.Creating a service that can compete with it on that regard implies you can scale GPU farms in a cost effective way.It&#x27;s not as easy as it sounds.Meanwhile, openai still improves their product very fast, and unlike google, it&#x27;s their only one. It&#x27;s their baby. It has their entire focus.Since for most consumers, AI == ChatGPT, they have the best market share right now, which mean the most user feedback to improve their product. Which they do at a fast pace.They also understand that to get mass adoption, they need to censor the AI, like MacDonald and Disney craft their family friendly image. Which irritate every geeks, including me, but make commercially sense.Plus, despite the fact you can torrent music and watch it with VLC, and that Amazon+Disney are competitors, netflix exists. Having a quality service has value in itself.I would not count open ai as dead as a lot of people seem to desperately want it to be. Just because Google missed the AI train doesn&#x27;t mean wishful thinking the market to be killed by FOSS is going to make it so.As usual with those things it&#x27;s impossible to know in advance what&#x27;s going to happen, but odds are not disfavoring chatgpt as much as this article says.

&gt; AI text generation competitors like Cohere and Anthropic will never be able to compete with Microsoft&#x2F;Google&#x2F;Amazon on marginal cost.Anthropic already does, with its models. They are same price or cheaper than OpenAI, with comparable quality.&gt; Having enough scale to perpetually offer free&#x2F;low-cost compute is a moat.Rather than a moat it is a growth strategy. At one point in time you need to start to monetize and this is the moment when rubber hits the road. If you can survive monetization and continue to grow, now you have a moat.

Having enough scale to perpetually offer free&#x2F;low-cost compute is a moat.
The primary reason ChatGPT went viral in the first place was because it was free, with no restrictions. Back in 2019, GPT-2 1.5B was made freely accessible by a single developer via the TalkToTransformers website, which was the very first time many people talking about AI text generation...then the owner got hit with sticker shock from the GPU compute needed to scale.AI text generation competitors like Cohere and Anthropic will never be able to compete with Microsoft&#x2F;Google&#x2F;Amazon on marginal cost.

&gt;Research institutions all over the world are building on each other’s work, exploring the solution space in a breadth-first way that far outstrips our own capacity.BroadMind beats DeepMind!

what this proves to me without a doubt is that silo&#x27;d and proprietary iteration i still very clearly also a massive disadvantage. i really hope companies internalize that. if they just keep scooping up and hiding open-source improvements they very well may still be left behind.the final quote from the doc:&gt; And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.

I agree with this over the next 10 years but disagree over the next 30.When&#x2F;If the innovation slows down, the open source stuff will be able to out compete commercial options. Something like this timeline played out for databases and operating systems.

&quot;People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. . .&quot;I&#x27;ll take the opposite side of that bet - MSFT &#x2F; Goog &#x2F; etc in the providers side will drive record revenues on the back of closed &#x2F; restricted models:1 - Table stakes for buying software at enterprise level is permissions based management &amp; standardized security &#x2F; hardening.2 - The corporate world is also the highest value spender of software3 - Corp world will find the &quot;proprietary trained models&quot; on top of vanilla MSFT OpenAI or Goog Bard pitch absolutely irresistible - creates a great story about moats &#x2F; compounding advantages etc. And the outcome is going to most likely be higher switching costs to leave MSFT for a new upstart etc

This looks very fake to me. I might be wrong. Yet, there is no &quot;document&quot; that was leaked, the original source is some blog post. If there is a document, share the document. Shared by &quot;anonymous individual on discord who granted permission for republication&quot;... I don&#x27;t know. If it was shared by anonymous, why ask for permission? Which discord server?

&gt; It&#x27;s a genuinely insightful summary of the last few months of activity in open source modelsYes this is an amazing summary! Just for its summary alone, it is probably one of the top five writings I saw on LLMs and I read every one!&gt; because I have trouble imagining anyone writing something this good and deciding NOT to take credit for the analysis themselves.Not everyone has a substack that they spam onto hacker news every time a thought enter their head. Or imagine that INTP exist. In my opinion the best take ever on LLMs is the simulators essay <a href="https:&#x2F;&#x2F;generative.ink&#x2F;posts&#x2F;simulators&#x2F;" rel="nofollow">https:&#x2F;&#x2F;generative.ink&#x2F;posts&#x2F;simulators&#x2F;</a> and the author is so shy they went pseudonymous and put their twitter as private and don&#x27;t even want their name in conferences.

Did you read it?I honestly don&#x27;t care if it&#x27;s really a leak from inside Google or not: I think the analysis stands on its own. It&#x27;s a genuinely insightful summary of the last few months of activity in open source models, and makes a very compelling argument as to the strategic impact those will have on the incumbent LLM providers.I don&#x27;t think it&#x27;s a leak though, purely because I have trouble imagining anyone writing something this good and deciding NOT to take credit for the analysis themselves.

Coder Radio podcast uploaded the document to the show notes for their latest episode[1]. The first link[2] in the PDF does link to some internal Google resource that requires an @google.com email address.1: <a href="https:&#x2F;&#x2F;jblive.wufoo.com&#x2F;cabinet&#x2F;af096271-d358-4a25-aedf-e56dab169109" rel="nofollow">https:&#x2F;&#x2F;jblive.wufoo.com&#x2F;cabinet&#x2F;af096271-d358-4a25-aedf-e56...</a>2: <a href="http:&#x2F;&#x2F;goto.google.com&#x2F;we-have-no-moat" rel="nofollow">http:&#x2F;&#x2F;goto.google.com&#x2F;we-have-no-moat</a>

Presumably they weren&#x27;t getting permission in the sense of &quot;this publication is authorized by the original author, or by Google&quot; but in the sense of &quot;thanks for leaking the document; can we publish it more widely, or will you get into trouble?&quot;

I tend to agree. For now the OpenAI APIs are so very easy to use and effective. I do try to occasionally use HF models, mostly running locally in order to keep my options open.My bet is that almost everyone wants to keep their options open.I am very much into auxiliary tools like LangChain and LlamaIndex, the topic of my last book, but I also like building up my own tools from scratch (mostly in Common Lisp and Swift for now), and I bet most devs and companies are doing the same.

Google&#x27;s moat is its data set. Imagine training an generative AI LLM on the entire set of YouTube training videos. No one else has this.

I wrote a whole gist about this exact thing!!!!<a href="https:&#x2F;&#x2F;gist.github.com&#x2F;Hellisotherpeople&#x2F;45c619ee22aac6865ca4bb328eb58faf" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;Hellisotherpeople&#x2F;45c619ee22aac6865c...</a>

Local LLMs already have a UI intentionally similar to AUTOMATIC1111, including LoRAs, training with checkpoints, various extensions including multimodal and experimental long-term memory etc.<a href="https:&#x2F;&#x2F;github.com&#x2F;oobabooga&#x2F;text-generation-webui">https:&#x2F;&#x2F;github.com&#x2F;oobabooga&#x2F;text-generation-webui</a>

I have been toying around with Stable Diffusion for a while now and becoming comfortable with the enormous community filled with textual inversions, LoRAs, hyper networks and checkpoints. You can get things with names like “chill blend”, a fine-tuned model on top of the SD with the author’s personal style.There is something called automatic1111 which is a pretty comprehensive web UI for managing all these moving parts. Filled to the brim with extensions to handle AI upscaling, inpainting, outpainting, etc.One of these is ControlNet where you can generate new images based on pose info extracted from an existing image or edited by yourself in the web based 3d editor (integrated, of course). Not just pose but depth maps, etc. All with a few clicks.The level of detail and sheer amount of stuff is ridiculous and it all has meaning and substantial impact on the end result. I have not even talked about the prompting. You can do stuff like [cow:dog:.25] where the generator will start with a cow and then switch over at 25% of the process to a dog. You can use parens like ((sunglasses)) to focus extra hard on that concept.There are so called LoRAs trained on specific styles and&#x2F;or characters. These are usually like 5-100MB and work unreasonably well.You can switch over to the base model easily and the original SD results are 80s arcade game vs GTA5. This stuff has been around for like a year. This is ridiculous.LLMs are enormously “undertooled”. Give it a year or so.My point by the way is that any quality issues in the open source models will be fixed and then some.

&quot;I Have No Moat, and I Must Scream&quot;<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;I_Have_No_Mouth,_and_I_Must_Scream" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;I_Have_No_Mouth,_and_I_Must_Sc...</a>

I get the feeling that at this point, the best thing Google could do is to go all in an open source their models and weights. They&#x27;d canibalize their own business, but they&#x27;d easily wipe out most of the competition.

I&#x27;m convinced that anyone sounding the alarm bells about AI has no idea whatsoever how these models are built.

Copied from other thread: I tend to agree. For now the OpenAI APIs are so very easy to use and effective. I do try to occasionally use HF models, mostly running locally in order to keep my options open.
My bet is that almost everyone wants to keep their options open.
I am very much into auxiliary tools like LangChain and LlamaIndex, the topic of my last book, but I also like building up my own tools from scratch (mostly in Common Lisp and Swift for now), and I bet most devs and companies are doing the same.

I was quite unimpressed when I interviewed with them recently. It’s no surprise their lunch is getting eaten.

I don’t think trying to be the hall monitor of humanity has been good for google. The more paternalistic, the less innovative.

&quot;Many of the new ideas are from ordinary people.&quot;Yeah. Google can fuck right off. Maybe this attitude is what got them in the weeds in the first place.

The entire set of YouTube training videos needs to be re-transcribed before they are useful for training LLMs.

This is the glaring omission in this piece.Googles know _so much_ about me. Is it not reasonable to assume powerful llm + personal data = personal tuned LLM?

If OpenAI has no moat, how come nobody has built a better alternative to GPT-4 yet?

Yes. Same here. I had flashes of Battle of Wesnoth.
Then I realized...

I&#x27;m feeling strangely comforted to have pictured the mythological creature before the meme.

dissaproving_drake.jpg: Giving evidence you can match the capabilities of OpenAIapproving_drake.jpg: Saying everything OpenAI does is easy

One way that I&#x27;ve been framing this in my head (and in an application I&#x27;m building) is that gpt-3 will be useful for analytic tasks but gpt-4 will be required for synthetic tasks. I&#x27;m using &quot;analytic&quot; and &quot;synthetic&quot; in the same way as in this writeup <a href="https:&#x2F;&#x2F;github.com&#x2F;williamcotton&#x2F;empirical-philosophy&#x2F;blob&#x2F;main&#x2F;articles&#x2F;from-prompt-alchemy-to-prompt-engineering-an-introduction-to-analytic-agumentation.md">https:&#x2F;&#x2F;github.com&#x2F;williamcotton&#x2F;empirical-philosophy&#x2F;blob&#x2F;m...</a>

Would you please be more explicit? I&#x27;m curious about the relative strength&#x27;s and weaknesses other&#x27;s see

This is my experience too. While I&#x27;d really love the Open Source models to catch up, currently they struggle even with dead-simple summarization tasks: they hallucinate too much, or omit essential points. ChatGPT don&#x27;t often hallucinate when summarizing, only when answering questions.

Personally, the difference between GPT4 and 3.5 is, pretty immense for what I am using it for. I can use GPT 3.5 for things like summarization tasks (as long as the text isn&#x27;t too complex), reformatting, and other transformation type tasks alright. I don&#x27;t even bother with using it for logical or programming tasks though.

I think this person is referring to the 80&#x2F;20 rule. Here are a few examples:20% of a plant contains 80% of the fruit80% of a company’s profits come from 20% of customers20% of players result in 80% of points scoredI&#x27;ve heard this stated as you can complete 80% of a project with 20% of the effort, and the last 20% of completeness will require 80% of the effort.

GPT feels like an upgrade from MapQuest to Garmin.Garmin was absolutely a better user experience. Less mental load, dynamically updating next steps, etc, etc.However, both MapQuest and Garmin still got things wrong. Interestingly, with Garmin, the lack of mental load meant people blindly followed directions. When it come something wrong, people would do really stupid stuff.

&gt; So I use ChatGPT every day. I like it a lot and it is useful but it is overhyped.It is incorrectly hyped. The vision most pundits have is horribly wrong. It is like people who thought librarians would be out of work because of ebooks, barking up the wrong tree.ChatGPT does amazing things, but it is also prone to errors, but so are people! So what, people still get things done.Imaging feeding ChatGPT an API for smart lights, a description of your house, and then asking it to turn on the lights in your living room. You wouldn&#x27;t have to name the lights &quot;living room&quot;, because Chat GPT knows what the hell a living room is.Meanwhile, if I&#x27;m in my car, and I ask my phone to open Spotify, it will occasionally open Spotify on my TV back home. Admittedly it hasn&#x27;t done for quite some time, I presume it may have been a bug Google fixed, but that bug only exists because Google Assistant is, well, not smart.Here is an app you could build right now with ChatGPT:1. Animatronics with voice boxes, expose an API with a large library of pre-canned movements and feed the API docs to ChatGPT2. Ask ChatGPT to write a story, complete with animations and poses for each character.3. Have ChatGPT emit code with API calls and timing for each character4. Feed each character&#x27;s lines through one of the new generation of TTS services, and once generation is done, have the play performed.Nothing else exists that can automate things to that extent. A specialized model could do some of it, but not all of it. Maybe in the near future you can chain models together, but right now ChatGPT does it all, and it does it really well.And ChatGPT does all sorts of cool things like that, mixing together natural language with machine parsable output (JSON, XML, or create your own format as needed!)

I also felt this way initially, like &quot;that&#x27;s it?&quot;. But overall the massive reduction in hallucinations and increase in general accuracy makes it almost reliable. Math is correct, it follows all commands far more closely, can continue when it&#x27;s cut off by the reply limit, etc.Then I tried it for writing code. Let&#x27;s just say I no longer write code, I just fine tune what it writes for me.

So I use ChatGPT every day. I like it a lot and it is useful but it is overhyped. Also from 3.5 to 4 the jump was nice but seemed relatively marginal to me.I think the head start OpenAi has will vanish. Iteration will be slow and painful giving google or whoever more than enough time to catch up.ChatGPT was a fantastic leap getting us say 80% to Agi but as we have seen time and time again the last 20% are excruciatingly slow and painful (see Self driving cars).

A lot of good points made here, thank you for sharing!

Posted a few notes on this here: <a href="https:&#x2F;&#x2F;simonwillison.net&#x2F;2023&#x2F;May&#x2F;4&#x2F;no-moat&#x2F;" rel="nofollow">https:&#x2F;&#x2F;simonwillison.net&#x2F;2023&#x2F;May&#x2F;4&#x2F;no-moat&#x2F;</a>

NVIDIA already has a big moat in this area. It might not last forever, but at least for a good while they have a big one.

The real moats in this field will come from the hardware industry. It&#x27;s way too expensive to train these models on general purpose compute. Vertically designed silicon that brings down the unit economics of training and inference workloads are already being designed, in industry and in academia.

A) When IMHO the underlying argument is not quite honest.B) When it comes from an interested party.C) When there&#x27;s enough of A and B that I feel it&#x27;s intentional.The underlying argument here would apply to some extremely profitable existing closed source software, so it&#x27;s obviously not complete. Even closed source software which is strictly inferior manages to find some very profitable moats.As for the source, it comes from Google Research, which has done a very poor job of using their knowledge to benefit Google. The article downplays the failures (we couldn&#x27;t have done anything differently, but it doesn&#x27;t matter anyway since Open Source will consume all!), and doesn&#x27;t even think about productization.The latter does give me a little bit of doubt: the article could also be emblematic of Google Research&#x27;s failures and not 100% spin...

How do you distinguish between an opinion you disagree with and &#x27;spin&#x27;?

The memo sounds like spin because it is. The surface argument is equivalent to arguing that no one could sell closed source software because open source exists, and that open source must also be commodotized (oddly, Apple and Microsoft are doing just fine). The implied argument is that Google Research was doing fine giving away their trade secrets and giving negative value to Google because it was going to happen anyway and the secrets are financially worthless anyhow.Nonsense. There are moats if one is willing to look for them. After all, productizing is a very different thing from an academic comparison. ChatGPT is way out there _as a product_, while open efforts are at 0% on this. You can&#x27;t lock down a technology*, but you can lock down an ecosystem, a product or hardware. OpenAI can create an API ecosystem which will be difficult to take down. They can try to make custom hardware to make their models really cheap to run. Monopoly? Nah. This won&#x27;t happen. But they could make some money - and reduce the value of Google&#x27;s search monopoly.* Barring software patents which fortunately aren&#x27;t yet at play.EDIT: I&#x27;ll give the memo a virtual point for identifying Meta (Facebook) as a competitor who could profit by using current OSS efforts. But otherwise it&#x27;s just spin.

Do you have a link to a decent tutorial for someone to do what you&#x27;re describing in the last paragraph?

&gt; Midjourney is easier, its not betterDoes being easier not influence whether it&#x27;s better? I mean this in that for many of the ways AI art would be used, MJ already seems to be &quot;good enough&quot; at a lot of it.Secondarily: doesn&#x27;t Midjourney&#x27;s increased user base and increased ratings they get from users help it refine its model, thus meaning that &quot;ease of use&quot; creates a feedback loop with &quot;quality of output&quot; because more users are engaged?I&#x27;m asking real questions, not making a statement I believe in and just adding a question mark.

&gt; SD is the holy grail of AI art, if you can afford a computer or server to run SD + have the ability to figure out how to install python, clone Automatic1111 from git, and run the installer, its the best.If you can afford Colab (which is free if you don’t want to use it too much), you can just click one of the existing A1111 colabs and run that, you don’t need to figure out python, git, or A1111 installs.

Are you sure about this? For the couple things I tried, a colleague with Midjourney managed to outperform my attempts with SD by leaps and bounds.

Midjourney is higher quality by a fair bit, from my personal experience and from being near a few of the top early AI artists for a good little while.

&gt; Midjourney is easier, its not better.By what measure? Midjourney v5 is massively better with every prompt topic I&#x27;ve thrown at it than SD. It&#x27;s not even close. SD, however, is much better if you want an actual customizable toolchain or to do things like train it on your own face&#x2F;character.

I have SD up on a machine with a 3090 and it can&#x27;t produce output half as good as MJ without a ton of work.I use SD to augment MJ, like fixing hands with specific LORAs for example, so I definitely appreciate that it exists. But for actually creating a full image in one shot, they&#x27;re not even comparable.

That seems to depend on your use case. Frankly, I don&#x27;t have much use for either of them but Midjourney was much closer.I&#x27;ve twice spent a couple of hours unsuccessfully trying to generate a simple background image that would be blurred out when rendering 3D models. SD out-of-the-box was far worse, but Midjourney still was not up to the task. It&#x27;s incredible how well they can generate images of nearly any subject&#x2F;object and make some changes to the style and placement, but trying to precisely achieve critical broad-stroke things like like perspective, sizing, lighting direction&#x2F;amount&#x2F;temperature, etc. was far too cumbersome. Prompt refining is just like having a program with a bunch of nebulous undocumented menu entries that you just have to click on to see what they do rather than just giving you the tools to make what you need to make. Was that the right entry or the wrong entry? Who knows! Maybe just try it again to see if it works better!There&#x27;s a fundamental disconnect between professional-level and consumer-level tools. Consumer tools must be approachable, easy to use, quickly yield appealing results, affordable, and require little maintenance. Professional tools need to be precise, reliable, capable of repeated results with the most demanding tasks, and easily serviceable into perfect working order.These are consumer-level tools. If you merely need a cool picture of a space ship done in such and such style with such and such guns blah blah blah (that for some reason always looks 10%-50% Thomas Kinkaid,) these tools are great, but they abstract away the controls that really matter in professional work. Novices who get overwhelmed by all of those factors love it because they don&#x27;t understand, and probably don&#x27;t care about what they&#x27;re giving up. For serious work, aside from getting inspo images or maybe generating deliberately weird bits of whatever, they&#x27;re hit-or-miss at best. Without exception, doing a rough mock-up in a modelling program took FAR less time than trying to wrangle exactly what I needed from one of those generators.I&#x27;m sure they&#x27;ll get there someday but right now they&#x27;re miles away from being professional-quality image generation tools.

I just want to add my $0.02 currently working at a games studio that is integrating AI generated art into our art pipelines.Midjourney definitely generates really high quality art based on simple prompts, but the inability to really customize the output basically kills its utility.We heavily use Stable Diffusion with specific models and ControlNet to get customizable and consistent results. Our artists also need to extensively tweak and post-process the output, and re-run it again in Stable Diffusion.This entire workflow is definitely beyond a Discord-based interface to say the least.

Midjourney retrains itself, I have one click installer apps for SD, Midjourney and the live prompt community is very goodNone of this stuff is copyrightable so I dont care that its not private

Midjourney is more niche. It’s great at photographs, digital art, concept art, game art and everything in that sphere. Because that’s what it was trained on. So it has a specific style. Dall-E in comparison produces kind of garbage looking pictures of many more styles

&gt;Midjourney is more popular (from what I&#x27;m seeing) than Stable Diffusion at the moment because it&#x27;s better at the moment. Midjourney is closed-source.Midjourney is easier, its not better. The low barrier to entry has it popular, but it isnt as realistic, doesnt follow the prompt as well, and has almost no customization.SD is the holy grail of AI art, if you can afford a computer or server to run SD + have the ability to figure out how to install python, clone Automatic1111 from git, and run the installer, its the best. Those 3 steps are too much for most people, so they default to something more like an app. Maybe it is too soon, but it seems SD has already won. MJ is like using MS paint, where SD is like photoshop.

The kernel isn&#x27;t the OS&#x2F;environment.  Distiling iOS to BSD is just not useful in the context of this discussion.

&gt; Linux won in servers and supercomputing, but not in end user computing&quot;End user computing&quot; these days means mobile, and mobile is dominated by Linux (in Apple&#x27;s case BSD, but we&#x27;re splitting hair) and Chrome&#x2F;WebKit - which began as KHTML.The only area where opensource failed is the desktop, and that&#x27;s also because of Microsoft&#x27;s skill in defending their moats.

It&#x27;s about using the right tool for the right job. GPT-4 is an incredibly versatile generalist tool and a fantastic jack of all trades. However, this comes with some drawbacks. While saying it &#x27;sucks&#x27; might be an exaggeration, I generally concur with the point you&#x27;re making.

And far more expensive than ChatGPT via API, so it makes sense to use ChatGPT3.5, or the locally run equivalents once they get as good, as much as possible.

It&#x27;s quite fast if you use it at ~4 AM in the US.  There&#x27;s definitely a cycle in time.  Putting things in a queue to run while you sleep is a good work around.

Yea 3.5 is more than good enough for a whole slew of tasks (especially code), and it&#x27;s ridiculously fast. I rarely find the need to use 4 but certainly if there was a usecase it was significantly better at that mattered to me, I would.

GPT4 sucks for many use cases because it&#x27;s SLOW. It will co-exist with ChatGPT variants.

&gt; users will go to whoever has the best modelDepends. You might want privacy, need low price in order to process big volumes, need no commercial restrictions, need a different tuning, or the task is easy enough and can be done by the smaller free model - why not? Why pay money, leak information, and get subjected to their rules?You will only use GPT-4 or 5 for that 10% of tasks that really require it. The future spells bad for OpenAI, there is less profit in the large and seldom used big models. For 90% of the tasks there is a &quot;good enough&quot; level, and we&#x27;re approaching it, we don&#x27;t need smarter models except rarely.Another concern for big model developers is data leaks - you can exfiltrate the skills of a large model by batch solving tasks. This works pretty well, you can make smaller models that are just as good as GPT-4 but on a single task. So you can do that if you need to call the API too many times - make your own free and libre model.I think the logical response in this situation would be to start working on AI-anti-malware, like filters for fake news and deceptive sites. It&#x27;s gonna be a cat and mouse game from now on. Better to accept this situation and move on, we can&#x27;t stop AI misuse completely, we&#x27;ll have to manage it, and learn quickly.

Pretty easy to get API access, I got it within a few days. Aware this is a sample of one, but also can’t believe they fast tracked me.

That&#x27;s also why I think OpenAI is in a tough spot in the long run. They just threw as much expensive hardware as they could to build this moat. There&#x27;s basically two things which can happen from now on:- Some scalability breakthrough will appear, if that&#x27;s the case their moat disappears pretty much instantly and the cost of LLMs will plunge close to zero being a commodity. That&#x27;s the future I&#x27;m betting on from what&#x27;s happening now.- No scalability breakthrough will appear and then it means that they will have a hard time to expand further as seen as the gpt4 limited access.Either way, they are in a tough spot.

Big, known companies are already getting their GPT-4 fix via Azure OpenAI Service, where they can get meaningful guarantees for their data, and even on-prem if they really want it.

We have gpt-4 deployed to production being used by fortune 100 labels.

If you think you can use GPT-4 then you don&#x27;t know what you&#x27;re talking about.API access is on waitlist.UI has limit of 25 messages in 3 hours.If you think big, known companies can get ahead of the waitlist and use it - short answer is no, they can&#x27;t because of their IP. Nobody is going to sign off leaking out all internal knowledge to play with something.ClosedAI seems to have big problem with capacity.Those poems about your colleague&#x27;s upcoming birthday do burn a lot of GPU cycles.

I think the best situation is when a company will perform an expensive but high value task that the open source community can&#x27;t and then give it back to them for further iterations and development. If the community isn&#x27;t able to perform a high value task again, a company steps in, does it, and gives it back to the community to restart the process.In this way, everyone&#x27;s skills are being leveraged to innovate at a rapid pace.

It&#x27;s likely they will all be free in time. That&#x27;s kind of the problem underlying the consternation here.It&#x27;s the internet all over again. How do you win the race to the bottom?Once thereHow do you compete effectively with free? Microsoft and Amazon will have billions on billions coming in to float their free offerings for what is effectively eternity in business terms. Probably Google and Meta will as well. What happens to everyone else?I think you have to be in some niche market where you can charge. Because for everyone else, free is unsustainable.Porn maybe? But there will be way too many competitors there. So something more like medical. Or semiconductors. Or construction or something.

&gt; The point I&#x27;m wanting to make is that users will go to whoever has the best model. So, the winning strategy is whatever strategy allows your model to compound in quality faster and to continue to compound that growth in quality for longer.Best only works till second best is &quot;close enough&quot; and cheaper&#x2F;free

&gt; Midjourney is more popular because it takes zero technical know-how compared to SDBoth take zero technical knowledge to use the base models via first-party online hosts, but Midjourney is superior there.SD offers a lot more capacity beyond what is available from the first-party online host, though, while with Midjourney, that’s where it begins and ends, there is nothing more.&gt; and doesn’t require a high-end gaming PC to run it.Neither does SD (I use a couple-year-old business laptop with a 4GB Nvidia card; no sane person would call it a “highend gaming PC” to run A1111 locally, and there are options besides running it locally.)

&gt; DALL-E lost because they let MJ eat their lunchI wonder why nobody is talking about Bing Image Creator<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create</a>which uses some much more advanced version of Dall-E 2 in the background (so Dall-E 2.5? 3?), while being completely free to use. It can produce some pretty mind blowing results with quite simple prompts, although apparently not as impressive as Midjourney V5. A few examples:hyperrealistic<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;hyperrealistic&#x2F;644fa0c48f8d4f04b33ca7c3552fc386?id=UOySTqHpq1q018d6HF%2byhg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;hyperrealistic&#x2F;644fa0c48f...</a>an allegory for femininity<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;an-allegory-for-femininity&#x2F;64519153f532435e9053d5e4e205af1c?id=VLiGLPbicVhfS8w9Z9mKjg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;an-allegory-for-femininit...</a>portrait of a strange woman, hyperrealistic<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;portrait-of-a-strange-woman2c-hyperrealistic&#x2F;644fe78d039348c695df418488748a87?id=nbnDmSSnJBmT997fzymUQw%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;portrait-of-a-strange-wom...</a>allegory of logic, portrait<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;allegory-of-logic2c-portrait&#x2F;644725e29fc54856a7a6671389994d9c?id=%2b32HaPz%2fbHN0ZnwhHeX5qA%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;allegory-of-logic2c-portr...</a>her strange bedfellow<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;her-strange-bedfellow&#x2F;6447109c23e14ab9a737374e3a8e01bf?id=rwYVs44xqL5jy2nMkXrsXw%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;her-strange-bedfellow&#x2F;644...</a>Mrs fox<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;mrs-fox&#x2F;6446e85a32134e649ab48967867a84b4?id=yKgIp7c6PVT2Hv8icrenRQ%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;mrs-fox&#x2F;6446e85a32134e649...</a>inside view<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;inside-view&#x2F;6446f1dc573f463e9a9fa924a71da6c3?id=xqEfDLAg7Q5lZpDiqxOhpw%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;inside-view&#x2F;6446f1dc573f4...</a>in the midst of it all<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;in-the-midst-of-it-all&#x2F;6446f8499c7b412ca19ae83d4d61ff79?id=zKszK%2fOAA5ACbFQ%2blJ61Rg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;in-the-midst-of-it-all&#x2F;64...</a>strange gal<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;strange-gal&#x2F;6446e2a2ea7a413b9a52e9093e5aae80?id=GsbjXXkwYDkBrL1gA2HT%2bg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;strange-gal&#x2F;6446e2a2ea7a4...</a>sighting of a strange entity in an abandoned library<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;sighting-of-a-strange-entity-in-an-abandoned-libra&#x2F;64451d025579482bbf024ef8724b6d1d?id=baZsjFMi%2bc7TWb4bmyfW4A%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;sighting-of-a-strange-ent...</a>sleeping marble woman next to a wall of strange pictures inside an abandoned museum, close-up<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;sleeping-marble-woman-next-to-a-wall-of-strange-pi&#x2F;6444b8be3bf74947b41032e2d8b49e20?id=yBX3nlC08MJGXtvrw563dA%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;sleeping-marble-woman-nex...</a>sculpture of a woman posing next to a wall of strange pictures, close-up<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;sculpture-of-a-woman-posing-next-to-a-wall-of-stra&#x2F;6444a97707b34371a98a0d18aadce8e8?id=D7Xpp9gK8VnWzlRFiAbSqg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;sculpture-of-a-woman-posi...</a>Easter<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;easter&#x2F;643ae4968aff432684b02d4bbe7c73b6?id=qSfaYs7b91zVdSdCum2ppQ%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;easter&#x2F;643ae4968aff432684...</a>Christmas on board a spaceship, DSLR photograph<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;christmas-on-board-a-spaceship2c-dslr-photograph&#x2F;643b11f856344beda54cf8d145625a36?id=2QI%2f5m5raxPlSmsxYWME%2bQ%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;christmas-on-board-a-spac...</a>an angel, dancing to heavy metal<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;an-angel2c-dancing-to-heavy-metal&#x2F;643b2a5144d9457ba3e63a75192e9815?id=%2b3J%2bFp3UaESTjaqJVRWlkw%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;an-angel2c-dancing-to-hea...</a>Saturday afternoon in the streets of a buzzing cyberpunk city, photo-realistic, DSLR<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;saturday-afternoon-in-the-streets-of-a-buzzing-cyb&#x2F;641cb3be6f604143b7ebe6eeabe5e954?id=JDvm4GA9aLZ%2bT05rMqL4mA%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;saturday-afternoon-in-the...</a>The Dogfather<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;the-dogfather&#x2F;6441d18950ba4e2eab5e7c6773fd94c0?id=QR%2fSef49ci15P32eAW43Hg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;the-dogfather&#x2F;6441d18950b...</a>the unlikely guest<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;the-unlikely-guest&#x2F;644446fcc1ba4478bd226ed1e0128b18?id=UWclQl%2fM2%2bYxjW9f0VW0vg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;the-unlikely-guest&#x2F;644446...</a>Strange pictures in an abandoned museum<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;strange-pictures-in-an-abandoned-museum&#x2F;6444559ec0394d7382b288d0676c39c1?id=i%2bf9ulLO%2b3yMNGR61EV%2fqw%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;strange-pictures-in-an-ab...</a>strange woman in an abandoned museum, close-up<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;strange-woman-in-an-abandoned-museum2c-close-up&#x2F;64447d1bc0ce4ad69beb19d92c42b828?id=1n9AjVx9Ob%2bj6d4mh04KOg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;strange-woman-in-an-aband...</a>strange woman in an abandoned museum, strange pictures in the background<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;strange-woman-in-an-abandoned-museum2c-strange-pict&#x2F;64447eed701f4ee8ac580c0e25c52ce9?id=E%2fJ%2fPsbeGoOWhZrPmnrQKQ%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;strange-woman-in-an-aband...</a>a wall of strange pictures in an abandoned museum in Atlantis, close-up<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;a-wall-of-strange-pictures-in-an-abandoned-museum-&#x2F;6444821e41b446aaac45b3c5e62a9a3e?id=N619lGeImVO%2bAkvss3mxdg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;a-wall-of-strange-picture...</a>female sculpture in an abandoned museum in Atlantis, close-up<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;female-sculpture-in-an-abandoned-museum-in-atlanti&#x2F;644489b3b03b4d5bb42c6f96bd0f6e33?id=WHbqlG%2fXxn1%2bUyMcTmIzmQ%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;female-sculpture-in-an-ab...</a>the unlikely guest<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;the-unlikely-guest&#x2F;6444908ea6264cbd987a75c16cbdbb8d?id=VbWd9SE%2f1agad%2bnjDobHsw%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;the-unlikely-guest&#x2F;644490...</a>an unlikely guest of the secret society in the lost city in a country without name, close-up<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;an-unlikely-guest-of-the-secret-society-in-the-los&#x2F;6444927c654f4a9289fdb744eb67a392?id=3aAG9FoYdJNY5lj4CWlb%2bg%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;an-unlikely-guest-of-the-...</a>I think the quality of most of these pictures is far beyond what is achievable with Dall-E 2. One issue that still exists (though to a lesser extent) is the fact that faces have to cover a fairly large area of the image. Smaller faces look strange, e.g. here:photograph of the unlikely guests<a href="https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;photograph-of-the-unlikely-guests&#x2F;64448f8abc834a28945e32118955d207?id=xSsSPWDVR47REfDaDw4RKw%3d%3d&amp;view=detailv2&amp;idpp=genimg" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;images&#x2F;create&#x2F;photograph-of-the-unlikel...</a>It is as if the model creates a good draft in low resolution, and another model scales it up, but the latter model doesn&#x27;t know what a face is? (I have no idea how diffusion models actually work.)

Midjourney is more popular because it takes zero technical know-how compared to SD (even with A1111 it took me nearly an hour to walk my competent-but-layman brother through installing it) and doesn&#x27;t require a high-end gaming PC to run it. (DALL-E lost because they let MJ eat their lunch)

None of the models will &quot;win&quot; because it is just a foundation. Google won because they leveeraged the linux ecosystem to build a monetizable business with a moat on top of it. The real moat will be some specific application on top of LLMs

This year is the year of the Linux Desktop!I kid. I&#x27;ve been primarily a Linux Desktop user for 20 years.

Linux did kind of win for end user computing. Android is based on a modified linux kernel.

&gt; Linux won in servers and supercomputing, but not in end user computing.Pardon the side discussion, but I think this is because of a few things.1. OS-exclusive &quot;killer apps&quot; (Office, anything that integrates with an iPhone)2. GamesThe killer apps have better alternatives now, and games are starting to work better on Linux. Microsoft&#x27;s business model no longer requires everyone to use Windows. (Mac is another story.) So I think that, at least for non-Macolytes, Linux end user dominance is certainly on the horizon.

I think that pouring a lot of money in open source, by bounties or crowdfunding can accelerate open source alternatives to closed LLMs. Perhaps a middle way in which software will be declared open source six month from now can give enough compensation to those institutions contributing big money for developing LLM technology. That is a crowdfunding in which the great contributors have a limited time to be compensated, but capping the total prize just like that of chatgpt 3.5 or 4 depending of the model.

GPT-4 is a must if tool using is your goal.GPT-3.5, I think it is mostly suitable for:1. Quick documentation lookup for non-essential facts2. Lightweight documents writing and rewriting3. TranslationOther use cases should go straightly to GPT-4

I don&#x27;t feel sorry for Google, nor the big amounts of PR nonsense they&#x27;re putting out there in order to try to spin their being too slow to move LLM tech to the side of the consumer. Get better or get out.

Yes, I&#x27;d readily pay for GPT-4 access, though not the limited 25 requests per 3 hours version. I ponied up $20 for a month of usage to check it out, and it performs head &amp; shoulders above 3.5 in its ability to comprehensively address more complex prompts and provide output that is more nuanced than ChatGPT.I&#x27;ll also point out that paid api access to 3.5 (davinci-03) is frequently better than ChatGPT&#x27;s use of 3.5. You get many fewer restrictions, and none the &quot;awe shucks, I&#x27;m just a little &#x27;ol LLM and so I couldn&#x27;t possibly answer that&quot;.If you&#x27;re frustrated by having to go to great lengths to prompt engineer and ask ChatGPT to &quot;pretend&quot; then it&#x27;s worth it to pay for API access. I&#x27;m just frustrated that I can&#x27;t use the GPT-4 API the same way yet (waitlist)

The question is how long will it take for open source to become just as good as GPT 4? If it is 3 years, then yes, this is copium. But if it is 1 year or less, then how much is google really missing out on?OpenAI spent 600m to improve GPT and made 200m from it and if costs dramatically fall for model development, it might be OpenAI that is shooting itself in the foot.

&gt; And you can&#x27;t even use Bard as a developer, there&#x27;s no API!There is an API for the underlying model, it&#x27;s just in alpha&#x2F;beta&#x2F;whatever they call limited invite-only release and you have to ask your devrel team to get access. I&#x27;m guessing we&#x27;ll see better models very soon.Google is, as usual, playing catch-up, but I have no doubt once the machine gets cranking they&#x27;ll be fully competitive, at least similar to how GCP is now a totally viable AWS alternative. They never lead the pack because they can&#x27;t (lawyers, regulation, monopoly, etc), but they know how to commit and execute.

SD and its configurability is miles and miles ahead of MJ. Sure if you want a fancy picture now it’s OK. How are you going to generate that same picture in another pose? Inpainting, outpainting.. I don’t even know where to begin. MJ is a toy compared to SD’s ecosystem.

&gt;Similarly, StableDiffusion&#x27;s image quality progress has in my experience stalled out, whereas Midjourney continues to dramatically improve every couple months, and easily beats SD. Open source isn&#x27;t a magic bullet for quality.MJ only does one style, and you can emulate that just fine in SD if thats what you want.

GPT-4 is so much better for complex tasks that I wouldn&#x27;t use anything else. Trying to get 3.5 to do anything complicated is like pulling teeth, and using something worse than 3.5... Oof.TBH this feels like cope from Google; Bard is embarrassingly bad and they expected to be able to compete with OpenAI. In my experience, despite their graph in the article that puts them ahead of Vicuna-13B, they&#x27;re actually behind... And you can&#x27;t even use Bard as a developer, there&#x27;s no API!But GPT-4 is so, so much better. It&#x27;s not clear to me that individual people doing LoRa at home is going to meaningfully close the gap in terms of generalized capability — at least, not faster than OpenAI itself improves its models. Similarly, StableDiffusion&#x27;s image quality progress has in my experience stalled out, whereas Midjourney continues to dramatically improve every couple months, and easily beats SD. Open source isn&#x27;t a magic bullet for quality.Edit: re: the complaints about Midjourney&#x27;s UI being Discord — sure, that definitely constrains what you can do with it, but OpenAI&#x27;s interface isn&#x27;t Discord, it has an API. And you can fine-tune the GPT-3 models programmatically too, and although they haven&#x27;t opened that up to GPT-4 yet, IME you can&#x27;t fine-tune your way to GPT-4 quality anyway with anything.&quot;There&#x27;s no moat&quot; and &quot;OpenAI is irrelevant&quot; feel like the cries of the company that&#x27;s losing to OpenAI and wants to save face on the way out. Getting repeated generational improvements without the dataset size and compute scale of a dedicated, well-capitalized company is going to be very tough. As a somewhat similar data+compute problem, I can&#x27;t think of an open-source project that effectively dethroned Google Search, for example... At least, not by being better at search (you can argue that maybe LLMs are dethroning Google, but on the other hand, it&#x27;s not the open source models that are the best at that, it&#x27;s closed-source GPT-4).

Appliances are not end user computing, but embedded computing - the OS is incidental and under full control of the manufacturer.
Some might argue that even mobile phones are not sufficiently under the control of end users to qualify.

&gt; Linux won in servers and supercomputing, but not in end user computing.It seems just about every computing appliance in my home runs Linux. Then you have Android, ChromeOS, etc. which are also quite popular with end users, the first one especially. It may not have won, but I think it is safe to say that it is dominating.

Worthy to note, it seems like there were some incredibly dedicated hardworking engineers that drove extremely hard for a really long time to make this happen.They did manage to get large buy in from the company after quite a significant journey. But it seems so much like a kind of outside event, something begat &amp; pushed for not because it was a smart top down move, but because a couple super driven engineers made it their cause.

This is what happened with kubernetes no? Open source was about to take over so google release the code not to loose out.

aren&#x27;t androids linux? thats the biggest by far end user platform.of course google doesnt want to acknowledge it too much.<a href="https:&#x2F;&#x2F;source.android.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;source.android.com&#x2F;</a>

&gt; users will go to whoever has the best modelNot me, I refuse to use OpenAI products but I do sometimes use vicuna 13b when I&#x27;m coding C. It&#x27;s pretty good and I&#x27;m happy to see the rapid advancement of open source LLMs. It gives me hope for the future.&gt; Linux won in servers and supercomputing, but not in end user computing.I use linux on all of my computers and I love it, many of us do (obviously). I&#x27;m aware that I&#x27;m a small minority even among other developers but I think looking at just statistics misses the point. Even if the majority will just use the most approachable tool (and there is nothing wrong with that), it&#x27;s important to have an alternative. For me this is the point of open software, not market domination or whatever.

Will the average use know or want to use different models when they can just go to ChatGPT?

How can a company keep up with the speed of what is happening in the open?Open AI had years of advanced that almost vanished in a few months.And we will see the rise of specialized models, smaller but targeted, working in team, delegating (Hugging GPT)I would use a small and fast model that only speak english, is expert at coding an science and not much more. Then you fire up an question to another model if yours is out of it’s area.

Adobe Firefly can&#x27;t be run locally and Adobe knows everything that their users generate. I can&#x27;t train my own LoRAs or checkpoints. Adobe also has proven that they can&#x27;t keep the data of their users secure. Which is why it&#x27;s better to use something else.

&gt;The point I&#x27;m wanting to make is that users will go to whoever has the best model.Best isn&#x27;t defined just by quality though. In some instances for some groups, things like whether the model is trained on licensed content (with permission) and &#x2F; or is safe for commercial use is more important.This is one reason why Adobe&#x27;s Firefly has been received relatively well. (I work for Adobe).

What?No, what the article said was:&gt; At that pace, it doesn’t take long before the cumulative effect of all of these fine-tunings overcomes starting off at a size disadvantage.&gt;Indeed, in terms of engineer-hours, the pace of improvement from these models vastly outstrips what we can do with our largest variants, and&gt; the best are already largely indistinguishable from ChatGPT.^ The author did not note chat gpt is better, the author claims that the 7B koala model is &#x27;largely indistinguishable from ChatGPT&#x27;.and:&gt; While ChatGPT still holds a slight edge, more than 50% of the time users either prefer Koala or have no preference.Which is highly misleading.The koala authors rated their model by passing it to 100 people using the mechanical turk, noting:&gt; To mitigate possible test-set leakage, we filtered out queries that have a BLEU score greater than 20% with any example from our training set. Additionally, we removed non-English and coding-related prompts, since responses to these queries cannot be reliably reviewed by our pool of raters (crowd workers).So.What you have is a model that performs pretty well for some trivial conversational prompting tasks.What you DO NOT have, is something that is: &quot;largely indistinguishable from ChatGPT&quot;.Anyway, regardless of the creative interpretation of the authors writing, the point that I&#x27;m making is that your point:&gt; So, the winning strategy is whatever strategy allows your model to compound in quality faster and to continue to compound that growth in quality for longer.Is founded on the assumption from the post that:&gt; While the individual fine tunings are low rank, their sum need not be, allowing full-rank updates to the model to accumulate over time.ie. If you fine tune it enough, it&#x27;ll get better and better in an unlimited fashion.Which is provably false.If I have a 10-parameter model, there is no possible way that the accumulation of low rank fine tunings will make it the equivalent of a 7B, 13B of 135B model.It is simply not complex enough to do some tasks.Similarly, smaller models like 3B or 7B model, appear to have an upper bound on what is possible to achieve with them regardless of the number of fine tunings applied to them, for the direct and obvious same reason.There is an upper bound on what is possible, based on the model size.The &#x27;best&#x27; size for a model hasn&#x27;t really been figured out, but... I&#x27;m getting pretty sick of people saying these 7B models are as good as &#x27;ChatGPT&#x27;.They. Are. Not.People will go to the best models, with the best licenses, but... those models are, it seems, unlikely to be fine tuned smallish models.

&gt; Linux won in servers and supercomputing, but not in end user computingAndroid is based on Linux.

The part of the post that resonates for me is that working with the open source community may allow a model to improve faster. And, whichever model improves faster, will win - if it can continue that pace of improvement.The author talks about Koala but notes that ChatGPT is better. GPT-4 is then significantly better than GPT-3.5. If you&#x27;ve used all the models and can afford to spend money, you&#x27;d be insane to not use GPT-4 over all the other models.Midjourney is more popular (from what I&#x27;m seeing) than Stable Diffusion at the moment because it&#x27;s better at the moment. Midjourney is closed-source.The point I&#x27;m wanting to make is that users will go to whoever has the best model. So, the winning strategy is whatever strategy allows your model to compound in quality faster and to continue to compound that growth in quality for longer.Open source doesn&#x27;t always win in producing better quality products.Linux won in servers and supercomputing, but not in end user computing.Open-source databases mostly won.Chromium sorta won, but really Chrome.Then in most other areas, closed-source has won.So one takeaway might be that open-source will win in areas where the users are often software developers that can make improvements to the product they&#x27;re using, and closed-source will win in other areas.

Taking the &quot;no moat&quot; argument at face value, I think it&#x27;s important to remember that some of the largest players in AI are lobbying for regulation too.

OpenAI doesn&#x27;t need a moat and it&#x27;s fine that they don&#x27;t have one. From their charter:&gt; We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.This was from 2018 and they&#x27;ve taken large strides away from their originally stated mission. Overall, though, they should be happy to have made progress in what they set out to do.

That is the nature of the singularity. Progress moves faster than any one person or any one company can keep up with.

Sorry for the  dumb question. But in the context of the AI space, what is moat?

I wonder if OpenAI knew they didn&#x27;t have a moat, and that&#x27;s why they&#x27;ve been moving so fast and opening ChatGPT publicly - making the most of their lead in the short time they have left.I find it incredibly cathartic to see these massive tech companies and their gatekeepers get their lunch eaten by OSS.

There definitely is. LLaMA is not licensed for commercial use. It&#x27;s impractical to prosecute 1,000 people tinkering on their laptops, but if Meta discovered that Amazon was using LLaMA for commercial purposes, it would be nuclear war.

&gt; Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.There’s also nothing stopping anybody else from incorporating it into their products.

Most of the code isn&#x27;t specific to a model. It happens that LLaMA is approximately the best LLM currently available to the public to run on their own hardware, so that&#x27;s what people are doing. But as soon as anyone publishes a better one, people will use that, using largely the same code, and there is no reason it couldn&#x27;t be open source.I&#x27;m also curious what the copyright status of these models even is, given the &quot;algorithmic output isn&#x27;t copyrightable&quot; thing and that the models themselves are essentially the algorithmic output of a machine learning algorithm on third party data. What right does Meta have to impose restrictions on the use of that data against people who downloaded it from The Pirate Bay? Wouldn&#x27;t it be the same model if someone just ran the same algorithm on the same public data?(Not that that isn&#x27;t an impediment to people who don&#x27;t want to risk the legal expenses of setting a precedent, which models explicitly in the public domain would resolve.)

this is a temporary state. Open source alternatives are already available and more are being trained.

Meh, you can experiment on it for personal use as much as you want and that&#x27;s all what&#x27;s needed in this short period of time before powerful, open base models start appearing like mushrooms at which point the whole thing is going to be moot.

&gt; Meta’s leaked model isn’t open-source.Meta’s leaked model has been a factor in spurring open source development, whether or not it is open source; the article also discusses the practical market of effect of leaked-but-not-open things like Meta’s model combined with the impracticality of prosecuting the vast hordes of individuals using it, and particularly notes that the open-except-for-the-base-model work on top of it is a major benefit for Meta (who can use the work directly) that locks out everyone else in the commercial market (who cannot), and leaning into open source base models is a counter to that.

I&#x27;m still not on board with calling it leaked...the weights were open for anyone to get and use as long as they agreed to use them academically.Basically, completely open source with a non-commercial license. I&#x27;m not sure why so many people keep saying it &#x27;leaked&#x27;. It&#x27;s just using open source weights not directly from the provider in a way that violates the software license.

I don&#x27;t think we know where weights stand legally yet. They may end up being like databases, uncopyrightable.

You are making an assumption that seems very strange to me - that the license matters for important use cases. It doesn&#x27;t. Access to the technology is the only important factor, because nothing interesting about AI involves commercialising anything. It&#x27;s a tool and now people have it. Whether they can make a company out of it is so far down the list of how it could make a difference that it doesn&#x27;t even register.

Meta&#x27;s leaked model isn&#x27;t open-source. I can found a business using Linux, that&#x27;s open-source. The LLM piracy community are unpaid FB employees; it is not legal for anyone but Meta to use the results of their labor.I know this might be hard news but it needs to be said... if you want to put your time into working on open source LLMs, you need to get behind something you have a real (and yes, open source) license for.

By that logic Google&#x27;s moat is the integration with Gmail and Google Docs.And frankly it&#x27;s not. People will decide to copy some text from Office or Docs to some other non-integrated tool, get LLMs to work, and then paste back to Office or Docs.

Openai moat is the upcoming first party integration with ms  office.

I think this type of comment is generally frowned upon on HN.Upvote serves the same purpose.

&gt; the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.This

&gt; Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.One interesting related point to this is Zuck&#x27;s comments on Meta&#x27;s AI strategy during their earnings call: <a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;MachineLearning&#x2F;comments&#x2F;1373nhq&#x2F;discussion_mark_zuckerberg_on_metas_strategy_on&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;MachineLearning&#x2F;comments&#x2F;1373nhq&#x2F;di...</a>Summary:&quot;&quot;&quot;
Some noteworthy quotes that signal the thought process at Meta FAIR and more broadly<pre><code> We’re just playing a different game on the infrastructure than companies like Google or Microsoft or Amazon

 We would aspire to and hope to make even more open than that. So, we’ll need to figure out a way to do that.

 ...lead us to do more work in terms of open sourcing, some of the lower level models and tools

 Open sourcing low level tools make the way we run all this infrastructure more efficient over time.

 On PyTorch: It’s generally been very valuable for us to provide that because now all of the best developers across the industry are using tools that we’re also using internally.

 I would expect us to be pushing and helping to build out an open ecosystem.</code></pre>
&quot;&quot;&quot;

&gt;&gt;”T ey have effectively garnered an entire planet’s worth of free labor.”-THIS IS WHY WE NEED DATA FUCKING OWNERSHIP.Users should be able to have a recourse to the use of their data in both of terms utility (for the parent company) and in terms of financial value to the parent company to the financial extraction of that value.Let me use cannabis as an example…When multiple cannabis cultivators (growers) combine their product for extraction into a singular product we have to figure out how to divide and pay the taxes..Same thing (I’ll edit this later because I’m at the dentist

Some snippets for folks who come just for the comments:&gt; While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.&gt; A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other.&gt; This recent progress has direct, immediate implications for our business strategy. Who would pay for a Google product with usage restrictions if there is a free, high quality alternative without them?&gt; Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.&gt; And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.

&gt; I&#x27;ve been saying the same things for weeks, right here and in the usual places. Basically - OpenAI will not be able to continue to commercialise chatGPT-3.5, they will have to move to GPT-4 because the open source alternatives will catch up. Their island of exclusivity is shrinking fast. In a few months nobody will want to pay for GPT-4 either when they can have private, cheap equivalents. So GPT-5 it is for OpenAI.It is worth $20 a month to have one UI on one service that does everything.Unless specialized models can far exceed what GPT4 can do, being general purpose is amazing.IMHO the future is APIs written for consumption by LLMs, and then natural language interfaces and just telling an AI literally anything you want done.

And where are these open source models where I can go to a url and do all the things I can do in ChatGPT or through api keys for OpenAI? I googled a couple of weeks ago to find hosted versions of these open source models to try, and every one was either down or woefully poor.OpenAI and MS are going to win because they have a package to go and it’s ready and available and working well - they have set the benchmark. I’m not seeing any evidence of this in the OSS community thus far.Until I can spin up a docker image capable of the same as OpenAI in hetzner for 30 bucks a month - it’s not in the same league.

I also would like to believe that, but there are countless examples which show the difference. Companies have no time to figure out which of the open source offerings is the best. Even worse, they don’t have the time to switch from one project to the other or back to OpenAI if OpenAI releases a new state-of-the-art model.

&gt; I&#x27;m wondering how many of the requests will be so complex as to require GPT-5I am not sure the pessimism is warranted. True that few people have the need to upgrade from GPT-3.5 to GPT-4 now, but if GPT-5 is another serious leap in capabilities, it might have an effect closer to the difference between old chatbots (useless, interesting) and ChatGPT (immediate economic impact, transforming some jobs). Or at any rate, we should expect such a leap to occur soon, even if it&#x27;s not GPT-5.

&gt; I&#x27;ve been saying the same things for weeks, right here and in the usual places. Basically - OpenAI will not be able to continue to commercialise chatGPT-3.5, they will have to move to GPT-4 because the open source alternatives will catch up. Their island of exclusivity is shrinking fast. In a few months nobody will want to pay for GPT-4 either when they can have private, cheap equivalents. So GPT-5 it is for OpenAI.I wonder if this effect will be compounded by regulatory pressure that seems poised to slow down progress at the bleeding edge of LLMs.Open source closing the gap at the bottom, and governments restricting further movement at the top...

I&#x27;ve been saying the same things for weeks, right here and in the usual places. Basically - OpenAI will not be able to continue to commercialise chatGPT-3.5, they will have to move to GPT-4 because the open source alternatives will catch up. Their island of exclusivity is shrinking fast. In a few months nobody will want to pay for GPT-4 either when they can have private, cheap equivalents. So GPT-5 it is for OpenAI.But the bulk of the tasks can probably be solved at 3.5 level, another more difficult chunk with 4, I&#x27;m wondering how many of the requests will be so complex as to require GPT-5. Probably less than 1%.There&#x27;s a significant distinction between web search and generative AI. You can&#x27;t download &quot;a Google&quot; but you can download &quot;a LLaMA&quot;. This marks the end of the centralisation era and increased user freedom. Engaging in chat and image generation without being tracked is now possible while searching, browsing the web or torrenting are still tracked.

This is easily among the rare highest quality articles&#x2F;comments I&#x27;ve read in the past weeks, perhaps months (on LLMs&#x2F;AI since that&#x27;s what I am particularly interested in). And this was for internal consumption before it was made public. Reinforces my recent impression that so much that&#x27;s being made for public consumption now is shallow and it is hard to find the good stuff. And sadly, increasing so even on HN. As I write this, I acknowledge I discovered this on HN :) Wish we had ways to incentivize the public sharing of such high-quality content that don&#x27;t die at the altar of micro rewards.

Google “We have no moat, and neither does OpenAI”