Can&#x27;t wait to play with this in llama.cpp; the 8B results look impressive.

They explain that they will be releasing longer context lengths in the future.It’s better to make your RAG system work well on small context first anyway.

If it&#x27;s limited to 8k context length then it&#x27;s not competing with sonnet at all IMO.  Sonnet has a 200k context length and it&#x27;s decent at pulling stuff from it, with just an 8k context length this model won&#x27;t be great for RAG applications, instead it&#x27;ll be used for chat and transforming data from one type to another.

Looks like 8K context length. Seems to compare well against Gemini Pro 1.5 and Claude 3 Sonnet according to the included benchmarks.

I do see on the bottom left:Log in to save your conversation history, sync with Messenger, generate images and more.

I imagine that is to compete with ChatGPT, which began doing the same.

Doesn&#x27;t work for me, I&#x27;m in EU.

Yeah, but not for image generation unfortunatelyI&#x27;ve never had a FaceBook account, and really don&#x27;t trust them regarding 
privacy

I had the same reaction, but when I saw the thumbs up and down icon, I realized this was a smart way to crowd source validation data.

Which indicates that they get enough value out of logged ~in~ out users. Potentially they can identify you without logging in, no need to. But also ofc they get a lot of value by giving them data via interacting with the model.

But not from Japan, and I assume most other non-English speaking countries.

1. Free rlhf
2. They cookie the hell out of you to breadcrumb your journey around the web.They don&#x27;t need you to login to get what they need, much like Google

And they even allow you to use it without logging in. Didnt expect that from Meta.

Spending millions&#x2F;billions to train these models is for a reason and it&#x27;s not just for funsies.

&gt; And announcing a lot of integration across the Meta product suite, ...That&#x27;s ominous...

ML Twitter was saying that they&#x27;re working on a 400B parameter version?

Llama is open weight, not open source. They don’t release all the things you need to reproduce their weights.

&quot;within stabbing distance&quot;dunno if english is your mother tongue, but this sounds really good (although a tad aggressive :-) )) !

On the topic of LoRAs and finetuning, have a Colab for LoRA finetuning Llama-3 8B :) <a href="https:&#x2F;&#x2F;colab.research.google.com&#x2F;drive&#x2F;135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing" rel="nofollow">https:&#x2F;&#x2F;colab.research.google.com&#x2F;drive&#x2F;135ced7oHytdxu3N2DNe...</a>

They didn&#x27;t compare against the best models because they were trying to do &quot;in class&quot; comparisons, and the 70B model is in the same class as Sonnet (which they do compare against) and GPT3.5 (which is much worse than sonnet). If they&#x27;re beating sonnet that means they&#x27;re going to be within stabbing distance of opus and gpt4 for most tasks, with the only major difference probably arising in extremely difficult reasoning benchmarks.Since llama is open source, we&#x27;re going to see fine tunes and LoRAs though, unlike opus.

(You can&#x27;t compare parameter count with a mixture of experts model, which is what the 1.8T rumor says that GPT-4 is.)

&gt; Neglected to include comparisons against GPT-4-Turbo or Claude Opus, so I guess it&#x27;s far from being a frontier modelYeah, almost like comparing a 70b model with a 1.8 trillion parameter model doesn&#x27;t make any sense when you have a 400b model pending release.

Just use the Replicate demo instead, you can even alter the inference parameters<a href="https:&#x2F;&#x2F;llama3.replicate.dev&#x2F;" rel="nofollow">https:&#x2F;&#x2F;llama3.replicate.dev&#x2F;</a>Or run a jupyter notebook from Unsloth on Colab<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;unsloth&#x2F;llama-3-8b-bnb-4bit" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;unsloth&#x2F;llama-3-8b-bnb-4bit</a>

&gt;We’re rolling out Meta AI in English in more than a dozen countries outside of the US. Now, people will have access to Meta AI in Australia, Canada, Ghana, Jamaica, Malawi, New Zealand, Nigeria, Pakistan, Singapore, South Africa, Uganda, Zambia and Zimbabwe — and we’re just getting started.<a href="https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;04&#x2F;meta-ai-assistant-built-with-llama-3&#x2F;" rel="nofollow">https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;04&#x2F;meta-ai-assistant-built-wi...</a>

This is so frustrating. Why don&#x27;t they just make it available everywhere?

Everyone saying it&#x27;s an EU problem. Same message in Guatemala.

Anakin AI has Llama 3 models available right now: <a href="https:&#x2F;&#x2F;app.anakin.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;app.anakin.ai&#x2F;</a>

&gt; Meta AI isn&#x27;t available yet in your countryWhere is it available? I got this in Norway.

Anyone have any informed guesstimations as to where we might expect a 400b parameter model for llama 3 to land benchmark wise and performance wise, relative to this current llama 3 and relative to GPT-4?I understand that parameters mean different things for different models, and llama two had 70 b parameters, so I&#x27;m wondering if anyone can contribute some guesstimation as to what might be expected with the larger model that they are teasing?

They also stated that they are still training larger variants that will be more competitive:&gt; Our largest models are over 400B parameters and, while these models are still training, our team is excited about how they’re trending. Over the coming months, we’ll release multiple models with new capabilities including multimodality, the ability to converse in multiple languages, a much longer context window, and stronger overall capabilities.

That realtime `&#x2F;imagine` prompt seems pretty great.

Can we stop referring to VS Code as &quot;their IDE&quot;?Do you support any other editors? If the list is small, just name them. Not everyone uses or likes VS Code.

Double seems more like a feature than a product. I feel like Copilot could easily implement those value-adds and obsolete this product.I also don&#x27;t understand why I can&#x27;t bring my own API tokens. I have API keys for OpenAI, Anthropic, and even local LLMs. I guess the &quot;secret&quot; is in the prompting that is being done on the user&#x27;s behalf.I appreciate the work that went into this, I just think it&#x27;s not for me.

That was fast! I&#x27;ve really been enjoying Double, thanks for your work.

Also added Llama 3 70B to our coding copilot <a href="https:&#x2F;&#x2F;www.double.bot">https:&#x2F;&#x2F;www.double.bot</a> if anyone wants to try it for coding within their IDE and not just chat in the console

Are there an stats on if llama 3 beats out chatgpt 3.5 (the free one you can use)?

Inference speed is not a great metric given the horizontal scalability of LLMs.

Tried a few queries and was surprised how fast it responded vs how slow chatgpt can be. Responses seemed just as good too.

Strange. The Llama 3 model card mentions that the knowledge cutoff dates are March 2023 for the 8B version and December 2023 for the 70B version (<a href="https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;MODEL_CARD.md">https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;MODEL_CARD.md</a>)

I&#x27;m based on LLaMA 2, which is a type of transformer language model developed by Meta AI. LLaMA 2 is a more advanced version of the original LLaMA model, with improved performance and capabilities. I&#x27;m a specific instance of LLaMA 2, trained on a massive dataset of text from the internet, books, and other sources, and fine-tuned for conversational AI applications. My knowledge cutoff is December 2022, and I&#x27;m constantly learning and improving with new updates and fine-tuning.

Disagree on Nvidia, most folks fine-tune model. Proof: there are about 20k models in huggingface derived from llama 2, all of them trained on Nvidia GPUs.

I also disagree on Google...Google&#x27;s business is largely not predicated on AI the way everyone else is. Sure they hope it&#x27;s a driver of growth, but if the entire LLM industry disappeared, they&#x27;d be fine. Google doesn&#x27;t need AI &quot;Superiority&quot;, they need &quot;good enough&quot; to prevent the masses from product switching.If the entire world is saturated in AI, then it no longer becomes a differentiator to drive switching. And maybe the arms race will die down, and they can save on costs trying to out-gun everyone else.

&gt;AMD, intel: these companies can focus on Chips for AI InferenceNo real evidence either can pull that off in any meaningful timeline, look how badly they neglected this type of computing the past 15 years.

Pretty sure meta still uses NVIDIA for training.

The memory chip companies were done for, once Bill Gates figured out no one would ever need more than 64K of memory

If anything a capable open source model is good for Nvidia, not commenting on their share price but business of course.Better open models lower the barrier to build products and drive the price down, more options at cheaper prices which means bigger demand for GPUs and Cloud. More of what the end customers pay for goes to inference and not IP&#x2F;training of proprietary models

Losers &amp; Winners from Llama-3-400B Matching &#x27;Claude 3 Opus&#x27; etc..Losers:- Nvidia Stock : lid on GPU growth in the coming year or two as &quot;Nation states&quot; use Llama-3&#x2F;Llama-4 instead spending $$$ on GPU for own models, same goes with big corporations.- OpenAI &amp; Sam: hard to raise speculated $100 Billion, Given GPT-4&#x2F;GPT-5 advances are visible now.- Google : diminished AI superiority postureWinners:- AMD, intel: these companies can focus on Chips for AI Inference instead of falling behind Nvidia Training Superior GPUs- Universities &amp; rest of the world : can work on top of Llama-3

Why does Meta embed a 3.5MB animated GIF (<a href="https:&#x2F;&#x2F;about.fb.com&#x2F;wp-content&#x2F;uploads&#x2F;2024&#x2F;04&#x2F;Meta-AI-Expasion_Header.gif" rel="nofollow">https:&#x2F;&#x2F;about.fb.com&#x2F;wp-content&#x2F;uploads&#x2F;2024&#x2F;04&#x2F;Meta-AI-Expa...</a>) on their announcement post instead of much smaller animated WebP&#x2F;APNG&#x2F;MP4 file? They should care about users with low bandwidth and limited data plan.

They&#x27;ve got a console for it as well,
<a href="https:&#x2F;&#x2F;www.meta.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.meta.ai&#x2F;</a>And announcing a lot of integration across the Meta product suite,
<a href="https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;04&#x2F;meta-ai-assistant-built-with-llama-3&#x2F;" rel="nofollow">https:&#x2F;&#x2F;about.fb.com&#x2F;news&#x2F;2024&#x2F;04&#x2F;meta-ai-assistant-built-wi...</a>Neglected to include comparisons against GPT-4-Turbo or Claude Opus, so I guess it&#x27;s far from being a frontier model. We&#x27;ll see how it fares in the LLM Arena.

To be fair, the Llama 2 instruction tuning was notably bad.

For the instruction tuned models, Llama 3 8B is even significantly better than Llama 2 70B!

They’re going to increase the context window<a href="https:&#x2F;&#x2F;www.threads.net&#x2F;@zuck&#x2F;post&#x2F;C56MOZ3xdHI&#x2F;?xmt=AQGzjzazsycd7hUW10eWagOXTUp4Tq351z9JKlDvMU0JKA" rel="nofollow">https:&#x2F;&#x2F;www.threads.net&#x2F;@zuck&#x2F;post&#x2F;C56MOZ3xdHI&#x2F;?xmt=AQGzjzaz...</a>

Disappointed to note that the 8k context length is far short of Mixtral 8x22B&#x27;s 64k context length.Still, the published performance metrics are impressive. Kudos to Meta for putting these models out there.

The model card has the benchmark results relative to other Llama models including Llama 2: <a href="https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;MODEL_CARD.md#base-pretrained-models">https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;MODEL_CARD.md...</a>The dramatic performance increase of Llama 3 relative to Llama 2 (even Llama 2 13B!) is very impressive. Doubling the context window to 8k will open a lot of new oppertunities too.

The 400B model seems to be a competitor, maybe not in parameter count, but benchmark-wise it seems to be similar.

Maybe not the reason, but claude sonnet obliterates gpt3.5 and there isn&#x27;t a direct llama competitor to gpt4.

Interesting to see that their model comparisons don’t include OpenAI models.

Why would the CTO&#x2F;lead engineer admit that they nerfed the model even if they did? It’s all closed, how does admitting it benefit them? I would much rather trust the people using it everyday.

I suspect that there is some psychological effect going on where people adjust their expectations and start to be more open to noticing flaws after working with it for a while. Seems to be a recurring thing with most models.

Over yonder: <a href="https:&#x2F;&#x2F;x.com&#x2F;alexalbert__&#x2F;status&#x2F;1780707227130863674" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;alexalbert__&#x2F;status&#x2F;1780707227130863674</a>my $0.02: it makes me very uncomfortable that people misunderstand LLMs enough to even think this is possible

It&#x27;s likely true that they didn&#x27;t change the model, same for the many claims of GPT-4 getting worse. But they do keep iterating a lot on the &quot;safety&quot; layers on top: classifiers to detect dangerous requests, the main system prompt...But I also think it&#x27;s partially a psychological phenomenon, just people getting used to the magic and finding more bad edge-cases as it is used more.EDIT: It seems that they do claim that the layers on top also didn&#x27;t change <a href="https:&#x2F;&#x2F;twitter.com&#x2F;alexalbert__&#x2F;status&#x2F;1780707227130863674" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;alexalbert__&#x2F;status&#x2F;1780707227130863674</a>

They can change the prompt without changing the model, since the prompt only affects current &quot;attention.&quot;And they do.

Why do people keep saying that Claude3 has been nerfed? Their CTO has said on Twitter multiple times that not a single byte has been changed since its launch, so I&#x27;m curious why I keep hearing this.edit: having trouble finding the tweet I saw recently, it might have been from their lead engineer and not the CTO.

Do they publish the parameter size for Sonnet and Opus?

Better than Claude 3 Sonnet but Claude 3 Opus is significantly more powerful, albeit I&#x27;m not sure how they compare accounting for parameter size.

They’ve been generous with their AI models for a while now. The Starspace embedding model comes to mind and that was 7 years ago<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1709.03856" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1709.03856</a>

&gt; I&#x27;m so surprised that Meta is actually leading the open source AI landscape?Why? Meta has one of the most impressive open source track records out of anyone.

I don&#x27;t know anything about AI, but I assume they didn&#x27;t have a choice after Llama was leaked onto 4chan last year.

Plus these tools get better faster when more people use them. It&#x27;s a win-win.

Meta has the massive advantage of not needing to sell the AI. The primary purpose of Llama is to make Facebook, Instagram, Whatsapp, Quest etc. better (well, &quot;better&quot; from the perspective of the company). It is basically an internal tool. So just like React, Cassandra, PyTorch, GraphQL, HHVM and all of their other open source work they benefit from sharing it with the rest of the world. There is very little incremental cost, and they get to generate massive goodwill and attract talent because of it.

No surprise, Meta AI has been there since 2013 as FAIR and Meta has been doing open source ever since Facebook was made.I am surprised that this is a surprise to some, it just that some have not been paying attention.<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Meta_AI" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Meta_AI</a>

Llama* aren&#x27;t open source. They just have a relatively open license except for competitors.

Really? Is Llama 2 (70b?) better than Claude 3 sonnet?

I&#x27;m so surprised that Meta is actually leading the open source AI landscape?! I&#x27;ve used llama2 extensively and can&#x27;t wait to try out llama3 now. I can&#x27;t believe that it does better than Claude 3 in benchmarks (though admittedly claude 3 seems to have been nerfed recently)I sure do wish there was more info about how its trained and its training data.

As a European working on relocating out of the EU this is beyond hilarious.

ie America + a selection of countries that mostly haven’t got their shit together yet on dealing with the threat of unregulated AI.

&gt; We’re rolling out Meta AI in English in more than a dozen countries outside of the US. Now, people will have access to Meta AI in Australia, Canada, Ghana, Jamaica, Malawi, New Zealand, Nigeria, Pakistan, Singapore, South Africa, Uganda, Zambia and Zimbabwe — and we’re just getting started.

* beats in some cases, losses in some cases (vs gemini pro) and has only 8k context window.

* 8k context length* Llama3 8B beats Mistral 7B* Llama3 70B beats Claude 3 Sonnet and Gemini Pro 1.5

As a user, I&#x27;ve been putting in some long mathematical research papers and asking detailed questions about them in order to understand certain parts better. I feel some benefit from it because it can access the full context of the paper so it is less likely to misunderstand notation that was defined earlier etc.

I don&#x27;t need a million tokens, but 8k is absolutely too few for many of the use cases that I find important. YMMV.

Honestly, I swear to god, been working 12 hours a day with these for a year now, llama.cpp, Claude, OpenAI, Mistral, Gemini:The long context window isn&#x27;t worth much and is currently creating more problems than it&#x27;s worth for the bigs, with their &quot;unlimited&quot; use pricing models.Let&#x27;s take Claude 3&#x27;s web UI as an example. We build it, and go the obvious route: we simply use as much of the context as possible, given chat history.Well, now once you&#x27;re 50-100K tokens in, the initial prefill takes forever, O(10 seconds). Now we have to display a warning whenever that is the case.Now we&#x27;re generating an extreme amount of load on GPUs for prefill, and it&#x27;s extremely unlikely it&#x27;s helpful. Writing code? Previous messages are likely to be ones that needed revisions. The input cost is ~$0.02 &#x2F; 1000 tokens and it&#x27;s not arbitrary&#x2F;free, prefill is expensive and on the GPU.Less expensive than inference, but not that much. So now we&#x27;re burning ~$2 worth of GPU time for the 100K conversation. And all of the bigs use a pricing model of a flat fee per month.Now, even our _paid_ customers have to take message limits on all our models. (this is true, Anthropic quietly introduced them end of last week)Functionally:Output limit is 4096 tokens, so tasks that are a map function (ex. reword Moby Dick in Zoomer), need the input split into 4096 tokens anyway.The only use cases I&#x27;ve seen thus far that _legitimately_ benefit are needle in a haystack stuff, video with Gemini, or cases with huuuuuge inputs and small outputs, like, put 6.5 Harry Potter books into Gemini and get a Mermaid diagram out connecting characters.

Based on your use cases. 
I thought it&#x27;s not hard to push the window to 32K or even 100k if we change the position embedding

Awesome, but I am surprised by the constrained context window as it balloons everywhere else.Am I missing something? 8k seems quite low in current landscape.

Lex walked so that Dwarkesh could run. He runs the best AI podcast around right now, by a long shot.

I can&#x27;t express how good Dwarkesh&#x27;s podcast is in general.

TikTok (ByteDance) is now building an AGI team to train and advance LLMs (towards AGI), probably after realizing they are in a similar scenario.

The only thing the Reels algorithm is showing me are videos of ladies with fat butts. Now, I must admit, I may have clicked once on such a video. Should I now be damned to spend an eternity in ass hell?

Very interesting part around 5 mins in where Zuck says that they bought a shit ton of H100 GPUs a few years ago to build the recommendation engine for Reels to compete with TikTok (2x what they needed at the time, just to be safe), and now they are accidentally one of the very few companies out there with enough GPU capacity to train LLMs at this scale.

Intense exercise, especially a competetive sport where you train with other people tends to do this

I&#x27;ve noticed the same thing!  I think the personal confidence you build training hard MMA is a lot more empowering than the presonal confidence you build from making billions of dollars and being CEO of a gigantic company.  For those of us without the money, it seems hard to believe, but people are people even when they&#x27;re rich, and I&#x27;ve seen MMA change a lot of people in the same way.

MMA has a way to humble anyone.  It&#x27;s a great way to train for body and mind.

People may not like Joe Rogan but he described BJJ perfectly: 3D chess with consequences. It is a great way to relieve stress and forces you to temporarily forget about whatever is bothering you that day.

Alternatively, he’s completely relaxed here because he knows what he’s doing is genuinely good and people will support it. That’s gotta be a lot less stressful than, say, a senate hearing.

Now he starts to look like a proper redneck with that necklace.

Seems like a year or two of MMA has done way more for his charisma than whatever media training he&#x27;s done over the years. He&#x27;s a lot more natural in interviews now.

Zuck has an interview out for it as well, 
<a href="https:&#x2F;&#x2F;twitter.com&#x2F;dwarkesh_sp&#x2F;status&#x2F;1780990840179187715" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;dwarkesh_sp&#x2F;status&#x2F;1780990840179187715</a>

Improvement over Gemini 1.5 and Claude Sonnet on everything except MATH....kind of weird.

&gt;Llama 3 is pretrained on over 15T tokensWell, I remember when training GPT-3 on 300B was a lot.

Thanks for the link I just tested them and they also weark in europe without the need to start a VPN. What specs are needed to run these models. I mean the llama 70B and the Wizard 8Bx22 model. 
On your site they run very nicely and the answears they provide are really good they booth passed my small test and I would love to run one of them locally. 
So far I only ran 8B models on my 16GB RAM pc using LM Studio but having such good models run locally would be awesome.
I would upgrade my ram for that. My pc has an 3080 laptop GPU and I can increase the RAM to 64GB. As I understood it a 70B model needs around 64 GB but maybe only if it quantized. Can you confirm that? Can I run Llama 3 as well as you when I simply upgrade my RAM sticks. Or are you running it on a cloud and you can&#x27;t say much about the requirements for windows pc users? Or do you have hardware usage data for all the models on your site and you can tell us what they need to run?

I just hosted both models here: <a href="https:&#x2F;&#x2F;chat.tune.app&#x2F;" rel="nofollow">https:&#x2F;&#x2F;chat.tune.app&#x2F;</a>Playground: <a href="https:&#x2F;&#x2F;studio.tune.app&#x2F;" rel="nofollow">https:&#x2F;&#x2F;studio.tune.app&#x2F;</a>

Just got uploaded to HuggingFace: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;meta-llama&#x2F;Meta-Llama-3-8B" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;meta-llama&#x2F;Meta-Llama-3-8B</a> <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;meta-llama&#x2F;Meta-Llama-3-70B" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;meta-llama&#x2F;Meta-Llama-3-70B</a>

<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;meta-llama&#x2F;Meta-Llama-3-8B" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;meta-llama&#x2F;Meta-Llama-3-8B</a><a href="https:&#x2F;&#x2F;huggingface.co&#x2F;meta-llama&#x2F;Meta-Llama-3-70B" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;meta-llama&#x2F;Meta-Llama-3-70B</a><a href="https:&#x2F;&#x2F;llama.meta.com&#x2F;llama-downloads" rel="nofollow">https:&#x2F;&#x2F;llama.meta.com&#x2F;llama-downloads</a><a href="https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;download.sh">https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;download.sh</a>

Is there a download link for this model like LLAMA2 or is it going to be exclusively owned and operated by Meta this time?

“In the coming months, we expect to introduce new capabilities, longer context windows, additional model sizes, and enhanced performance, and we’ll share the Llama 3 research paper.”

Where does it say much better than gpt4 for the 400B model?

that&#x27;s very exciting. are you quoting same benchmark comparisons?

It absolutely does not say that. It in fact provides benchmarks that show it under performing them.Not great to blindly trust benchmarks, but there are no claims it will outperform GPT-4 or Opus.It was a checkpoint, so it&#x27;s POSSIBLE it COULD outperform.

Indeed. But if GPT-4 is actually 1.76T as rumored, an open-weight 400B is quite the achievement even if it&#x27;s only just competitive.

Comparing to the numbers here <a href="https:&#x2F;&#x2F;www.anthropic.com&#x2F;news&#x2F;claude-3-family" rel="nofollow">https:&#x2F;&#x2F;www.anthropic.com&#x2F;news&#x2F;claude-3-family</a> the ones of Llama 400B seem slightly lower, but of course it&#x27;s just a checkpoint that they benchmarked and they are still training further.

Is it decentralized? You can run it multiple places I guess, but it’s only available from one place.And it’s not open source.

It&#x27;s not open source or decentralized.

The blog did not state what you said, sorry I’ll have to downvote your comment

Lots of great details in the blog: <a href="https:&#x2F;&#x2F;ai.meta.com&#x2F;blog&#x2F;meta-llama-3&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ai.meta.com&#x2F;blog&#x2F;meta-llama-3&#x2F;</a>Looks like there&#x27;s a 400B version coming up that will be much better than GPT-4 and Claude Opus too. Decentralization and OSS for the win!

Use the API (or a product that uses the API)If you can trust Azure or AWS or GCP with your IP, you can get Claude 3 and GPT-4 Turbo through at least one of themIf your IP is so secret you can&#x27;t do that, then I wouldn&#x27;t imagine you&#x27;d be using the chat interface

I haven’t used it but I’ve heard <a href="https:&#x2F;&#x2F;cursor.sh&#x2F;" rel="nofollow">https:&#x2F;&#x2F;cursor.sh&#x2F;</a> might work?

Personally I don&#x27;t paste anything. I ask for code examples that demonstrate what I want, and then I adapt it to my needs. It&#x27;s definitely less powerful than directly sharing code, but it is what it is.I also run a personal language model server, but that is far less capable than the models available as services. It can still be better than nothing for code O can&#x27;t share with APIs.I also use gpt.el a but for editor integration, but I honestly haven&#x27;t workeded that into my workflow very much yet.

Unless you have the privilege of being an enterprise customer with an SLA guaranteeing privacy, there&#x27;s not much you can do other than using local models. I believe OpenAI says they don&#x27;t train based on API requests but that&#x27;s more of a &quot;trust me bro&quot; than any kind of guarantee.

Vscode with GitHub copilot is great, been using it for about a year and a half, no complaints. The business tier allegedly doesn’t save&#x2F;train on your data

I&#x27;m building Plandex (<a href="https:&#x2F;&#x2F;github.com&#x2F;plandex-ai&#x2F;plandex">https:&#x2F;&#x2F;github.com&#x2F;plandex-ai&#x2F;plandex</a>), a terminal-based AI coding tool which currently uses the OpenAI api--I&#x27;m working on support for Anthropic and OSS models right now and hoping I can ship it later today.You can self-host it so that data is only going to the model provider (i.e. OpenAI) and nowhere else, and it gives you fine-grained control of context, so you can pick and choose exactly which files you want to load in. It&#x27;s not going to pull in anything in the background that you don&#x27;t want uploaded.There&#x27;s a contributor working on integration with local models and making some progress, so that will likely be an option in the future as well, but for now it should at least be a pretty big improvement for you compared to the copy-paste heavy ChatGPT workflow.

I built a desktop tool to help reduce the amount of copy-pasting and improve the output quality for coding using ChatGPT or Claude: <a href="https:&#x2F;&#x2F;prompt.16x.engineer&#x2F;" rel="nofollow">https:&#x2F;&#x2F;prompt.16x.engineer&#x2F;</a>

Do you mind my asking, if you&#x27;re working on private codebases, how you go about using GPT&#x2F;Claude as a code assistant?
I&#x27;m just removing IP and pasting into their website&#x27;s chat interface.  I feel like there&#x27;s got to be something better out there but I don&#x27;t really know anyone else that&#x27;s using AI code assistance at all.

You&#x27;re asking for something that doesn&#x27;t exist yet. Command R+, WizardLM-2-8x22B probably come closest.

Nope, I don&#x27;t even see what the excitement is for.We seem to be in denial of the scaling problems we face in that we can&#x27;t even beat out the 1 year model.I subscribed and unsubscribed to Claude 3 in about an hour. It is just not better than chatGPT4.It is incredible to me that with all the motivation and resources of Meta, the best they can do is to produce a language model that isn&#x27;t worth the time to even bother trying if a chatGPT4 subscriber.

I am always excited to see these Open Weight models released, I think its very good for the ecosystem and definitely has its place in many situations.However since I use LLMs as a coding assistant (mostly via &quot;rubber duck&quot; debugging and new library exploration) I really don&#x27;t want to use anything other than the absolutely best in class available now. That continues to be GPT4-turbo (or maybe Claude 3).Does anyone know if there is any model out there that can be run locally and compete with GPT4-turbo? Or am I asking for something that is impossible?

Would love to see similar chart but llama 3 400b compared to the closed-source models like opus

You just need to give the wrong answer ;)

Fun fact, it&#x27;s impossible to 100% the MMLU because 2-3% of it has wrong answers.

If any one is interesting in seeing how 400B model compares with other opensource models, here is a useful chart: <a href="https:&#x2F;&#x2F;x.com&#x2F;natolambert&#x2F;status&#x2F;1780993655274414123" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;natolambert&#x2F;status&#x2F;1780993655274414123</a>

&gt; are all open models still just derivatives of llama?No there are several others but by far the best alongside llama are Mistral’s mistral and mixtral models. Those are called the foundation models which everyone else builds on top of

No, but there&#x27;s a reason so many adjacent libraries and software packages name themselves after Llama (Ollama, llama.cpp, etc)

Definitely not. Here&#x27;s 300+ LLMs without derivatives:<a href="https:&#x2F;&#x2F;lifearchitect.ai&#x2F;models-table&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lifearchitect.ai&#x2F;models-table&#x2F;</a>

this space moves impossibly fast...are all open models still just derivatives of llama?

It still say it is using llama2 if you ask it

I have been using Claude and ChatGPT in parallel for 2 months now. Claude is slow to respond, but I love it. So I will stay on Claude for next 3-6 months for now. I find it unnecessary to pay $40 for both, but not use to the fullest.So my plan is to use one chat subscription at any time, and then use APIs for the rest. Right now I have chatbox.app on my laptop, so have the freedom to play around w&#x2F; the rest as well as needed.

Any thoughts on which one you prefer? I have the ChatGPT subscription but curious about alternatives.

Just use the openrouter API to switch models with a chat frontend like Chatbox for Mac

Last week, I made the explicit decision to circle through paid subscriptions of various providers every few months, instead of sticking to just ChatGPT.Glad to see Meta.ai on the list, in addition to Claude, Gemini, Mistral and Perplexity.

15T tokens of data wow. Wonder if it a single epoch 15T data or multi epoch.

Did some quick testing of code tasks... Given db schema A, query and transform to data schema B... accomplished it very quickly.

Not an expert, but often weights are mentioned as not being open sourced.
Happy to get corrected, as I&#x27;m not really sure.

The parameters and the license. Mistral uses Apache 2.0, a neatly permissive open source license. As such, it&#x27;s an open source model.Models are similar to code you might run on a compiled vm or native operating system. Llama.cpp is to a model as Python is to a python script. The license lays out the rights and responsibilities of the users of the software, or the model, in this case. The training data, process, pipeline to build the model in the first place is a distinct and separate thing from the models themselves. It&#x27;d be nice if those were open, too, but when dealing with just the model:If it uses an OSI recognized open source license, it is an open source model.
If it doesn&#x27;t use an OSI recognized open source license, it&#x27;s not.Llama is not open source. It&#x27;s corporate freeware.

See this discussion and blog post about a model called OLMo from AI2 (<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39974374">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39974374</a>). They try to be more truly open, although here are nuances even with them that make it not fully open. Just like with open source software, an open source model should provide everything you need to reproduce the final output, and with transparency. That means you need the training source code, the data sets, the evaluation suites, the inference code, and more.Most of these other models, like Llama, are open weight not open source - and open weight is just openwashing, since you’re just getting the final output like a compiled executable. But even with OLMo (and others like Databrick’s DBRX) there are issues with proprietary licenses being used for some things, which prevent truly free use. For some reason in the AI world there is heavy resistance to using OSI-approved licenses like Apache or MIT.Finally, there is still a lack of openness and transparency on the training data sets even with models that release those data sets. This is because they do a lot of filtering to produce those data sets that happen without any transparency. For example AI2’s OLMo uses a dataset that has been filtered to remove “toxic” content or “hateful” content, with input from “ethics experts” - and this is of course a key input into the overall model that can heavily bias its performance, accuracy, and neutrality.Unfortunately, there is a lot missing from the current AI landscape as far as openness.

I think the proper way is to start the consideration from the perspective of what the user can do, for example, from the four freedoms of FSF (<a href="https:&#x2F;&#x2F;www.gnu.org&#x2F;philosophy&#x2F;free-sw.html.en#four-freedoms" rel="nofollow">https:&#x2F;&#x2F;www.gnu.org&#x2F;philosophy&#x2F;free-sw.html.en#four-freedoms</a>)The starting point is the ability to run the LLM as you wish, for any purpose - so if a license prohibits some uses and you have to start any usage with thinking whether it&#x27;s permitted or not, that&#x27;s a fail.Then the freedom where &quot;source&quot; matters is the practical freedom to change the behavior so it does your computing as you wish. And that&#x27;s a bit tricky - since one interpretation would require having the training data, training code and parameters; but for current LLMs the training hardware and cost of running it is a major practical limitation, so much that one could argue that the ability to change the behavior (which is the core freedom that we&#x27;d like) is separate from the ability to recreate the model, and would be more relevant in the context of the &quot;instruction training&quot; which happens after the main training, is the main determiner of behavior (as opposed to capability), and so the main &quot;source would be the data for that (instruct training data, and the model weights before that finetuning) so that you can fine-tune the model on different instructions, which requires much less resources than training it from scratch, and don&#x27;t have to start with the instructions and values imposed on the LLM by someone else.

What is &quot;source&quot; regarding an LLM? Public training data and initial parameters?

Models are mostly fungible, if meta decided to play games it&#x27;s not too hard to switch models.  I think this is mostly a CYA play.

If they can develop any evidence at all (perhaps from a whistleblower, perhaps from some characteristic unique to their model), they can sue and then there&#x27;s they get to do &quot;discovery&quot;, which would force the sued party to reveal details.

&gt; When the threshold between annoying and profitable is crossed, Meta&#x27;s lawyers will start sniping and acquiring users of their IP.I&#x27;m curious: given that the model will probably be hosted in a private server, how would meta know or prove that someone is using their model against the license?

I don&#x27;t support GP&#x27;s claims, but you have to realize that you&#x27;re &quot;almost everyone&quot; up until you build something very successful with lots of capital at stake, and then you definitely become &quot;someone special&quot; and have to think ahead about how the licenses of your models impact you.Of course random individuals don&#x27;t care much about the licenses on their personal AI projects. But if you intend to grow something significant, you better read the label from the start.

Would you then say that in general Open Source doesn&#x27;t matter for almost everyone? Most people running Linux aren&#x27;t serving 700 million customers or operating military killbots with it after all.

Yes or no, do you conceed that for almost everyone, none of what you said matters, and almost everyone can use llama 3 for their use case, and that basically nobody is going to have to worry about being sued, other than maybe like Google, or equivalent?You are using all these scary words without saying the obvious, which is that for almost everyone, none of that matters.

Those additional restrictions mean it&#x27;s not an open source license by the OSI definition, which matters if you care about words sometimes having unambiguous meanings.I call models like this &quot;openly licensed&quot; but not &quot;open source licensed&quot;.

That &quot;etc&quot; is doing a lot of work here. The point of OSI licenses like MIT, Apache 2.0 is to remove the &quot;etc&quot;. The licensing company gives up its right to impose acceptable use policies. More restrictive, but still OSI approved, licenses are as clear as they possibly can be about allowed uses and the language is as unambiguous as possible. Neither is the case for the Llama AUP.

That&#x27;s convenient because I only have 699,999,999 customers.

any scale restrictions plus the &quot;etc.&quot; means it&#x27;s not open source.

&gt; &quot;Nothing about Meta&#x27;s license is open source. It&#x27;s a carefully constructed legal agreement intended to prevent any meaningful encroachment by anyone, ever, into any potential Meta profit, and to disavow liability to prevent reputational harm in the case of someone using their freeware for something embarrassing.&quot;You seem to be making claims that have little connection to the actual license.The license states you can&#x27;t use the model if, at the time Llama 3 was released, you had &gt;700 million customers. It also says you can&#x27;t use it for illegal&#x2F;military&#x2F;etc uses. Other than that, you can use it as you wish.

The license prohibits Snap from using it. Meta knows that it would be useful to Snap and that they don’t have the resources to train their own model.

What are the practical use cases where the license prohibits people from using llama models? There are plenty of startups and companies that already build their business on llamas (eg phind.com). I do not see the issues that you assume exist.If you get that successful that you cannot use it anymore (have 10% of earth&#x27;s population as clients) probably you can train your own models already.

&gt; Turning everything into some sort of theological debate takes away a lot of credit that Meta deserves.It&#x27;s not theological, it&#x27;s the misuse of a specific legal definition that we all have interest in maintaining. &quot;Freely available models&quot; or &quot;open license&quot; are accurate.Other companies keeping things for themselves doesn&#x27;t warp reality, or the existing definitions we use to describe it. Giving them the credit they deserve, especially in comparison to the others, should be enough.

It doesn’t mean it’s a bad license, just that it doesn’t meet the definition. There are legitimate reasons for companies to use source-available licenses. You still get to see the source code and do some useful things with it, but read the terms to see what you can do.Meanwhile, there are also good reasons not to water down a well-defined term so it becomes meaningless like “agile” or “open.”This gets confusing because people want to use “open source” as a sort of marketing term that just means it’s good, so if you say it’s not open source that’s taken to imply it’s bad.

&gt; In spirit I think this is “open source”, and I think that’s how the majority of people think.No, it isn&#x27;t. You do, but, as evidenced by other comments, there&#x27;s clearly people that don&#x27;t. Thinking that you&#x27;re with the majority and it&#x27;s just a vocal minority is one thing, but it could just as easily be said that the vocal groups objecting to your characterization are representative of the mainstream view.If we look at these models as the output of a compiler, that we don&#x27;t have the inputs to, but that we are free (ish) to use and modify and redistribute, it&#x27;s a nice grant from the copyright holder, but that very much doesn&#x27;t look like open source. Open source, applied to AI models would mean giving us (a reference to) the dataset and the code used to train the model so we could tweak it to train the model slightly differently. To be less apologetic or something by default, instead of having to give it additional system instructions.Model Available(MA) is freer than Model unavailable, and it&#x27;s more generous than model unavailable, but it&#x27;s very much not in the spirit of open source. I can&#x27;t train my own model using what Meta has given us here.And just to note, Google Gemma is the one they are releasing weights for. They are doing this and deserve credit for it.

I think you&#x27;re conferring one hell of a lot of credit to Meta that is entirely undeserved. This is not a charitable, net benefit to humanity organization. These are not the good guys. These people are responsible for one hell of a lot of harm, and imagining they have good intentions is naive at best. I don&#x27;t doubt the individual software engineers and researchers are good people. It&#x27;s the corporation that&#x27;s in charge of the llama product, however, and it&#x27;s the lawyers, executives, and middle management that will start cracking down on technicalities and violations of the license. The precise instant that it becomes more profitable and less annoying to sue someone for violation of the license, Meta&#x27;s lawyers will do so, because that&#x27;s what companies are obligated to do. The second some group of shareholders start pointing out blatant violations of the license in products using llama, the lawyers will be obligated to crack down.Meta is a corporation, and not subject to rational, good faith human judgment. It&#x27;s a construct that boils down to an algorithmic implementation of the rules, regulations, internal policies, communication channels, and all those complex interactions that effectively prevent sensible, good faith human intervention at any given stage that would even allow the company to just let people continue to violate their stated license. Like trademarks, if you don&#x27;t enforce a contract, the inaction dissipates your ability to enforce it later on. They don&#x27;t pay these lawyers to come up with these licenses and contracts for shits and giggles.The license is not the outcome of a happy weekend brainstorm session tacked on ad hoc just to maximize the benefit to humanity and blissfully join the wide world of open source.The license is intended to prevent any serious competitive use of their AI models by third parties. It was crafted deliberately and carefully and expensively. They didn&#x27;t use existing open source licenses because no license offered them the particular mix of rights and restrictions that fit their overall strategy. It&#x27;s for PR, the ability to stifle competition, to get free beta testing and market research, and 100% of every part of the license is intentional and an insidious perversion of the idea of &quot;open.&quot;Meta doesn&#x27;t deserve credit, they deserve condemnation. They could have gone with any number of open source licenses, using GPL or CC licensing with specific provisions to protect their interests and prevent commercial exploitation, or use dual licensing to incentivize different tiers of access. They deliberately and with a high level of effort pursued their own invented license. They are using weasel words and claiming they are open source all over the place in order to foster good will.The argument &quot;but nobody has been sued&quot; is more than a little silly. There&#x27;s simply no product known to use their models currently on the market that&#x27;s both a blatant enough violation and worth enough money to sacrifice the good will they&#x27;ve been fostering. There&#x27;s no human in organizations that size with the capacity to step in and prevent the lawsuits from happening. It&#x27;ll be a collective, rules and policies decision completely out of anyone&#x27;s hands to prevent, even if Zuck himself wanted to intervene. The shareholders&#x27; interests reign supreme.Meta isn&#x27;t a moral institution. 
It&#x27;s a ruthlessly profitable one.

I don’t understand how the idea of open source become some sort of pseudo-legalistic purity test on everything.Models aren’t code, some of the concepts of open source code don’t map 1:1 to freely available models.In spirit I think this is “open source”, and I think that’s how the majority of people think.Turning everything into some sort of theological debate takes away a lot of credit that Meta deserves. Google isn’t doing this. OpenAI sure as fuck isn’t.

Actually right now the OSI is hosting ongoing discussion this year on what it means for AI to be open source. Here is their latest blog post on the subject:<a href="https:&#x2F;&#x2F;opensource.org&#x2F;blog&#x2F;open-source-ai-definition-weekly-update-april-15" rel="nofollow">https:&#x2F;&#x2F;opensource.org&#x2F;blog&#x2F;open-source-ai-definition-weekly...</a>Here is the latest draft definition:<a href="https:&#x2F;&#x2F;hackmd.io&#x2F;@opensourceinitiative&#x2F;osaid-0-0-7" rel="nofollow">https:&#x2F;&#x2F;hackmd.io&#x2F;@opensourceinitiative&#x2F;osaid-0-0-7</a>And a discussion about the draft:<a href="https:&#x2F;&#x2F;discuss.opensource.org&#x2F;t&#x2F;draft-v-0-0-7-of-the-open-source-ai-definition-is-available-for-comments&#x2F;298" rel="nofollow">https:&#x2F;&#x2F;discuss.opensource.org&#x2F;t&#x2F;draft-v-0-0-7-of-the-open-s...</a>

&quot;Llama is not open source.&quot;This is interesting. Can you point me to an OSI discussion what would constitute an open source license for LLMs? Obviously they have &quot;source&quot; (network definitions) and &quot;training data&quot; and &quot;weights&quot;.I&#x27;m not aware of any such discussion.

(We detached this subthread from <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40077832">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40077832</a>)

<a href="https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;LICENSE">https:&#x2F;&#x2F;github.com&#x2F;meta-llama&#x2F;llama3&#x2F;blob&#x2F;main&#x2F;LICENSE</a>Llama is not open source. It&#x27;s corporate freeware with some generous allowances.Open source licenses are a well defined thing. Meta marketing saying otherwise doesn&#x27;t mean they get to usurp the meaning of a well understood and commonly used understanding of the term &quot;open source.&quot;<a href="https:&#x2F;&#x2F;opensource.org&#x2F;license" rel="nofollow">https:&#x2F;&#x2F;opensource.org&#x2F;license</a>Nothing about Meta&#x27;s license is open source. It&#x27;s a carefully constructed legal agreement intended to prevent any meaningful encroachment by anyone, ever, into any potential Meta profit, and to disavow liability to prevent reputational harm in the case of someone using their freeware for something embarrassing.If you use it against the license anyway, you&#x27;ll just have to hope you never get successful enough that it becomes more profitable to sue you and take your product away than it would be annoying to prosecute you under their legal rights. When the threshold between annoying and profitable is crossed, Meta&#x27;s lawyers will start sniping and acquiring users of their IP.

What does not get a lot of attention is the work they have done with OCP (Open Compute: <a href="https:&#x2F;&#x2F;www.ocp.org&#x2F;en-us" rel="nofollow">https:&#x2F;&#x2F;www.ocp.org&#x2F;en-us</a>) to first launch it in 2012 and then nurture and grow it since then.One project I am really excited to see grow is Overture Maps(<a href="https:&#x2F;&#x2F;overturemaps.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;overturemaps.org&#x2F;</a>), I think it&#x27;ll have an impact, from end user perspective, far larger than perhaps all other projects.

Yeah, I think Meta is the reason LLMs is widely available to everyone instead of reliant on something proprietary

The amount of open source stuff Facebook (Meta) has given us over the years is astounding. pytorch, dall-e, react, react native, graphql, cassandra, tor. Commoditized VR, love my quest 3. Just an incredible track record. We&#x27;re lucky they release all this stuff for free. The Zuck is one of a kind.

&gt; Meta AI isn&#x27;t available yet in your countrySad this geofencing

The &quot;catch&quot; is that they are releasing these models in order to commoditize the technology of their competitors.By releasing these models for free, they drive down the price&#x2F;advantage of competing companies.

It&#x27;s by Facebook, currently hidden behind the veil of Meta, so what&#x27;s the catch?

&gt; Might be time to invest in a Mac studio!it&#x27;s wild isn&#x27;t itfor so long a few years old macbook is fine for everything, in desperation Apple waste their time with VR goggles in search of a use-case... then suddenly ChatGPT etc comes along and despite relatively weak GPU Apple accidentally have stuff worth upgrading toimagine when they eventually take the goggles off and start facing in the right direction...

The money making opportunity is releasing PCs&#x2F;laptops with 10x as much RAM.

It&#x27;s a dense one, zuck confirms this a couple minutes into the interview posted in this thread

Just FYI on the podcast video
Zuck seems to let it slip that the exact number is 405B. (2-3mins in)

It might be large enough Q2 won&#x27;t impact performance too much (not guaranteed), but that&#x27;s a great point.Time to invest in a Milan-X! (Already have the 512GB of DDR4)

&gt; Might be time to invest in a Mac studio!The highest end Mac Studio with 196GB of ram won&#x27;t even be enough to run a Q4 quant of the 400B+ (don&#x27;t forget the +) model. At this point, one have to consider an Epyc for CPU inference or costlier gpu solutions like the &quot;popular&quot; 8xA100 80GB...An if it&#x27;s a dense model like the other llamas, it will be pretty slow..

Quick thoughts -Major arch changes are not that major, mostly GQA and tokenizer improvements. Tokenizer improvement is a under-explored domain IMO.15T tokens is a ton!400B model performance looks great, can’t wait for that to be released. Might be time to invest in a Mac studio!OpenAI probably needs to release GPT-5 soon to convince people they are still staying ahead.

It is. You can see a little &quot;G&quot; icon indicating that it searched the web with Google.

It certainly doesn&#x27;t, I&#x27;m running the 7B locally with ollamaIt provided a lot more detail about the case, but does not have current information. It hallucinated the question about juror count, or maybe confused it with a different case seems more likely, one of the E Jean Carroll cases or the SDNY Trump Org financial fraud case?

It&#x27;s likely RAG &#x2F; augmented with web data.  Would be interested if local execution returned the same results.

Do you mean this? <a href="https:&#x2F;&#x2F;trackingai.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;trackingai.org&#x2F;</a>

I recall there was a website tracking the ideological bias of LLMs, but I can’t find it now. But it was showing where all the LLMs rank on a political graph with four quadrants. I think we need something like that, ranking these LLMs on aspects like censorship.Example: <a href="https:&#x2F;&#x2F;www.technologyreview.com&#x2F;2023&#x2F;08&#x2F;07&#x2F;1077324&#x2F;ai-language-models-are-rife-with-political-biases&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.technologyreview.com&#x2F;2023&#x2F;08&#x2F;07&#x2F;1077324&#x2F;ai-langu...</a>But I think some other site was doing this ‘live’ and adding more models as they appeared.

Initial observations from the Meta Chat UI...1. fast2. less censored than other mainstream models3. has current data, cites sourcesI asked about Trump&#x27;s trial and it was happy to answer. It has info that is hours old---Five jurors have been selected so far for the hush money case against former President Donald Trump ¹. Seven jurors were originally selected, but two were dismissed, one for concerns about her impartiality and the other for inaccuracies in his responses during questioning ².

Where did you find this number? Not doubting it, just want to get a better idea of how precise the estimate may be.

Once benchmarks exist for a while, they become meaningless - even if it&#x27;s not specifically training on the test set, actions (what used to be called &quot;graduate student descent&quot;) end up optimizing new models towards overfitting on benchmark tasks.

I actually can&#x27;t wrap my head around this number, even though I have been working on and off with deep learning for a few years. The biggest models we&#x27;ve ever deployed on production still have less than 1B parameters, and the latency is already pretty hard to manage during rush hours. I have no idea how they deploy (multiple?) 1.8T models that serve tens of millions of users a day.

The original GPT4 may have been around that size (16x 110B).But it&#x27;s pretty clear GPT4 Turbo is a smaller and heavily quantized model.

For the still training 400B:<pre><code> Llama 3 GPT 4(Published)
 BBH 85.3 83.1
 MMLU 86.1 86.4
 DROP 83.5 80.9
 GSM8K 94.1 92.0 
 MATH 57.8 52.9
 HumEv 84.1 74.4
</code></pre>
Although it should be noted that the API numbers were generally better than published numbers for GPT4.[1]: <a href="https:&#x2F;&#x2F;deepmind.google&#x2F;technologies&#x2F;gemini&#x2F;" rel="nofollow">https:&#x2F;&#x2F;deepmind.google&#x2F;technologies&#x2F;gemini&#x2F;</a>

Not quite there yet, but very close and not done training! It&#x27;s quite plausible that this model could be state of the art over GPT-4 in some domains when it finishes training, unless GPT-5 comes out first.Although 400B will be pretty much out of reach for any PC to run locally, it will still be exciting to have a GPT-4 level model in the open for research so people can try quantizing, pruning, distilling, and other ways of making it more practical to run. And I&#x27;m sure startups will build on it as well.

This is tantalizingly close in multiple benchmarks though. Pretty sure this one will finally be the open GPT-4 match.

The bottom of <a href="https:&#x2F;&#x2F;ai.meta.com&#x2F;blog&#x2F;meta-llama-3&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ai.meta.com&#x2F;blog&#x2F;meta-llama-3&#x2F;</a> has in-progress results for the 400B model as well. Looks like it&#x27;s not quite there yet.<pre><code> Llama 3 400B Base &#x2F; Instruct
 MMLU 84.8 86.1
 GPQA - 48.0
 MATH - 57.8
 HumanEval - 84.1
 DROP 83.5 -</code></pre>

also curious how it compares to WizardLM 2 8x22B

it&#x27;s in the official post the comparison with Mixtral 8x22B

Has anyone prepared a comparison to Mixtral 8x22B?  (Life sure moves fast.)

Via Microsoft Copilot  (and perhaps Bing?) you can get access to GPT-4 for free.

I was particularly excited for the high HumanEval score, and this is before the 400B model and the CodeLlama tune!I just added Llama 3 70B to our coding copilot <a href="https:&#x2F;&#x2F;www.double.bot">https:&#x2F;&#x2F;www.double.bot</a> if anyone wants to try it for coding within their IDE

But I&#x27;m waiting for the finetunedz&#x2F;merged models. Many devs produced great models based on Llama 2, that outperformed the vanilla one, so I expect similar treatment for the new version. Exciting nonetheless!

I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don&#x27;t compare directly themselves.<pre><code> Llama 3 8B Llama 3 70B GPT-4
 MMLU 68.4 82.0 86.5
 GPQA 34.2 39.5 49.1
 MATH 30.0 50.4 72.2
 HumanEval 62.2 81.7 87.6
 DROP 58.4 79.7 85.4
</code></pre>
Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven&#x27;t found comprehensive eval numbers for the latest GPT-3.5, however I believe Llama 3 70B handily beats it and even the 8B is close. It&#x27;s very exciting to have models this good that you can run locally and modify!GPT-4 numbers from from <a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;simple-evals">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;simple-evals</a> gpt-4-turbo-2024-04-09 (chatgpt)

Meta Llama 3