But that would always be the case. Twitter will not last forever; heck, it may not even be long before an open alternative like Bluesky competes with it. Would be interesting to know what percentage of the original mined data was from Twitter.

The point was it’s getting harder and harder to do that as things get locked down or go behind a massive paywall to either profit off of or avoid being used in generative AI. The places where previous versions got data is impossible to gather from anymore so the dataset you would collect would be completely different, which (might) cause weird skewing.

It might be fun to collect the same data if not for any other reason than to note the changes but adding the caveat that it doesn’t represent human output.Might even change the tool name.

Yeah pay an illustrator if this is important to you.See a lot of people upset about AI still using AI image generation because it&#x27;s not in their field so they feel less strongly about it and can&#x27;t create art themselves anyway, hypocritical either use it or don&#x27;t but don&#x27;t fuss over it then use it for something thats convenient for you.

The buildings and shipping containers that store low background steel aren&#x27;t built out of the stuff either.

I am not sure we should trust a site contaminated by AI graphics. &#x2F;s

The sources are just aggregated. The source doesn&#x27;t change.The new stuff generated does (and this is honestly already captured).This author doesn&#x27;t generate content. They analyze data from humans. That &quot;from humans&quot; is the part that can&#x27;t be discerned enough and thus the project can&#x27;t continue.Their research and projects are great.

The main concerns expressed in Robyn&#x27;s note, as I read them, seem to be 1) generative AI has polluted the web with text that was not written by humans, and so it is no longer feasible to produce reliable word frequency data that reflects how humans use natural language; and 2) simultaneously, sources of natural language text that were previously accessible to researchers are now less accessible because the owners of that content don&#x27;t want it used by others to create AI models without their permission. A third concern seems to be that support for and practice of any other NLP approaches is vanishing.Making resources like wordfreq more visible won&#x27;t exacerbate any of these concerns.

That&#x27;s exactly the opposite of what the author wanted IMO. The author no more wants to be a part of this mess. Aggregating these sources would just makes it so much more easier for the tech giants to scrape more data.

I was taking &quot;Release of ChatGPT&quot; as the Trinity date.

Congratulations on &quot;shipping&quot;, I&#x27;ve had a background task to create pretty much exactly this site for a while. What is your cutoff date? I made this handy list, in research for mine:<pre><code> 2017: Invention of transformer architecture
 June 2018: GPT-1
 February 2019: GPT-2
 June 2020: GPT-3
 March 2022: GPT-3.5
 November 2022: ChatGPT
</code></pre>
You may want to add kiwix archives from before whatever date you choose. You can find them on the Internet Archive, and they&#x27;re available for Wikipedia, Stack Overflow, Wikisource, Wikibooks, and various other wikis.

I too realised this parallel and frequently tell people about it.Edit: just the first one

:&#x27;( I thought I was clever for realising this parallel myself! Guess it&#x27;s more obvious than I thought.Another example is how data on humans after 2020 or so can&#x27;t be separated by sex because gender activists fought to stop recording sex in statistics on crime, medicine, etc.

You can either add them to the site yourself via Tumblr or send them to me via email (jgc@cloudflare).

FYI: My two datasets, DebateSum and OpenDebateEvidence&#x2F;OpenCaseList in their current forms qualify for this, as they end at latest in 2022.

I created <a href="https:&#x2F;&#x2F;lowbackgroundsteel.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;lowbackgroundsteel.ai&#x2F;</a> in 2023 as a place to gather references to unpolluted datasets. I&#x27;ll add wordfreq. Please submit stuff to the Tumblr.

One way to tackle it would be to use LLMs to generate synthetic corpuses, so you have some good fingerprints for pollution. But even there I&#x27;m not sure how doable that is given the speed at which LLMs are being updated. Even if I know a particular page was created in, say, January 2023, I may no longer be able to try to generate something similar now to see how suspect it is, because the precise setups of the moment may no longer be available.

Arguably it is a form of language evolution. I bet humans have started using &quot;delve&quot; more too, on average. I think the best we can do is look at the trends and think about potential causes.

I guess it would be interesting but differentiating pollution from language evolution seems very tricky since getting a non polluted corpus gets harder and harder

I wonder if anyone will fork the project. Apart from anything else, the data may still be useful given that we know it is polluted. In fact, it could act as a means of judging the impact of LLMs via that very pollution.

perhaps germane to this thread, I think the scooter thing was an investment bubble. it was easier to burn investment money on new scooters than to collect and maintain old ones. until the money ran out.

Did they? A lot of then were barely used, got damaged or vandalized, etc. And when the companies folded or communities outlawed the scooters, they end up as trash. I don&#x27;t believe for a second that the amount of pollutants and greenhouse gasses saved by usage is larger than the amount produced by manufacturing, shipping and trashing all those scooters.

At least scooters did something useful for the environment.

Man the AI folks really wrecked everything. Reminds me of when those scooter companies started just dumping their scooters everywhere without asking anybody if they wanted this.

Is it the same companies that simply took all the writers&#x27; previous work (hoping to be billionaires before the courts understand)?

This idea could also be extended to domains like Art. Create new art styles for AI to learn from. But in future, that will also get automated. AI itself will create art styles and all humans would do is choose whether something is Hot or Not. Sort of like art breeder.

AI companies are indeed hiring such people to generate customized training data for them.

Transformers read a lot during training, it might actually be beneficial for the companies to the point those works never see the light of day, only machines would read them. That&#x27;s so dystopian I&#x27;d say those works should be published so they eventually get into the public domain.

There are already several companies doing this - I do occasional contract work for a couple -, and paying rates sometimes well above what an average earning writer can expect elsewhere. However, the vast majority of writers have never been able to make a living from their writing. The threshold to write is too love, too many people love it, and most people read very little.

Have you ever read american history? Lol.

People have been paid to generate noise for a decade+ now. Garbage in, garbage out will always be true.Next token-seeking is a solved problem. Novel thinking can be solved by humans and possibly by AI soon, but adding more garbage to the data won&#x27;t improve things.

_Thank you_.  I read this story probably around 1980 (I think in a magazine that was subsequently trashed or garage-saled), and I have spent my adult life remembering the bones of the story, but not the author or the title.

Who programs the tapes? <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Profession_(novella)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Profession_(novella)</a>

All those writers who&#x27;ll soon be out of job and&#x2F;or already are and basically unhireable for their previous tasks should be paid for by the AI hyperscalers to write anything at all on one condition: not a single sentence in their works should be created with AI.(I initially wanted to say &#x27;paid for by the government&#x27; but that&#x27;d be socialising losses and we&#x27;ve had quite enough of that in the past.)

<a href="https:&#x2F;&#x2F;pshapira.net&#x2F;2024&#x2F;03&#x2F;31&#x2F;delving-into-delve&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pshapira.net&#x2F;2024&#x2F;03&#x2F;31&#x2F;delving-into-delve&#x2F;</a> &quot;Delving into “delve”&quot;

From the submission you&#x27;re commenting on:&gt; As one example, Philip Shapira reports that ChatGPT (OpenAI&#x27;s popular brand of generative language model circa 2024) is obsessed with the word &quot;delve&quot; in a way that people never have been, and caused its overall frequency to increase by an order of magnitude.

The fun thing is that while GPTs initially learned from humans (because ~100% of the content was human-generated), future humans will learn from GPTs, because almost all available content would be GPT-generated very soon.This will surely affect how we speak. It&#x27;s possible that human language evolution could come to a halt, stuck in time as AI datasets stop being updated.In the worst case, we will see a global &quot;model collapse&quot; with human languages devolving along with AI&#x27;s, if future AIs are trained on their own outputs...

&gt; I&#x27;d love to see by how much the use of world &quot;delve&quot; has increased since 2021...There are charts &#x2F; graphs in the link, both since 2021, and since earlier.The final graph suggests the phenomenon started earlier, possibly correlated in some way to Malaysian &#x2F; Indian usages of English.It does seem OpenAI&#x27;s family of GPTs as implemented in ChatGPT unspool concepts in a blend of India-based-consultancy English with American freshmen essay structure, frosted with superficially approachable or upbeat blogger prose ingratiatingly selling you something.Anthropic has clearly made efforts to steer this differently, Mistral and Meta as well but to a lesser degree.I&#x27;ve wondered if this reflects training material (the SEO is ruining the Internet theory), or is more simply explained by selection of pools of Hs hired for RLHF.

Same for me but with the word “crucial”.

Amusing that we now have a feedback loop. Let&#x27;s see... delve delve delve delve delve delve delve delve. There, I&#x27;ve done my part.

Sad. I&#x27;d love to see by how much the use of world &quot;delve&quot; has increased since 2021...

It certainly feels like the amount of regurgitated, nonsensical, generated content (nontent?) has risen spectacularly specifically in the past few years. 2021 sounds about right based on just my own experience, even though I can&#x27;t point to any objective source backing that up.

Aunt may&#x27;s brownie recipe (or at least her thoughts on it) are likely something you&#x27;d want if you want to reflect how humans use language. Both news-style and encyclopedia-style writing represent a pretty narrow slice.

&gt; The training data is weighted by a quality metricAt least in Googles case, they&#x27;re having so much difficulty keeping AI slop out of their search results that I don&#x27;t have much faith in their ability to give it an appropriately low training weight. They&#x27;re not even filtering the comically low-hanging fruit like those YouTube channels which post a new &quot;product review&quot; every 10 minutes, with an AI generated thumbnail and AI voice reading an AI script that was never graced by human eyes before being shat out onto the internet, and is of course always a glowing recommendation since the point is to get the viewer to click an affiliate link.Google has been playing the SEO cat and mouse game forever, so can startups with a fraction of the experience be expected to do any better at filtering the noise out of fresh web scrapes?

The current state of things leads me to believe that Google&#x27;s current ranking system has been somehow too transparent for the last 2-3 years.The top of search results is consistently crowded by pages that obviously game ranking metrics instead of offering any value to humans.

Yes but not quite as far as you imply.  The training data is weighted by a quality metric, articles written by journalists and wikipedia contributors are given more weight than Aunt May&#x27;s brownie recipe and corpoblogspam.

Indexability is orthogonal to readability.

We do know that the open web consitutes the bulk of the trainig data, although we don&#x27;t get to know the specific webpages that got used. Plus some more selected sources, like books, of which again we only know that those are books but not which books were used. So it&#x27;s just a matter of probability that there was a good amount of SEO spam as well.

&gt;And yet LLMs were still fed articles written for Googlebot, not humans.How do we know what content LLMs were fed? Isn&#x27;t that a highly guarded secret?Won&#x27;t the quality of the content be paramount to the quality of the generated output or does it not work that way?

I think it&#x27;s an arms race, but it&#x27;s an open question who wins.For a while I thought email as a medium was doomed, but spammers mostly lost that arms race. One interesting difference is that with spam, the large tech companies were basically all fighting against it. But here, many of the large tech companies are either providing tools to spammers (LLMs) or actively encouraging spammy behaviors (by integrating LLMs in ways that encourage people to send out text that they didn&#x27;t write).

I hope this trend accelerates to force us all into grass-touching and book-reading.  The sooner, the better.

This feels like a second, magnitudes larger Eternal September. I wonder how much more of this the Internet can take before everyone just abandons it entirely. My usage is notably lower than it was in even 2018, it&#x27;s so goddamn hard to find anything worth reading anymore (which is why I spend so much damn time here, tbh).

At some point though you have to acknowledge that a specific use of language belongs to the medium through which you&#x27;re counting word frequencies. There are also specific writing styles (including sentence&#x2F;paragraph sizes, unnecessary repetitions, focusing on other metrics than readability) associated with newspapers, novels, e-mails to your boss, anything really. As long as text was written by a human who was counting on at least some remote possibility that another human might read it, this is way more legitimate use of language than just generating it with a machine.

Isn&#x27;t it the other way around?SEO text carefully tuned to tf-idf metrics and keyword stuffed to them empirically determined threshold Google just allows should have unnatural word frequencies.LLM content should just enhance and cement the status quo word frequencies.Outliers like the word &quot;delve&quot; could just be sentinels, carefully placed like trap streets on a map.

serpent eating its own tailGOGI.

&gt; I agree in general but the web was already polluted by Google&#x27;s unwritten SEO rules. Single-sentence paragraphs, multiple keyword repetitions and focus on &quot;indexability&quot; instead of readability, made the web a less than ideal source for such analysis long before LLMs.Blog spam was generally written by humans. While it sucked for other reasons, it seemed fine for measuring basic word frequencies in human-written text. The frequencies are probably biased in some ways, but this is true for most text. A textbook on carburetor maintenance is going to have the word &quot;carburetor&quot; at way above the baseline. As long as you have a healthy mix of varied books, news articles, and blogs, you&#x27;re fine.In contrast, LLM content is just a serpent eating its own tail - you&#x27;re trying to build a statistical model of word distribution off the output of a (more sophisticated) model of word distribution.

It&#x27;s crazy to attribute the downfall of the web&#x2F;search to Google. What does Google have to do with all the genuine open web content, Google&#x27;s source of wealth, getting starved by (increasingly) walled gardens like Facebook, Reddit, Discord?I don&#x27;t see how Google&#x27;s SEO rules being written or unwritten has any bearing. Spammers will always find a way.

Prior to Google we had Altavista and in those days it was incredibly common to find keywords spammed hundreds of times in white text on a white background in the footer of a page. SEO spam is not new, it&#x27;s just different.

&gt; ML&#x2F;LLM is the second iteration of writing pollution. The first was humans writing for corporate bots, not other humans.Based on the process above, naturally, the third iteration then is LLMs writing for corporate bots, neither for humans nor for other LLMs.

Don&#x27;t forget Google&#x27;s adsense rules which penalized useful straightforward websites and mandated websites be full of &quot;content&quot;.  Doesn&#x27;t matter if the &quot;content&quot; is garbage nonsense rambling and excessive word use - it&#x27;s content and much more likely to be okayed by adsense!

I agree in general but the web was already polluted by Google&#x27;s unwritten SEO rules. Single-sentence paragraphs, multiple keyword repetitions and focus on &quot;indexability&quot; instead of readability, made the web a less than ideal source for such analysis long before LLMs.It also made the web a less than ideal source for training. And yet LLMs were still fed articles written for Googlebot, not humans. ML&#x2F;LLM is the second iteration of writing pollution. The first was humans writing for corporate bots, not other humans.

What was Reddit’s excuse? They did roughly the same thing (and have just as much garbage content).In other words, why is it wrong for X but okay for Reddit? If you ignore one individual’s politics, the two services did the same thing.

&gt; What beautiful doublethink.Given just how many AI bots scrape up everything they can, oftentimes ignoring robots.txt or any rate limits (there have been a few complaint threads on HN about that), I can hardly blame the operators of large online services just cutting off data feeds.Twitter however didn&#x27;t stop their data feeds due to AI or because they wanted money, they stopped providing them because its new owner does everything he can to hinder researchers specializing in propaganda campaigns or public scrutiny.

&gt;&quot;Now Twitter is gone anyway, its public APIs have shut down, and the site has been replaced with an oligarch&#x27;s plaything, a spam-infested right-wing cesspool called X. Even if X made its raw data feed available (which it doesn&#x27;t), there would be no valuable information to be found there.&gt;Reddit also stopped providing public data archives, and now they sell their archives at a price that only OpenAI will pay.&gt;And given what&#x27;s happening to the field, I don&#x27;t blame them.&quot;What beautiful doublethink.

Hah! I&#x27;m trying to figure out the exact date that crossed from &quot;plausible line from a Stross or Sterling novel&quot; [1] to &quot;of course they did&quot;.[1] Or maybe Sheckley or Lem, now that I think about it.

I read that as 3 books per year at first and thought to myself that that was a rather harsh limitation but surely any true respectable author wouldn&#x27;t be spitting more than that......and then I realized you wrote 3 books a day. What the hell.

It&#x27;s okay. Amazon has limited authors to self publishing only 3 books per day (yes, really). That will surely solve the problem.

&gt; A good 70% of my Facebook feed is now obviously AI generated images with AI generated text blurbs that have nothing to do with the accompanying images likely posted by overseas bot farms.This is a self-inflicted problem, IMO.Do you just have shitty friends that share all that crap? Or are you following shitty pages?I use Facebook a decent amount, and I don&#x27;t suffer from what you&#x27;re complaining about. Your feed is made of what you make it. Unfollow the pages that make that crap. If you have friends that share it, consider unfriending or at the very least, unfollowing. Or just block the specific pages they&#x27;re sharing posts from.

Enshittification is accelerating.  A good 70% of my Facebook feed is now obviously AI generated images with AI generated text blurbs that have nothing to do with the accompanying images likely posted by overseas bot farms.  I&#x27;m also noticing more and more &quot;books&quot; on Amazon that are clearly AI generated and self published.

&gt; Did we (the humans) somehow managed to pollute the internet so much with AI that&#x27;s it&#x27;s now barely usableIf we haven&#x27;t already, we will be very soon. I&#x27;m sure there are people working on this problem, but I think we&#x27;re starting to hit a very imminent feedback loop moment. Most of human&#x27;s recorded information is digitized and most of that is generating non-human content at an incredible pace. We&#x27;ve injected a whole lot of noise into our usable data.I don&#x27;t know if the answer is more human content (I&#x27;m doing my part!) or novel generative content but this interim period is going to cause some medium-term challenges.I like to think the LLM more-tokens-equals-better era is fading and we&#x27;re getting into better use of existing data, but there&#x27;s a very real inflection point we&#x27;re facing.

That&#x27;s a nice analogy. Fortunately (un)real estate is easier to manufacture out of thin air online. We have lost some valuable spaces like Twitter and Reddit to some degree though.

Yes. Here are practical instructions on how to turn it into an even more of a cesspit <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=endHz0jo9Ck" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=endHz0jo9Ck</a> I think it&#x27;s now a law of nature that any new tech leads to SEO amplification. AI has become the Degelman M34 Manure Spreader of the internet <a href="https:&#x2F;&#x2F;degelman.com&#x2F;products&#x2F;manure-spreaders" rel="nofollow">https:&#x2F;&#x2F;degelman.com&#x2F;products&#x2F;manure-spreaders</a>

I&#x27;ve seen plenty of comments here that read like they&#x27;ve been generated by an LLM, if this is a gated community we need a better gate.

Also our collective unwillingness to pay for subscriptions for publications

There are smaller, gated communities that are still very valuable. You&#x27;re posting in one. But yes, the open Internet is basically useless now, thanks ultimately to advertising as a business model.

Tragedy of the Commons Ruins Everything Around Me

&gt;We the humansNice tryIf it’s not clear, I’m joking.

&gt; Did we (the humans) somehow managed to pollute the internetCorporations did that, not humans.&quot;few people recognize that we already share our world with artificial creatures that participate as intelligent agents in our society: corporations&quot;
- <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1204.4116" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1204.4116</a>

The public Internet has been relentlessly strip-mined for profit by ever since Canter &amp; Siegel posted their immigration services ad to every single Usenet newsgroup.

Did we (the humans) somehow managed to pollute the internet so much with AI that&#x27;s it&#x27;s now barely usable ?In my opinion the internet can be considered as the equivalent of a natural environment like the earth. it&#x27;s a space where people share, meet, talk, etc.I find it astonishing that after polluting our natural environment we know polluted the internet.

&gt;our AIs may not be very good at learning but our brains areBrains aren&#x27;t nearly as good at slightly adjusting the statistical properties of a text corpus as computers are.

You&#x27;re exactly right. You only have to look at the prevalence of the word &quot;unalive&quot; in real life contexts to find an example.

&gt; The more we use AI, the more we integrate LLMs and other tools into our life, the more their output will influence usHmm I don’t disagree but I think it will be valuable skill going forward to write text that doesn’t read like it was written by an LLMThis is an arms race that I’m not sure we can win though. It’s almost like a GAN.

It could be used to differentiate LLM text from pre-LLM human text maybe. The thing, our AIs may not be very good at learning but our brains are. The more we use AI, the more we integrate LLMs and other tools into our life, the more their output will influence us. I believe there was a study (or a few anecdotes) where college papers checked for AI material were marked AI written even though they were written by humans because the students used AI during their studying and learned from it.

&gt; ... compare the frequency of words to those used in human natural writings and you spot the computer from the human.But that&#x27;s a losing endeavor: if you can do that, you can immediately ask your LLM to fix its output so that it passes that test (and many others). It can introduce typos, make small errors on purpose, and anything you can think of to make it look human.

it may work for a short time, but after a while natural language will evolve due to natural exposure of those new words or word patterns and even human will write in ways that, while being different from the LLMs, will also be different from the snapshot captured by this snapshot. It&#x27;s already the case that we used to write differently 20 years ago from 50 years ago and even more so 100 years ago, etc

Hardly. You are talking about a statistical test, which will have rather large errors (since it is based on word frequencies). Not to mention word frequencies will vary depending on the type of text (essay, description, advertisement, etc).

It could be used to spot LLM generated text.compare the frequency of words to those used in human natural writings and you spot the computer from the human.

Why is this a HN comment on a thread about it ending due to AI pollution?

&quot;Multi-script languagesTwo of the languages we support, Serbian and Chinese, are written in multiple scripts. To avoid spurious differences in word frequencies, we automatically transliterate the characters in these languages when looking up their words.Serbian text written in Cyrillic letters is automatically converted to Latin letters, using standard Serbian transliteration, when the requested language is sr or sh.&quot;I&#x27;d support keeping both scripts (српска ћирилица and latin script) , similarly to hiragana (ひらがな) and katakana (カタカナ) in Japanese.

To support well-known “philanthropists” like Sam Altman or Mark Zuckerberg that many consider as their heroes here.

Yes. I&#x27;ve always loved my books but now consider them my most valuable possessions.

And I thought I had some kind of mental illness collecting all those books, barely reading them. Need to do that more now.

Or AI talking heads posing as the author of books written by AIs. <a href="https:&#x2F;&#x2F;youtu.be&#x2F;pAPGRGTqIgI" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;pAPGRGTqIgI</a> (warning: state sponsored disinformation AI)

Somehow related, paper books from before 2020 could be a valuable commodity in a in a decade or two, when the Internet will be full of slop and even contemporary paper books will be treated with suspicion. And there will be human talking heads posing as the authors of books written by very smart AIs. God, why are we doing this????

I find issue with this statement as content was never a clean representation of human actions or even thought. It was always driven by editorials, SEO, bot remixing and whatnot that heavily influences how we produce content. One might even argue that heightened content distrust is _good_ for our society.

It&#x27;s worse because many humans don&#x27;t know they are.I see a lot of outrage around fake posts already. People want to believe bad things from the other tribes.And we are going to feed them with it, endlessly.

It&#x27;s worse: they don&#x27;t even care.

This video&#x27;s worth a watch if you want to get a sense of the current state of things. Despite the (deliberately) clickbait title, the video itself is pretty even-handed.It&#x27;s by Language Jones, a YouTube linguist. Title: &quot;The AI Apocalypse is Here&quot;<a href="https:&#x2F;&#x2F;youtu.be&#x2F;XeQ-y5QFdB4" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;XeQ-y5QFdB4</a>

It&#x27;s even worse than that. Most people have no idea how far CGI has come, and how easily it is wielded even by a couple of dedicated teens on their home computer, let alone people with a vested interest in faking something for some financial reason. People think they know what a &quot;special effect&quot; looks like, and for the most part, people are wrong. They know what CGI being used to create something obviously impossible, like a dinosaur stomping through a city, looks like. They have no idea how easy a lot of stuff is to fake already. AI just adds to what is already there. Heck, to some extent it has caused scammers to overreach, with things like obviously fake Elon Musk videos on YouTube generated from (pure) AI and text-to-speech... when with just a little bit more learning, practice, and amounts of equipment completely reasonable for one person to obtain, they could have done a much better fake of Elon Musk using special effects techniques rather than shoveling text into an AI. The fact that &quot;shoveling text into an AI&quot; may in another few years itself generate immaculate videos is more a bonus than a fundamental change of capability.Even what&#x27;s free &amp; open source in the special effects community is astonishing lately.

I mean, it&#x27;s already apparent to me that a lot of people don&#x27;t have a basic process in place to detect fact from fiction. And it&#x27;s definitely not always easy, but when I hear some of the dumbest conspiracy theories known to man actually get traction in our media, political figures, and society at large, I just have to shake my head and laugh to keep from crying. I&#x27;m constantly reminded of my favorite saying, &quot;people who believe in conspiracy theories have never been a project manager.&quot;

Oh they definitely are. A lot of people are now calling out real photos as fake. I frequently get into stupid Instagram political arguments and a lot of times they come back with &quot;yeah nice profile with all your AI art haha&quot;. It&#x27;s all real high quality photography. Honestly, I don&#x27;t think the avg person can tell anymore.

&gt;Humans losing their ability to detect AI content from reality ? It&#x27;s frightening.And it already happened, and no one pushed back while it was happening.

I never thought about that. Humans losing their ability to detect AI content from reality ? It&#x27;s frightening.

+100w chargers are one of the products I prefer to spend a little more on, so I get something from a company that knows it can be sued if they make a product that burns down your house or fries your phone.Flashlights? Sure, bring on aliexpress. USB cables with pop-off magnetically attached heads, no problem. But power supplies? Welp, to each their own!

&gt; One revolution I&#x27;m still coming to grips with is automated manufacturing. Going on aliexpress, so much stuff is basically free. I bought a 5-port 120W (total) charger for less than 2 minutes of my time. It literally took less time to find it than to earn the money to buy it.Is there a big recent qualitative change here? Or is this a continuation of manufacturing trends (also shocking, not trying to minimize it all, just curious if there’s some new manufacturing tech I wasn’t aware of).For some reason, your comment got me thinking of a fully automated system, like: you go to a website, pick and choose charger capabilities (ports, does it have a battery, that sort of stuff). Then an automated factor makes you a bespoke device (software picks an appropriate shell, regulators, etc). I bet we’ll see it in our lifetimes at least.

&gt; so much stuff is basically freeIt really isn&#x27;t. Have a look at daily median income statistics for the rest of the planet:<a href="https:&#x2F;&#x2F;ourworldindata.org&#x2F;grapher&#x2F;daily-median-income?tab=table">https:&#x2F;&#x2F;ourworldindata.org&#x2F;grapher&#x2F;daily-median-income?tab=t...</a><pre><code> $2.48 Eastern and Southern Africa (PIP)
 $2.78 Sub-Saharan Africa (PIP)
 $3.22 Western and Central Africa (PIP)
 $3.72 India (rural)
 $4.22 South Asia (PIP)
 $4.60 India (urban)
 $5.40 Indonesia (rural)
 $6.54 Indonesia (urban)
 $7.50 Middle East and North Africa (PIP)
 $8.05 China (rural)
 $10.00 East Asia and Pacific (PIP)
 $11.60 Latin America and the Caribbean (PIP)
 $12.52 China (urban)
</code></pre>
And more generally:<pre><code> $7.75 World
</code></pre>
I looked around on Ali, and the cheapest charger that doesn&#x27;t look too dangerous costs around five bucks. So it&#x27;s roughly equal to one day&#x27;s income of at least half the population of our planet.

&gt; &quot;The revolutions shorten in time, seemingly exponentially.&quot;The Technological Singularity - <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Technological_singularity" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Technological_singularity</a>

Democracy is 200? You&#x27;re off by a full order of magnitude.Progress isn&#x27;t inevitable. It&#x27;s possible for knowledge to be lost and for civilization to regress.

Democracy (and Republics) are thousands of year old. Computation is also quite old though it only sky-rocketed with electricity and semiconductors. This is not the first time the global world created a potential for exponential growth (I&#x27;ll consider the Pharaohs and Roman empires to be ones).There is the very real possibility that everything just stalls and plateau where we are at. You know, like our population growth, it should have gone exponentially but it did not. Actually, quite the reverse.

There are a series of challenges like:<a href="https:&#x2F;&#x2F;www.nytimes.com&#x2F;interactive&#x2F;2024&#x2F;09&#x2F;09&#x2F;technology&#x2F;ai-video-deepfake-runway-kling-quiz.html" rel="nofollow">https:&#x2F;&#x2F;www.nytimes.com&#x2F;interactive&#x2F;2024&#x2F;09&#x2F;09&#x2F;technology&#x2F;ai...</a><a href="https:&#x2F;&#x2F;www.nytimes.com&#x2F;interactive&#x2F;2024&#x2F;01&#x2F;19&#x2F;technology&#x2F;artificial-intelligence-image-generators-faces-quiz.html" rel="nofollow">https:&#x2F;&#x2F;www.nytimes.com&#x2F;interactive&#x2F;2024&#x2F;01&#x2F;19&#x2F;technology&#x2F;ar...</a>These are a little bit unfair, in that we&#x27;re comparing handpicked examples, but I don&#x27;t think many experts will pass a test like this. Technology only moves forward (and seemingly, at an accelerating pace).What&#x27;s a little shocking to me is the speed of progress. Humanity is almost 3 million years old. Homosapiens are around 300,000 years old. Cities, agriculture, and civilization is around 10,000. Metal is around 4000. Industrial revolution is 500. Democracy? 200. Computation? 50-100.The revolutions shorten in time, seemingly exponentially.Comparing the world of today to that of my childhood....One revolution I&#x27;m still coming to grips with is automated manufacturing. Going on aliexpress, so much stuff is basically free. I bought a 5-port 120W (total) charger for less than 2 minutes of my time. It literally took less time to find it than to earn the money to buy it.I&#x27;m not quite sure where this is all headed.

&gt; When I get asked if the person in a video is real, I still feel pretty confident to answerI don&#x27;t share your confidence in identifying real people anymore.I often flag as &quot;false-ish&quot; a lot of things from genuinely real people, but who have adopted the behaviors of the TikTok&#x2F;Insta&#x2F;YouTube creator. Hell, my beard is grey and even I poked fun at &quot;YouTube Thumbnail Face&quot; back in 2020 in a video talk I gave. AI twigs into these &quot;semi-human&quot; behavioral patterns super fast and super hard.There is a video floating around with pairs of young ladies with &quot;This is real&quot;&#x2F;&quot;This is not real&quot; on signs. They could be completely lying about both, and I really can&#x27;t tell the difference. All of them have behavioral patterns that seems a little &quot;off&quot; but are consistent with the small number of &quot;influencer&quot; videos I have exposure to.

That is very true, but for now we have a baseline of videos that we either remember or that we remember key details of, like the persons in the video. I&#x27;m pretty sure if I watch The Primeagen or Tom Scott today, that they are real. Ask me in year, I might not be so sure anymore.

&gt; When I get asked if the person in a video is real, I still feel pretty confident to answerI don&#x27;t. I mean, I can identify the bad ones, sure, but how do I know I&#x27;m not getting fooled by the good ones?

&quot;I don&#x27;t think anyone has reliable information about post-2021 language usage by humans.&quot;We&#x27;ve been past the tipping point when it comes to text for some time, but for video I feel we are living through the watershed moment right now.Especially smaller children don&#x27;t have a good intuition on what is real and what is not. When I get asked if the person in a video is real, I still feel pretty confident to answer but I get less and less confident every day.The technology is certainly there, but the majority of video content is still not affected by it. I expect this to change very soon.

&gt; every minute, it&#x27;s harder to distinguish AI output from actual output unless you&#x27;re approaching expertise in the subject being written about.So, then what really is the problem with just including LLM-generated text in wordfreq?If quirky word distributions will remain a &quot;problem&quot;, then I&#x27;d bet that human distributions for those words will follow shortly after (people are very quick to change their speech based on their environment, it&#x27;s why language can change so quickly).Why not just own the fact that LLMs are going to be affecting our speech?

&gt; I hear this complaint often but in reality I have encountered fairly little content in my day to day that has felt fully AI generated?How confident are you in this assessment?&gt; straight drivelWe&#x27;re past the point where what AI generates is &quot;straight drivel&quot;; every minute, it&#x27;s harder to distinguish AI output from actual output unless you&#x27;re approaching expertise in the subject being written about.&gt; a team of copywriters who pumped out mountains the most inane keyword packed text designed for literally no one but Google to read.And now a machine can generate the same amount of output in 30 seconds. Scale matters.

I hear this complaint often but in reality I have encountered fairly little content in my day to day that has felt fully AI generated? AI assisted sure, but is that a problem if a human is in the mix, curating?I certainly have not encountered enough straight drivel where I would think it would have a significant effect on overall word statistics.I suspect there may be some over-identification of AI content happening, a sort of Baader–Meinhof effect cognitive bias. People have their eye out for it and suddenly everything that reads a little weird logically &quot;must be AI generated&quot; and isn&#x27;t just a bad human writer.Maybe I am biased, about a decade ago I worked for an SEO company with a team of copywriters who pumped out mountains the most inane keyword packed text designed for literally no one but Google to read. It would rot your brain if you tried, and it was written by hand by a team of humans beings. This existed WELL before generative AI.

Most of the time, HN is that quiet corner. I just hope it stays that way.

&gt; At some point all of this will become so awful that nerds will create new and quiet corners of real people and real informationIt&#x27;s already happening. There is a growing number of groups that are forming their own &quot;private internets&quot; that is separated from the internet-at-large, precisely because the internet at large is becoming increasingly useless for a whole lot of valuable things.

I really like the fact that the content of the conventional user content internet is becoming willfully polluted and ever more useless by the incessant influx of &quot;ai&quot;-garbage. At some point all of this will become so awful that nerds will create new and quiet corners of real people and real information while the idiot rabble has to use new and expensive tools peddled by scammy tech bros to handle the stench of automated manure that flows out of stagnant llms digesting themselves.

Yes, but the material presented in no way makes distiction between potential organic growth of &#x27;delve&#x27; vs. LLM induced use. They just note that even though &#x27;delve&#x27; was on the rise, in 23-24 the word gains more popularity, at the same time ChatGPT rose. Word adoption is certainly not a linear phenomenon. And as the author states &#x27;I don&#x27;t think anyone has reliable information about post-2021 language usage by humans&#x27;So I would still state noun-phrase frequency in LLM output would tend to reflect noun-phrase frequency in training data in a similar context (disregarding enforced bias induced through RLHF and other tuning at the moment)I&#x27;m sure there will be cross-fertilization from LLM to Human and back, but I&#x27;m not seeing the data yet that the influence on word-frequency is that outspoken.The author seems to have some other objections to the rise of LLM&#x27;s, which I fully understand.

there is almost certainly organic growth as well as more people in Nigeria and other SSA countries are getting very good internet penetration in recent years

‘delve’ is given as an example right there in TFA.

It&#x27;d be in fact quite the opposite. There comes a turning point where the majority of language usage would actually be written by AI, at which point we&#x27;d no longer be analysing the word frequency&#x2F;usage by actual humans and so it wouldn&#x27;t be representative of how humans actually communicate.Or potentially even more dystopian would be that AI slop would be dictating&#x2F;driving human communication going forward.

Yes but this person learned to mimic the internet at large. Theoretically its preferred way of talking would be the average of all training data, as mimicry is GPT&#x27;s training objective, and would therefore have very similar word distributions. Only, this doesn&#x27;t account for RLHF and prompts spreading memetically among users.

Think of an LLM as a person on the internet. Just like everyone else, they have their own vocabulary and preferred way of talking which means they’ll use some words more than others. Now imagine we duplicate this hypothetical person an incredible amount of times and have their clones chatter on the internet frequently. ‘Certainly’ this would have an effect.

If only we had a data set that measured word frequency across the internet as we&#x27;re getting more and more into AI being used... Maybe with a baseline from before 2021 for comparison... But no let&#x27;s just stop measuring word frequency entirely because we can just assume what will happen and we&#x27;re angry.

Intuitively I feel like word frequency would be one of the things least impacted by LLM output, no?

If the language you’re processing was generated by AI, it’s no longer NLP, it’s ALP.

&gt; As soon as generative AI has the capability to take someone&#x27;s job, that is political.What is political is people enshrining themselves in chokepoints and demanding a toll for passing through or getting anything done. That is what you do when you make a certain job politically &#x27;untakable&#x27;.People who espouse that the &#x27;personal is political&#x27; risk making the definition of politics so broad that it is useless.

Generative AI is inherently a political issue, its not surprising at all.There is the case of what is &quot;truth&quot;. As soon as you start to ensure some quality of truth to what is generated, that is political.As soon as generative AI has the capability to take someone&#x27;s job, that is political.The instant AI can make someone money, it is political.When AI is trained on something that someone has created, and now they can generate something similar, it is political.

The simplest example that comes to mind of something frequency analysis might be useful for would be if you had simple ciphertext where you knew that the characters probably 1:1 mapped, but you didn&#x27;t know anything about how.It could also be useful for guessing whether someone might have been trying to do some kind of steganographic or additional encoding in their work, by telling you how abnormal compared to how many people write it is that someone happened to choose a very unusual construction in their work, or whether it&#x27;s unlikely that two people chose the same unusual construction by coincidence or plagiarism.You might also find statistical models interesting for things like noticing patterns in people for whom English or others are not their first language, and when they choose different constructions more often than speakers for whom it was their first language.I&#x27;m not saying you can&#x27;t use an LLM to do some or all of these, but they also have something of a scalar attached to them of how unusual the conclusion is - e.g. &quot;I have never seen this construction of words in 50 million lines of text&quot; versus &quot;Yes, that&#x27;s natural.&quot;, which can be useful for trying to inform how close to the noise floor the answer is, even ignoring the prospect of hallucinations.

Yes, it&#x27;s become extremely politicized and its very tiresome.  Tech in general, to be frank. Pray that your field of interest never gets covered in the NYT.

Reading through this entire thread, I suspect that somehow generative AI actually became a political issue. Polarized politics is like a vortex sucking all kinds of unrelated things in.In case that doesn&#x27;t get my comment completely buried, I will go ahead and say honestly that even though &quot;AI slop&quot; and paywalled content is a problem, I don&#x27;t think that generative AI in itself is a negative at all. And I also think that part of this person&#x27;s reaction is that LLMs have made previous NLP techniques, such a those based on simple usage counts etc., largely irrelevant.What was&#x2F;is wordfreq used for, and can those tasks not actually be done more effectively with a cutting edge language model of some sort these days? Maybe even a really small one for some things.

I feel like you&#x27;re giving certain entities too much credit there. Yes text is generated to do _something_, but it may not be to communicate in good-faith; it could be keyword-dense gibberish designed to attract unsuspecting search engine users for click revenue, or generate political misinformation disseminated to a network of independent-looking &quot;news&quot; websites, or pump certain areas with so much noise and nonsense information that those spaces cannot sustain any kind of meaningful human conversation.The issue with generative &#x27;AI&#x27; isn&#x27;t that they generate text, it&#x27;s that they can (and are) used to generate high-volume low-cost nonsense at a scale no human could ever achieve without them.&gt; Life’s too short to go through it hating othersOnly when they don&#x27;t deserve it. I have my doubts about Google, but I&#x27;ve no love for OpenAI.&gt; Plagiarism has a particular definition ... no-one is required to append a statement crediting every text he has ever readOf course they aren&#x27;t, because we rightly treat humans learning to communicate differently from training computer code to predict words in a sentence and pass it off as natural language with intent behind it. Musicians usually pay royalties to those whose songs they sample, but authors don&#x27;t pay royalties to other authors whose work inspired them to construct their own stories maybe using similar concepts. There&#x27;s a line there somewhere; falsely equating plagiarism and inspiration (or natural language learning in humans) misses the point.

This is just the &quot;guns don&#x27;t shoot people, people do.&quot; argument except in this case we quite literally have a massive upside incentive to remove people from the process entirely (i.e. websites that automatically generate new content every day) - so I don&#x27;t buy it.This kind of AI slop is quite literally written by no one (an algorithm pushed it out), and it doesn&#x27;t communicate anything since communication first requires some level of understanding of the source material - and LLM&#x27;s are just predicting the likely next token without understanding. I would also extend this to AI slop written by someone with a limited domain understanding, they themselves have nothing new to offer, nor the expertise or experience to ensure the AI is producing valuable content.I would go even further and say it&#x27;s &quot;read by no one&quot; - people are sick and tired of reading the next AI slop article on google and add stuff like &quot;reddit&quot; to the end of their queries to limit the amount of garbage they get.Sure there are people using LLMs to enhance their research, but a vast, vast majority are using it to create slop that hits a word limit.

&gt; It is not at all clear that a machine learning from text should be treated any differently from a human being learning from textGiven that LLMs and human creativity work on fundamentally different principles, there is every reason to believe there is a difference.

&gt; the Web at large is full of slop generated by large language models, written by no one to communicate nothingThat’s neither fair nor accurate. That slop is ultimately generated by the humans who run those models; they are attempting (perhaps poorly) to communicate something.&gt; two companies that I already despiseLife’s too short to go through it hating others.&gt; it&#x27;s very likely because they are creating a plagiarism machine that will claim your words as its ownThat begs the question. Plagiarism has a particular definition. It is not at all clear that a machine learning from text should be treated any differently from a human being learning from text: i.e., duplicating exact phrases or failing to credit ideas may in some circumstances be plagiarism, but no-one is required to append a statement crediting every text he has ever read to every document he ever writes.Credits: every document I have ever read grin

That sentence stood out to me too, very powerful. Felt it right in the feels.

&gt; Now the Web at large is full of slop generated by large language models, written by no one to communicate nothing.Fair and accurate. In the best cases the person running the model didn&#x27;t write this stuff and word salad doesn&#x27;t communicate whatever they meant to say. In many cases though, content is simply pumped out for SEO with no intention of being valuable to anyone.

&gt; curate a whitelist of sources that don&#x27;t use AI,I like this.Maybe even take it a step further - have a badge on the source that is both human and machine visible to indicate that the content is not AI generated.

I guess a manageable, still-useful alternative would be to curate a whitelist of sources that don&#x27;t use AI, and without making that list public, derive the word frequencies from only those sources. How to compile that list is left as an exercise for the reader. The result would not be as accurate as a broad sample of the web, but in a world where it&#x27;s impossible to trust a broad sample of the web, it the option you are left with. And I have no reason to doubt that it could be done at a useful scale.I&#x27;m sure this has occurred to them already. Apart from the near-impossibility of continuing the task in the same way they&#x27;ve always done it, it seems like the other reason they&#x27;re not updating wordfreq is to stick a thumb in the eye of OpenAI and Google. While I appreciate the sentiment, I recognize that those corporations&#x27; eyes will never be sufficiently thumbed to satisfy anybody, so I would not let that anger change the course of my life&#x27;s work, personally.

I fail to see the difference. Actually, programming was one of the first field where LLMs shown proficiency. The helper nature of LLMs is true in all the fields so far, in the future this may change. I believe that for instance in the case or journalism the issue was already there: three euros per post written without clue by humans.Anyway in the long run AI will kill tons of jobs. Regardless of blog posts like that. The true key is governments assistance.

I think programmers are in the perfect profession to call LLMs out for just how bad they are. They are fancy auto-complete and I love them in my daily usage, but a big part of that is because I can tell when they are ridiculously wrong. Which is so often you really have to question how useful they would be for anything where they aren’t just fancy auto-complete.Which isn’t AIs fault. I’m sure they can be great in cancer detection, unless they replace what we’re already doing because they are cheaper than doctors. In combination with an expert AI is great, but that’s not what’s happening is it?

This is a tech site, where &gt;50% of us are programmers who have achieved greater productivity thanks to LLM advances.And yet we&#x27;re filled to the gills with Luddite sentiments and AI content fearmongering.Imagine the hysteria and the skull-vibrating noise of the non-HN rabble when they come to understand where all of this is going. They&#x27;re going to do their darndest to stop us from achieving post-economy.

Ok so post author is AI skeptic and this is his retaliation, likely because his work is affected. I believe governments should address the problem with welfare but being against technical advances is always being in the wrong side of history.

I assume you are correct but how can we know rather than assume? I am not sure we can, so why get worked up about &quot;internet died in 2021&quot; when many would claim with similar conviction that it&#x27;s been dead since 2012, or 2007, or ...

&gt;AI didn&#x27;t just occur in 2021. Nobody knows how much text was machine generated prior to 2021But we do know that now it&#x27;s a lot more, with a big LOT.

I think this person has too high a view of pre-2021, probably for ego reasons. In fact, their attitude seems very ego driven. AI didn&#x27;t just occur in 2021. Nobody knows how much text was machine generated prior to 2021, it was much harder if not impossible to detect. If anything, it&#x27;s probably easier now since people are all using the same ai that use words like delve so much much it becomes obvious.

yes, unfortunately <a href="https:&#x2F;&#x2F;www.latent.space&#x2F;p&#x2F;nov-2023" rel="nofollow">https:&#x2F;&#x2F;www.latent.space&#x2F;p&#x2F;nov-2023</a>

Is 2023 going to be for data what the trinity test was for iron? Eg post 2023 all data now contains trace amounts of ai?

&gt; when did we stop being hopeful and start seeing the darkest side of everything?I think a decade or two ago, when most of the new tech being introduced (at least by our industry) started being unmistakably abusive and dehumanizing. When the recent past shows a strong trend, it&#x27;s not unreasonable to expect the the near future will continue that trend. Particularly when it makes companies money.

Give us examples of generative AI in challenging applications (biology, medicine, physical sciences), and you&#x27;ll get a lot of optimism. The text LLM stuff is the brute force application of the same class of statistical modeling. It&#x27;s commercial, and boring.

Aaron Swartz would have loved open source GenAI models.

When?For some of us, it was 1994, the eternal September.For some of us, it was when Aaron Swartz left us.For some of us, it was when Google killed Google Reader (in hindsight, the turning point of Google becoming evil).For some others, like the author of this post, it&#x27;s when twitter and reddit closed their previously open APIs.

Wow there is so much vitriol both in this post and in the comments here. I understand that there are many ethical and practical problems with generative AI, but when did we stop being hopeful and start seeing the darkest side of everything? Is it just that the average HN reader is now past the age where a new technological development is an exciting opportunity and on to the age where it is a threat? Remember, the Luddites were not opposed to looms, they just wanted to own them.

&gt; Have &quot;good&quot; small webs EVER prevailed?You ask on HN, one of the highest quality sites I&#x27;ve ever visited in any age of the Internet.IRC is still alive and well among pretty much the same audience as always. I&#x27;m not sure it&#x27;s fair to compare that with the others.

its so easy to solve this problem, not sure why anyone hasnt done it yet.1. build a userbase, free product2. once userbase get big enough, any new account requires a monthly fee, maybe $13. keep raising the fee higher and higher, until you get to the point that the userbase is manageable.no ads, simple.

Any curation mechanism that depends on passion and&#x2F;or the goodwill of volunteers is unsustainable.

Have &quot;good&quot; small webs EVER prevailed?Every content system seems to get polluted by noise once it hits mainstream usage: IRC, Usenet, reddit, Facebook, geocities, Yahoo, webrings, etc. Once-small curated selections eventually grow big enough to become victims of their own successes and taken over by spam.It&#x27;s always an arms race of quality vs quantity, and eventually the curators can&#x27;t keep up with the sheer volume anymore.

I don&#x27;t know, individually finely tuned addictive content served as real time interactive feedback loops is an other level of propaganda and attention capture tool than largest common denominator of the general crowd served as static passive content.

&gt; people consuming auto-generated content to keep the masses from away from critical thinking. This is now happeningThe people who stay away from critical thinking were doing that already and will continue to do so, &#x27;AI&#x27; content or not.

tangentially related, but Marx also predicted that crypto and NFT&#x27;s would exist in 1894 [1] and I only bring it up because its kind of wild how we keep crossing these &quot;red lines&quot; without even blinking. It&#x27;s like that meme:Sci-fi author:I created the Torment Nexus to serve as a cautionary tale...Tech Company:Alas, we have created the Torment Nexus from the classic Sci-fi novel &quot;Don&#x27;t Create the Torment Nexus&quot;1. <a href="https:&#x2F;&#x2F;www.marxists.org&#x2F;archive&#x2F;marx&#x2F;works&#x2F;1894-c3&#x2F;ch25.htm" rel="nofollow">https:&#x2F;&#x2F;www.marxists.org&#x2F;archive&#x2F;marx&#x2F;works&#x2F;1894-c3&#x2F;ch25.htm</a>

&gt; Good will prevail in the end.Even if, this is a dangerous thought that discourages decisive action that is likely to be necessary for this to happen.

What if the way for good to prevail is to reject technologies and beliefs that have become destructive?

I regret the situation led to the OP feel discourage about the NLP community, wo which I belong, and I just want to say &quot;we&#x27;re not all like that&quot;, even though it is a trend and we&#x27;re close to peak hype (slightly past even?).The complaint about pollution of the Web with artificial content is timely, and it&#x27;s not even the first time due to spam farms intended to game PageRank, among other nonsense. This may just mean there is new value in hand-curated lists of high-quality Web sites (some people use the term &quot;small Web&quot;).Each generation of the Web needs techniques to overcome its particular generation of adversarial mechanisms, and the current Web stage is no exception.When Eric Arthur Blair wrote 1984 (under his pen name &quot;George Orwell&quot;), he anticipated people consuming auto-generated content to keep the masses from away from critical thinking. This is now happening (he even anticipated auto-generated porn in the novel), but the technologies criticized can also be used for good, and that is what I try to do in my NLP research team. Good will prevail in the end.

Most of the &quot;random&quot; bot content pre-2021 was low-quality Markov-generated text. If anything, these genitive AI tools would improve the accuracy of scraping large corpora of text from the web.

Delve was also an addition to PoE, which I imagine had its own spike in google searches relative to that word.

Delves are a new thing in World of Warcraft released 9&#x2F;10 this year. Delve is also an M365 product that has been around for some time and is being discontinued in December. So no, that has nothing to do with LLMs.

Exactly, like how &quot;mindful&quot; and &quot;demure&quot; recently became more popular for seemingly no reason. Humans do this all the time.And language in general stagnates and shrinks in vocabulary over time ( <a href="https:&#x2F;&#x2F;evoandproud.blogspot.com&#x2F;2019&#x2F;09&#x2F;why-is-vocabulary-shrinking.html?m=1" rel="nofollow">https:&#x2F;&#x2F;evoandproud.blogspot.com&#x2F;2019&#x2F;09&#x2F;why-is-vocabulary-s...</a> ). (Link that ChatGPT helped me find :P) I think AI will increase the average persons vocabulary, since it appears to in general be better&#x2F;more professionally written than a lot of what the average person is exposed to online.

We’ve seen this with a couple words and expressions, and I don’t doubt that AI is somewhat likely to “like” some phrases for whatever reason. Big eigenvaues of the latent space or whatever, hahaha (I don’t know AI).But also, words and phrases do become popular among humans, right? It would be a shame if AI caused the language to get more stagnant, as keeping up with which phrases are popular get you labeled as an AI.

If your select only USA, the trend disappears.

One of the examples is the increased usage of &quot;delve&quot; which Google Trends confirms increased in usage since 2022 (initial ChatGPT release): <a href="https:&#x2F;&#x2F;trends.google.com&#x2F;trends&#x2F;explore?date=all&amp;q=delve&amp;hl=en" rel="nofollow">https:&#x2F;&#x2F;trends.google.com&#x2F;trends&#x2F;explore?date=all&amp;q=delve&amp;hl...</a>It seems however it started increasing most in usage just these last few months, maybe people are talking more about &quot;delve&quot; specifically because of the increase in usage? A usage recursion of some sorts.

I think that text on the internet will tainted by AI the same way that steel has being tainted by nuclear devices.

If generative AI has a significantly different word frequency from humans then it also shouldn&#x27;t be hard to detect text written generative AI. However my last information is that tools to detect text written by generative AI are not that great.

The unspoken question here, of course, is &quot;you wouldn&#x27;t happen to have already done this for me?&quot; ;)

Has anyone taken a look at a random sample of web data? It&#x27;s mostly crap. I was thinking of making my own search engine, knowledge database etc based on a random sample of web pages, but I found that almost all of them were drivel. Flame wars, asinine blog posts, and most of all, advertising. Forget spam, most of the legit pages are trying to sell something too!The conclusion I arrived at was that making my own crawler actually is feasible (and given my goals, necessary!) because I&#x27;m only interested in a very, very small fraction of what&#x27;s out there.

Back in the day they did. I checked handbrake but now there&#x27;s nothing specific about DVD compatibility there. I could have picked something like Super HQ 576p, and there&#x27;s a good chance that would have sufficed, but old DVD players were extremely finicky about filenames, extensions, interlacing, etc. I didn&#x27;t want to risk the DVD traveling half way across the world only to find that it&#x27;s not playable.

&gt; Back in pre-LLM days, it&#x27;s not like I would have hired a x264 expert to do this job for me. I would have either had to spend hours more on this task, or more likely, this 97 year old man would never have seen his great granddaughter&#x27;s danceDidn&#x27;t most DVD burning software include video transcoding as a standard feature? Back in the day, you&#x27;d have used Nero Burning ROM, or Handbrake - granted, the quality may not have been optimized to your standards, but the result would have been a watchable video (especially to 97 year-old eyes)

I feel so conflicted about this.On the one hand, I completely agree with Robyn Speer. The open web is dead, and the web is in a really sad state. The other day I decided to publish my personal blog on gopher. Just cause, there&#x27;s a lot less crap on gopher (and no, gopher is not the answer).But...A couple of weeks ago, I had to send a video file to my wife&#x27;s grandfather, who is 97, lives in another country, and doesn&#x27;t use computers or mobile phones. Eventually we determined that he has a DVD player, so I turned to x264 to convert this modern 4K HDR video into a form that can be played by any ancient DVD player, while preserving as much visual fidelity as possible.The thing about x264 is, it doesn&#x27;t have any docs. Unlike x265 which had a corporate sponsor who could spend money on writing proper docs, x264 was basically developed through trial and error by members of the doom9 forum. There are hundreds of obscure flags, some of which now operate differently to what they did 20 years ago. I could spend hours going through dozens of 20 year old threads on doom9 to figure out what each flag did, or I could do what I did and ask a LLM (in this case Claude).Claude wasn&#x27;t perfect. It mixed up a few ffmpeg flags with x264 ones (easy mistake), but combined with some old fashioned searching and some trial and error, I could get the job done in about half an hour. I was quite happy with the quality of the end product, and the video did play on that very old DVD player.Back in pre-LLM days, it&#x27;s not like I would have hired a x264 expert to do this job for me. I would have either had to spend hours more on this task, or more likely, this 97 year old man would never have seen his great granddaughter&#x27;s dance, which apparently brought a massive smile to his face.Like everything before them, LLMs are just tools. Neither inherently good nor bad. It&#x27;s what we do with them and how we use them that matters.

It has been for me the last 15 years like this.

I have been noticing this trend increasingly myself. It&#x27;s getting more and more difficult to use tools like Google search to find relevant content.Many of my searches nowadays include suffixes like &quot;site:reddit.com&quot; (or similar havens of, hopefully, still mostly human-generated content) to produce reasonably useful results. There&#x27;s so much spam pollution by sites like Medium.com that it&#x27;s disheartening. It feels as if the Internet humanity is already on the retreat into their last comely homes, which are more closed than open to the outside.On the positive side:1. Self-managed blogs (like: not on Substack or Medium) by individuals have become a strong indicator for interesting content. If the blog runs on Hugo, Zola, Astro, you-name-it, there&#x27;s hope.2. As a result of (1), I have started to use an RSS reader again. Who would have thought!I am still torn about what to make of Discord. On the one hand, the closed-by-design nature of the thousands of Discord servers, where content is locked in forever without a chance of being indexed by a search engine, has many downsides in my opinion. On the other hand, the servers I do frequent are populated by humans, not content-generating bots camouflaged as users.

Yeah, this is why I still use the web to order things in a nutshell lol

&gt; If I can in any way purchase something without the web, I&#x27;mma do thatTo get to the milk you&#x27;ll have to walk by 3 rows of chips and soda.

R.I.P. Sheldon Brown T_T(The Agner Fog of cycling?)

No disagreement for the most part.I used to be able to say search for Trek bike derailleur hanger and the first result would be what I wanted. Now I have to scroll past 5 ads to buy a new bike, one that&#x27;s a broken link to a third party, and if I&#x27;m really lucky, at the bottom of page 1 will be the link to that part&#x27;s page.The shitification of the web is real.

It is, but the SEO spammers who ruined the web want it to be shopping mall, and they can&#x27;t even do a particularly good job at being one.

The web is much more than a shopping site.

Sounds like your laptop is wholly out of date, you need to buy the next generation of laptops on Amazon that can handle the modern SEO load. I recommend the:LEEZWOO 15.6&quot; Laptop - 16GB RAM 512GB SSD PC Laptop, Quad-Core N95 Processor Up to 3.1GHz, Laptop Computers with Touch ID, WiFi, BT4.2, for Students&#x2F;BusinessName rolls off the tongue doesn’t it

If search is your metric, the web was dead long before OpenAI&#x27;s release of GPT. I gave up on web search a long time ago.

You can still get to product reviews directly and search them. Here&#x27;s an example:Product page (copy the identifier at the end): <a href="https:&#x2F;&#x2F;www.amazon.com&#x2F;Long-Thanks-Hitchhikers-Guide-Galaxy-ebook&#x2F;dp&#x2F;B001OF5F1E&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.amazon.com&#x2F;Long-Thanks-Hitchhikers-Guide-Galaxy-...</a>Review page (paste the identifier at the end): <a href="https:&#x2F;&#x2F;www.amazon.com&#x2F;product-reviews&#x2F;B001OF5F1E&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.amazon.com&#x2F;product-reviews&#x2F;B001OF5F1E&#x2F;</a>This seems to bypass all of the LLM stuff for now.

Ran into this the other day. Amazon.ca still has the old version for now

On Amazon, you used to be able to search the reviews and Q&amp;A section via a search box. This was immensely useful. Now, that search box first routes your search to an LLM, which makes you wait 10-15 seconds while it searches for you. Then it presents its unhelpful summary, saying &quot;some reviews said such and such&quot;, and I can finally click the button to show me the actual reviews and questions with the term I searched.This is going to be the thing that makes me quit Amazon. If I&#x27;m missing something and there&#x27;s still a way to to a direct search, please tell me.

Can vouch for this. It’s the first non-Google search alternative I’ve used that has 100% replaced Google. I don’t need Google as a fallback like I did with others.

There is a startup whose product is better search. The killer feature is that you pay for it, so you aren&#x27;t the produdct. <a href="https:&#x2F;&#x2F;kagi.com&#x2F;welcome" rel="nofollow">https:&#x2F;&#x2F;kagi.com&#x2F;welcome</a>

I suppose it is just Amazon problems. I have never lived in the area where Amazon is prevalent. Where I live, search engines still can&#x27;t find synonyms or process misspellings.

for tech stuff I just use documentation, bug trackers and source code now. Web searching has become useless.

&gt; Their old website was plain and worked great, letting me quickly search through their products and quickly purchase them. Last night I literally struggled to add things to cart and check out; it was actually harrowing.Hey, who cares about making services that work when we can give people a cool chatbot assistant and a 1800 number with no real-person alternative to the decision tree

I&#x27;ve been slowly detaching myself from the web for the past 10 years. These days I mostly build offline apps using native technologies. Those capabilities are still around. They just receded for a while because they&#x27;d gotten so polluted with toolbars and malware. But now the malware is on the other side, and native apps are cool again. If you know where to look. Here&#x27;s my shingle: <a href="https:&#x2F;&#x2F;akkartik.name&#x2F;freewheeling-apps" rel="nofollow">https:&#x2F;&#x2F;akkartik.name&#x2F;freewheeling-apps</a>On the other hand, what you call &quot;The Web&quot; seems to be just what you can get at through search engines. There&#x27;s still the old web, the thing that&#x27;s mediated by relationships and reputation rather than aggregation services with billions of users. Like the link I shared above. Or this heroically moderated site we&#x27;re using right now.

I&#x27;m going to call it: The Web is dead. Thanks to &quot;AI&quot; I spend more time now digging through searches trying to find something useful than I did back in 2005. And the sites you do find are largely garbage.As a random example: just trying to find a particular popular set of wireless earbuds takes me at least 10 minutes, when I already know the company, the company&#x27;s website, other vendors that sell the company&#x27;s goods, etc. It&#x27;s just buried under tons of dreck. And my laptop is &quot;old&quot; (an 8-core i7 processor with 16GB of RAM) so it struggles to push through graphics-intense &quot;modern&quot; websites like the vendor&#x27;s. Their old website was plain and worked great, letting me quickly search through their products and quickly purchase them. Last night I literally struggled to add things to cart and check out; it was actually harrowing.Fuck the web, fuck web browsers, web design, SEO, searching, advertising, and all the schlock that comes with it. I&#x27;m done. If I can in any way purchase something without the web, I&#x27;mma do that. I don&#x27;t hate technology (entirely...) but the web is just a rotten egg now.

Why wordfreq will not be updated