You would be absolutely correct. the lossiness is in the resolution of the image (512x512 is pretty terrible) but given enough image resolution it&#x27;s just an FFT transform, and the only reason that stuff falls short is because people don&#x27;t give it, in turn, enough resolution. If you did wild overkill of the resolution of an FFT transform you could do anything you wanted with no loss of tone quality. If you turned that to visual images and did diffusion with it you could do AI diffusion at convincing audio quality.In theory the tone quality is not an objection here. When it sounds bad it&#x27;s because it&#x27;s 512x512, because the FFT resolution isn&#x27;t up to the task, etc. People cling to very inadequate audio standards for digital processing, but you don&#x27;t have to.

regarding those usual objections, i&#x27;d argue that a spectrograph representation of a given piece of audio is just a different (lossy) encoding of the same content&#x2F;information, so any hypothetical objections would still apply here.

Why not? Music copyright was not even about audio recordings originally.

This is so good that I wondered if it&#x27;s fake.  Really impressive results from generated spectrographs!  Also really interesting that it&#x27;s not exactly trained on the audio files themselves - wonder if the usual copyright-based objections wild even apply here.

Me neither, perhaps the web app is a bit buggy?

I also don&#x27;t hear anything, even when my prompt was selected...

Same here, servers are overloaded probably. Shame, I was really looking forward to a Wu Tang Clan and Jamiroquai collab

I&#x27;m getting this back when I try to hear cats sing me a rock opera:{&quot;data&quot;:{&quot;success&quot;:true,&quot;worklet_output&quot;:{&quot;error&quot;:&quot;Model version 5qekv1q is not healthy&quot;},&quot;latency_ms&quot;:530}}

Same earlier, but I can now get it to work very intermittently, with the error &quot;Uh oh! Servers are behind, scaling up...&quot;

Really cool. Can&#x27;t get this to work on the homepage though.Might be a traffic thing?Edit: Works now. A bit laggy but it works. Brilliant!

Xmd5a is already a real track.<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=crcqADcAusg" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=crcqADcAusg</a>

Rearrange that trip through latent space a little, jumping back and forth through different stages of the interpolation in a pattern resembling those customary chorus&#x2F;verse things and you&#x27;ve got a hit. And you could reuse the exact same rearrange recipe for just about any interpolation between prompt pairs.Plenty of times this has been called before, and it&#x27;s certainly possible that this wont be the last time this is called, but allow me to declare this the end of the bedroom producer (stage performers remain unaffected). And no two elevators will ever sound the same again, on any day.

Very impressive. I am quite confident that next years number one Christmas hit will start like &quot;church bells to electronic beats&quot;.

I think you&#x27;d have to start with separate spectrograms per instrument, then blend the complete track in &quot;post&quot; at the end.

Producing images of spectrograms is a genius idea. Great implementation!A couple of ideas that come to mind:- I wonder if you could separate the audio tracks of each instrument, generate separately, and then combine them. This could give more control over the generation. Alignment might be tough, though.- If you could at least separate vocals and instrumentals, you could train a separate model for vocals (LLM for text, then text to speech, maybe). The current implementation doesn&#x27;t seem to handle vocals as well as TTS models.

great stuff, while it comes with the usual smeary iFFT artifacts that AI-generated sound tends to have the results are surprisingly good. 
i especially love the nonsense vocals it generates in the last example, which remind me of what singing along to foreign songs felt like in my childhood.

impressive stuff. reminds me of when ppl started using image classifier networks on spectrograms in order to classify audio. i would not have thought to apply a similar concept for generative models, but it seems obvious in hindsight.

The interpolation from keyboard typing to jazz is incredible. This is what AI art should be.

Nice reference! I had never seen that site before, but those albums had a significant impact on my musical journey.There was a purple victorian house in Colorado Springs where the living room was converted into a record and cd store called Life By Design. I picked up these albums and a ton of other obscure music there. I was so happy to not have to drive all the way up to Wax Trax in Denver to be able to discover new artists.

For the 30 anniversary? <a href="https:&#x2F;&#x2F;warp.net&#x2F;gb&#x2F;artificial-intelligence" rel="nofollow">https:&#x2F;&#x2F;warp.net&#x2F;gb&#x2F;artificial-intelligence</a>

Fun! I tried something similar with DCGAN when it first came out, but that didn&#x27;t exactly make nice noises. The conversion to and from Mel spectrograms was lossy (to put it mildly), and DCGAN, while impressive in its day, is nothing like the stuff we have today.Interesting that it gets so good results with just fine tuning the regular SD model. I assume most of the images it&#x27;s trained on are useless for learning how to generate Mel spectrograms from text, so a model trained from scratch could potentially do even better.There&#x27;s still the issue of reconstructing sound from the spectrograms. I bet it&#x27;s responsible for the somewhat tinny sound we get from this otherwise very cool demo.

Does anyone have any good guides&#x2F;tutorials for how to fine-tune Stable Diffusion? I&#x27;m not talking about textual inversion or dreambooth.

This is amazing! Would it be possible to use it to modify this interpolate between two existing songs (i.e. generate spectrograms from audio and transition between them)?

Absolutely incredible - from idea to implementation to output.

You can also add another neural-network to &quot;smooth&quot; the spectrogram, increase the resolution and remove artefacts, just like they do for image generation.

In very limited situations. You can move a frequency around (or drop it entirely) if it&#x27;s being masked by a nearby loud frequency. Otherwise, you would be amazed at the sensitivity of pitch perception.

You probably mean Karlheinz Brandenburg, the developer of MP3, who worked on psychoacoustics. Not completely off though, as he did the research at a Fraunhofer research institute, which takes its name from Joseph von Fraunhofer, the inventor of the spectroscope.

Wasn&#x27;t this Fraunhofer&#x27;s big insight that led to the development of MP3? Human perception actually is pretty forgiving of perturbations in the Fourier domain.

Any chance of spoken voice-work being possible? It would be interesting to see if a model could &quot;speak&quot; like James Earl Jones or Steve Blum.

Author here: We were blown away too. This project started with a question in our minds about whether it was even possible for the stable diffusion model architecture to output something with the level of fidelity needed for the resulting audio to sound reasonable.

I think the progression from church bells to electronic beats is especially good: <a href="https:&#x2F;&#x2F;www.riffusion.com&#x2F;about&#x2F;church_bells_to_electronic_beats.mp3" rel="nofollow">https:&#x2F;&#x2F;www.riffusion.com&#x2F;about&#x2F;church_bells_to_electronic_b...</a>

I think we&#x27;re at the point, with these AI generative model thingies, where the practitioners are mesmerized by the mechatronic aspect like a clock maker who wants to recreate the world with gears, so they make a mechanized puppet or diorama and revel in their ingenuity.

Look at GAN art from a few years ago, compared to MidJourney v4.

Really? They sound quite clearly like the prompt to me if I “squint my ears” a little

It&#x27;s...not effective though. Am I listening to the wrong thing here? Everything I hear from the web app is jumbled nonsense.

This really is unreasonably effective. Spectrograms are a lot less forgiving of minor errors than a painting. Move a brush stroke up or down a few pixels, you probably won&#x27;t notice. Move a spectral element up or down a bit and you have a completely different sound. I don&#x27;t understand how this can possibly be precise enough to generate anything close to a cohesive output.Absolutely blows my mind.

Indeed, I think this would be a cost-effective way to go forward.

Perhaps GPT could run on top of Stable-diffusion, generating output in the form of written text (glyphs).

Maybe next up is a model that generates Piet code<a href="https:&#x2F;&#x2F;www.dangermouse.net&#x2F;esoteric&#x2F;piet.html" rel="nofollow">https:&#x2F;&#x2F;www.dangermouse.net&#x2F;esoteric&#x2F;piet.html</a>

Makes me wonder if we will see a generalization of this idea. Just like in a CPU 90%+ of want you want to do can be modeled with very few instructions (mov, add, jmp..) we could see a set of very refined models (Stable difussion, GPT, etc) and all of their abstractions on top (ChatGPT, Rifussion, etc).

For what is worth, people were trying the same thing with GANs (I also played with doing it with stylegan a bit) but the results weren&#x27;t as good.The amazing thing is that the current diffusion models are so good that the spectograms are actually reasonable enough despite the small room for error.

Not only that but sampling. I&#x27;d say there&#x27;s at least one sample from something in most modern music. This can essentially create &quot;sounds&quot; that you&#x27;re looking for as an artist. I need a sort of high pitched drone here... Rather than dig through sample libraries you just generate a few dozen results from a diffusion model with some varying inputs and you&#x27;d have a small sample set on the exact thing you&#x27;re looking for. There&#x27;s already so much processing of samples after the fact, the actual quality or resolution of the sample is inconsequential. In a lot of music, you&#x27;re just going after the texture and tonality and timbre of something... This can be seen in some Hans Zimmer videos of how he slows down certain sounds massively to arrive at new sounds... or in granular synthesis... This is going to open up a lot of cool new doors.

I was thinking gaming where music can and should dynamically shift based on different environmental and player conditions.

&gt; I can&#x27;t wait to hear some serious AI music-making a few years from now.I think this will be particularly useful for musical compositions in movies and film, where the producer can &quot;instruct&quot; the AI about what to play, when, and how to transition so that the music matches the scene progression.

I suspect that if you had tried this with previous image models the results would have been terrible. This only works since image models are so good now.

I&#x27;m super excited about the Audio AI space, as it seems permanently a few years behind image stuff - so I think we&#x27;re going to see a lot more of this.If you&#x27;re interested, the idea of applying Image processing techniques to Spectrograms of audio is explored in brief in the first lesson of one of the most recommended AI courses on HN: Practical Deep Learning for Coders <a href="https:&#x2F;&#x2F;youtu.be&#x2F;8SF_h3xF3cE?t=1632" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;8SF_h3xF3cE?t=1632</a>

Do you have any sources for more information about this?

I dabble in music production and know some of the people in the &quot;Lofi&quot; world, so I know for a fact that this is not true. It&#x27;s just a formulaic sub-genre where people are trying to make similar instrumentals with the same vibe. It would be jarring to listen to a playlist while studying and each song had wildly different tempos, instruments, etc.Also, the music doesn&#x27;t sound &quot;Lofi&quot; because it&#x27;s generated by algorithms. A lot of hard work and software goes into taking a clean, pitch-perfect digital signal and making it sound like something playing on a record player from the 70s.

You already hear a ton of them. Lofi music on these massively popular channels are basically auto-generated &quot;music&quot; + auto generated artwork.

The idea of connecting CV to audio via spectrograms pre dates Jeremy Howard&#x27;s course by quite a bit. That&#x27;s not really the interesting part here though. The fact that a simple extension of an image generation pipeline produces such impressive results with generative audio is what is interesting. It really emphasizes how useful the idea of stable diffusion is.edit: added a bit more to the thought

The idea to apply computer vision algorithms to spectrograms is not new. I don&#x27;t know who first came up with it, but I first came across it about a decade ago.I just ran a quick Google Scholar search, and the first result is <a href="https:&#x2F;&#x2F;ieeexplore.ieee.org&#x2F;abstract&#x2F;document&#x2F;5672395" rel="nofollow">https:&#x2F;&#x2F;ieeexplore.ieee.org&#x2F;abstract&#x2F;document&#x2F;5672395</a>This is from 2010. I didn&#x27;t go looking, but it wouldn&#x27;t surprise me if the idea is older than that.

This idea is presented by Jeremy Howard on literally their first Deep Learning for Coders class (most recent edition). A student wanted to classify sounds, but only knew how to do vision, so they converted sounds to spectrograms, fine tuned the model on the labelled spectra, and the classification worked pretty well on test data.
That of course does not take the merit away from the Riffusion authors though.

As someone who loves making music and loves listening to music (regardless of its origins, in my case), it doesn&#x27;t make me that sad. Sure, at first, I had an uncomfortable feeling that AI could make this sacred magic thing that only I and other fellow humans know how to do... But then I realized same thing is happening with visual art, so I applied the same counterarguments that&#x27;ve been cooking in my head.I think that kind of attitude is defeatist - it&#x27;s implying that humans will be stopped from making music if AI learns how to do it too. I don&#x27;t think that will happen. Humans will continue making music, as they always have. When Kraftwerk started using computers to make music back in the 70s, people were also scared of what that will do to musicians. To be fair, live music has died out a bit (in a sense that there aren&#x27;t that many god-on-earth-level rockstars), but it&#x27;s still out there, people are performing, and others who want to listen can go and listen.Maybe consumers will start consuming more and more AI music, instead of human music [0], but the worst thing that can happen is that music will no longer be a profitable activity. But then again, today&#x27;s music industry already has some elements of the automation - washed-out rhythms, sexual thematics over and over again, re-hashing same old songs in different packages... So nothing&#x27;s gonna change in the grand scheme of things.[0] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=S1jWdeRKvvk" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=S1jWdeRKvvk</a>

I would rather live in a world where humans get to do things they love because they can (and not because they have to earn their bread), and machines get to do basically everything that needs to be done but no human is willing to do it.Advancing AI capabilities in no way detracts from this. You talk about humans being &quot;relegated to the remaining tasks&quot; - but that&#x27;s a consequence of our socioeconomic system, not of our technology.

I&#x27;d rather live in the world where humans do things that are actually unique and interesting, and aren&#x27;t essentially being artificially propped up by limiting competition.I don&#x27;t see this as a threat to human ingenuity in the slightest.

I play the piano (badly). There are many other people who can play much better than I. There are simple computer programs which can play better. It doesn&#x27;t stop me from enjoying it or playing it. Computers have been beating people at Chess for years yet you still see people everywhere enjoying the game. At some point computers will be better than humans at absolutely everything but it shouldn&#x27;t stop you as a human from enjoying anything.

Musicians already make much (most?) of their money via gigs and I don&#x27;t think going to watch an AI play at a gig will be all too common. I think we&#x27;ll be fine. Might have to adapt though.

There are still chess tournaments for humans, even though our smartphone could play chess better than any grandmaster.

As someone who loves making music and loves listening to music made by other humans with intention, it just makes me sad.Sure, AI can do lots of things well. But would you rather live in a world where humans get to do things they love (and are able to afford a comfortable life while doing so) or a world where machines do the things humans love and humans are relegated to the remaining tasks that machines happened to be poorly suited for?

This is a genius idea. Using an already-existing and well-performing image model, and just encoding input&#x2F;output as a spectrogram... It&#x27;s elegant, it&#x27;s obvious in retrospective, it&#x27;s just pure genius.I can&#x27;t wait to hear some serious AI music-making a few years from now.

Congratulations this is an amazing application of technology and truly innovative. This could be leveraged by a wide range of applications that I hope you&#x27;ll capitalize on.

I agree but it&#x27;s better than listening to Ed SheeranEdit: To be honest, I find something like &#x27;Band In A Box&#x27; to be more impressive and actually useful, I don&#x27;t understand how I would ever use this or listen to this. To me, it&#x27;s further proof that Stable Diffusion really just doesn&#x27;t work that well

Doesn&#x27;t have enough training data for odd time signatures music... or John Cages&#x27; 4&#x27;33 :D

Very cool! I was wondering why there wasnt any music diffusion apps out there, it seems more useful because music has stricter copyright and all content creators need some background music ...

They were killed off by the ability to record the data. Every city used to have their own music stars :)

The raw outputs of these tools will be best consumed by experts. Until general AI, these are just better tools for the same workers.

&quot;We should place a tax on all copyright lawyers and use it to fund GPU manufacturing and AI development. At your next stump speech, mention how the entertainment industry is stealing jobs from construction workers. Your corruption charges won&#x27;t matter because voters only care about corruption when it&#x27;s not in their favor.&quot;

Politicians, bureaucracy.GPT-3, what policy should we apply to increase tax revenue by 5% given these constraints?GPT-3, please tell me some populist thing to say to win the next election, or how should I deflect these corruption charges.

Although I do wonder how much an earlier technology, audio reproduction, contributed to that. My grandmother worked for a time as a piano player as part of a nightclub orchestra. It was a stable job back then. I have to wonder how many musician jobs were killed off by the jukebox and related technologies.

Musicians were made to get a day job long before you were born ;)

If a hack based on an image generator already has promising results for music generation, then imagine what will happen if something dedicated to music is built from the ground up.

If I was a musician, this post would not make me worry for a second

What is with the hyperbole in this thread? This stuff sounds like incoherent noise. It is noticeably worse than AI audio stuff I heard 5 years ago. What is going on with the responses here?

This was the first AI thing to fill me with a feeling of existential dread.

Have you tried AI asset generators?  They are working extremely good. Just yesterday a friend of mine has shown me the progress they made in their game. It is incredible.
Designers are 100% loosing their job over this.

Honestly none of them should. I think the moral panic around these things is way overstated. They are cool but hardly about to replace anyone&#x27;s job.

Earlier this year, graphic designers, last month it was software engineers, and now musicians are also feeling the effects.Who else will AI make looking for a new job?

Wow, diffusion could be a game changer for audio restoration.

Really fascinating. I&#x27;d be interested to know more about how it was trained, with what data exactly.

No.And I suspect this will always have phase smearing, because it&#x27;s not doing any kind of source separation or individual synthesis. It&#x27;s effectively a form of frequency domain data compression, so it&#x27;s always going to be lossy.It&#x27;s more like a sophisticated timbral morph, done on a complete short loop instead of an individual line.It would sound better with a much higher data density. CD quality would be 220500 samples for each five second loop. Realtime FFTs with that resolution aren&#x27;t practical on the current generation of hardware, but they could be done in non-realtime. But there will always be the issue of timbres being distorted because outside of a certain level of familiarity and expectation our brains start hearing gargly disconnected overtones instead of coherent sound objects.What this is not doing is extracting or understanding musical semantics and reassembling them in interesting ways. The harmonies in some of these clips are pretty weird and dissonant, and not what you&#x27;d get from a human writing accessible music. This matters because outside of TikTok music isn&#x27;t about 5s loops, and longer structures aren&#x27;t so amenable to this kind of approach.This won&#x27;t be a problem for some applications, but it&#x27;s a long way short of the musical equivalent of a MidJourney image.Generally we&#x27;re a lot more tolerant of visual &quot;bugs&quot; than musical ones.

&gt; composition&#x27;s lacking, too. It&#x27;s loop-y.Well no wonder, it has absolutely no concept of composition beyond a single 5s loop, if I understand correctly.&gt; It absolutely sucks at cymbals, though. Everything sounds like realaudio :)&gt; It could probably do bad modern production fairly well even now :) exaggeration, but not much, when stuff is really overproduced it starts to get way more indistinct, and this can do indistinct. It&#x27;s realaudio grade, it needs to be more like 128kbps mp3 grade.I haven&#x27;t sat down yet to calculate it, but is the output of SD at 512*512px at 24bit enough to generate audio CD quality in theory?

Some of this is really cool! The 20 step interpolations are very special, because they&#x27;re concepts that are distinct and novel.It absolutely sucks at cymbals, though. Everything sounds like realaudio :) composition&#x27;s lacking, too. It&#x27;s loop-y.Set this up to make AI dubtechno or trip-hop. It likes bass and indistinctness and hypnotic repetitiveness. Might also be good at weird atonal stuff, because it doesn&#x27;t inherently have any notion of what a key or mode is?As a human musician and producer I&#x27;m super interested in the kinds of clarity and sonority we used to get out of classic albums (which the industry has kinda drifted away from for decades) so the way for this to take over for ME would involve a hell of a lot more resolution of the FFT imagery, especially in the highs, plus some way to also do another AI-ification of what different parts of the song exist (like a further layer but it controls abrupt switches of prompt)It could probably do bad modern production fairly well even now :) exaggeration, but not much, when stuff is really overproduced it starts to get way more indistinct, and this can do indistinct. It&#x27;s realaudio grade, it needs to be more like 128kbps mp3 grade.

I wonder if this is true for music. Our ears are much more discerning than our eyes when it comes to art it seems.

Because art is the low hanging fruit of &quot;close enough&quot; applications.

&gt; You can&#x27;t automate a live performance or an oil painting with AI in this way.You&#x27;d have to combine it with these guys <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=WqE9zIp0Muk" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=WqE9zIp0Muk</a>

You can&#x27;t automate a live performance or an oil painting with AI in this way. This isn&#x27;t going to replace musicians and artists. If anything, I think a preponderance of AI art would make people appreciate the real stuff more.As to why, music is fun to create, and this is just a tool.

Everyone asked for this, including artists. If you make a living off of making art, having the best tools to help you do that is a constant, and the tools are finally starting to get properly good. Will &quot;the job&quot; change because of the tools? Of course. Will the nature of what it means for something to be art change? Also of course. Art isn&#x27;t some static, untouchable thing. It changes as humanity does.

&gt; Who asked for this?I did.&gt; The future seems depressing when I look at all this AI generated art.You should talk about your concerns with an AI psychotherapist.

That just means there is going to be a whole lot more bad art in the world

Because tons of people want to make art, and a lot of art currently requires years of training to make anything close to &quot;good&quot;. Making art more accessible to create is a boon to everyone who&#x27;s dreamed of being able to make their own paintings and music, but doesn&#x27;t have the skills required.

Actually I agree with you, but HN is not really a place where you will find artists defending themselves. However you will find alot of people defending the automation of art. Generative art has it&#x27;s place. But ultimately until humans are extinct, human generated art is the only thing which really represents the species. Everything else is an advanced form of puppetry or mimicry.

I would say it&#x27;s not &quot;generated,&quot; but interpolated...It doesn&#x27;t make anything new or fresh. It doesn&#x27;t pull any real-life emotions or experiences into a synthesis that a person can relate to. It&#x27;s more like asking a teenaged comedian to imitate numerous impressions of music styles. e.g. in Clerks when the Russian guy does &quot;metal&quot;: <a href="https:&#x2F;&#x2F;youtu.be&#x2F;7gFoHkkCaRE?t=55" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;7gFoHkkCaRE?t=55</a>Of course the modern conception of music in the West is as an accompaniment to other, mostly drudging, activities, as opposed to something to be paid singularly attention. Therefore, there are many &quot;valuable&quot;(*) occasions to produce &quot;impressions&quot; of music. E.g. in advertisements and social media flexes where identity and attitude are the purpose of music. For these, a shallow interpretation or reflection of loosely amalgamated sound clips will suffice. But we don&#x27;t just attend concerts or focus sustained energy on sonic impressions. We listen to lyrics and give over our consciousness to composed works because we want to find secrets others give away in dealing with this crazy thing called life- ideas to succeed, admissions of failure, and what the expected emotional arcs of these trajectories looks like. This lofty goal is to date not within the scope of AI stunts.As Solzheinetysn said, &quot;Too much art is like candy and not bread.&quot;

We&#x27;re not automating art, we&#x27;re creating tools that make it easier for humans to create art. These are nothing more than new and exciting tools. The cream will still rise to the top, same as it ever was.

This is really cool but can someone tell me why we are automating art? Who asked for this? The future seems depressing when I look at all this AI generated art.

I think AI would be great at generating similar things. Might be very nice for generating fake languages, too.

That&#x27;s glossolalia, and it&#x27;s not that uncommon in human-created art.

The vocals in these tracks are so interesting. They sound like vocals, with the right tone, phonemes. and structure for the different styles and languages but no meaning.Reminds me of the soundtrack to Nier Automata which did a similar thing: <a href="https:&#x2F;&#x2F;youtu.be&#x2F;8jpJM6nc6fE" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;8jpJM6nc6fE</a>

&gt; Hard for me to imagine how AI would be able to create something as unique and human as D&#x27;Angelo&#x27;s Voodoo (esp. before he existed)There’s always that immortal randomly typing monkey with a typewriter thing [1]. And, in our case, it seems to be better than random.So, yes, perhaps.
But perhaps we could instead build and create things that are yet unimaginable upon it. We’ll see.[1]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Infinite_monkey_theorem" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Infinite_monkey_theorem</a>

I dont think its possible to make a compelling song in quite the same way. At Neptunely, we are pursuing a route that keeps the human in the picture. More of a collaboration. <a href="https:&#x2F;&#x2F;neptunely.com" rel="nofollow">https:&#x2F;&#x2F;neptunely.com</a>

Do you guys think AI creative tools will completely subsume the possibility space of human made music? Or does it open up a new dimension of possibilities orthogonally to it? Hard for me to imagine how AI would be able to create something as unique and human as D&#x27;Angelo&#x27;s Voodoo (esp. before he existed) but maybe it could (eventually).If I understand these AI algorithms at a high level, they&#x27;re essentially finding patterns in things that already exist and replicate it w some variation quite well. But a good song is perfect&#x2F;precise in each moment in time. Maybe we&#x27;ll only be ever be able to get asymptotically closer but never _quite_ there to something as perfectly crafted a human could make? Maybe there will always be a frontier space only humans can explore?

&gt;in the workplaceOr at a house party, club or restaurant... as more people arrive or leave and the energy level rises or declines..or human rhythms speed up or slow down...so does the music...

I would not want to go to the hospital for a mild ear infection and hear the AI start blasting death metal.

Or perhaps use it in a hospital to play music that matches the state of a patient’s health as they are passing away.

I bet a cool riff on this would be to simply sample an ambient microphone in the workplace and use that the generate and slowly introduce matching background music that fits the current tenor of the environment. Done slowly and subtly enough I&#x27;d bet the listener may not even be entirely aware its happening.If we could measure certain kinds of productivity it might even be useful as a way to &quot;extend&quot; certain highly productive ambient environments a la &quot;music for coding&quot;.

Awesome, there is another project out there that does it with CPU <a href="https:&#x2F;&#x2F;github.com&#x2F;marcoppasini&#x2F;musika" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;marcoppasini&#x2F;musika</a> maybe mix the both, ie take initial output of musika, convert to spectrogram and feed it to riffusion to get more variation...

Fascinating stuff.One of the samples had vocals. Could the approach be used to create solely vocals?Could it be used for speech? If so, could the speech be directed or would it be random?

&quot;fine-tuned on images of spectrograms paired with text&quot;How many paired training images &#x2F; text and what was the source of your training data? Just curious to know how much fine tuning was needed to get the results and what the breadth &#x2F; scope of the images were in terms of original sources to train on to get sufficient musical diversity.

Authors here: Fun to wake up to this surprise! We are rushing to add GPUs so you can all experience the app in real-time. Will update asap

I&#x27;m floored, the typing to jazz demo is WILD! Please keep pushing this space, you&#x27;ve got something real special here.

This happened earlier than I expected, and using a much different technique than I expected.Bracing myself for when major record labels enter the copyright brawl that diffusion technology is sparking.

yeah, it&#x27;s pretty unsurprising, that they&#x27;re both uncanny valley like a messy circus.

I find it really cool that the &quot;uncanny valley&quot; that&#x27;s audible on nearly every sample is exactly as I would imagine that the visual artifacts would sound that crop up in most generated art. Not really surprising I guess, but still cool that there&#x27;s such a direct correlation between completely different mediums!

RTX 3070 can generate SD results in under 5 seconds, depending. Euler A 20 samples, 512x512. it can almost do 4 images in 5 seconds with those settings.It&#x27;s possible a 3060 might work, depending. in my experience the 3060 is about 50% slower than the 3070, but that may be a bad 3060 in our test rig. but a 3060 gets pretty close to 5 seconds for an image, so try it, if you have one.just tested prompt &quot;a test pattern for television&quot; on both cards and 3070 took 1.87s and the 3060 took 2.93s. Similar results for the prompt &quot;an intricate cityscape, like new york&quot;edit: i should note we&#x27;re using SD 1.4, not 1.5, although i think that just has to do with the checkpoint of the model, not the algorithm, but i could be wrong.Also the model is over 14GB, so perhaps the 3070 can&#x27;t do it after all. i&#x27;ll test it later as soon as the local admin wakes up and downloads it onto our machine.

Author here: fwiw we are running the app on a10g GPUs, which generally can turn around a 512x512 in 3.5s with 50 inference steps. This time includes converting the image into audio which should be done on the GPU as well for real-time purposes. We did some optimization such as a traced unet, fp16 and removing autocast. There are lots of ways it could be sped up further I&#x27;m sure!

I read the article:&quot;If you have a GPU powerful enough to generate stable diffusion results in under five seconds, you can run the experience locally using our test flask server.&quot;Curious what sort of GPU the author was using or what some of the min requirements might be?

Yes from <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;runwayml&#x2F;stable-diffusion-v1-5" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;runwayml&#x2F;stable-diffusion-v1-5</a>. Our checkpoint works with automatic1111, and if you&#x27;d like to make an extension to decode to audio, it should be pretty straightforward: <a href="https:&#x2F;&#x2F;github.com&#x2F;hmartiro&#x2F;riffusion-inference&#x2F;blob&#x2F;main&#x2F;riffusion&#x2F;audio.py" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;hmartiro&#x2F;riffusion-inference&#x2F;blob&#x2F;main&#x2F;ri...</a>

When you say fine tuned do you mean fine tuned on an existing stable diffusion checkpoint? If so which?It would be very interesting to see what the stable diffusion community that is using automatic1111 version would do with this if it were made into an extension.

I was very surprised this was not mentioned.

Amazing work! Did you use CLIP or something like that to train genre + mel-spectrogram? What datasets did you use?

Presumably for similar reasons that the vast majority of AI generated art and text is off-puttingly hideous or bland. For every stunning example that gets passed around the internet, thousands of others sucked. Generating art that is aesthetically pleasing to humans seems like the Mt. Everest of AI challenges to me.

You&#x27;re probably talking about the artifacts of converting a low resolution spectrogram to audio.

It sounds kind of like the visual artifacts that are generated by resampling in two dimensions. Since the whole model is based on compressing image content, whatever it&#x27;s doing DSP-wise is more-or-less &quot;baked in&quot;, and a probable fix would lie in doing it in a less hacky way.

The link is down now, so I don&#x27;t know about this one.  But most generated music is generated in the note domain, rather than the audio domain.  Any unpleasant resonance would  introduced in the audio synthesis step.  And audio synthesis from note data is a very solved problem for any kind of timbre you can conceive of, and some you can&#x27;t.

I think this is because the generation is done in the frequency domain. Phase retrieval is based on heuristics and not perfect, so it leads to this &quot;compressed audio&quot; feel. I think it should be improvable

I&#x27;ve done some work on AI audio synthesis and the artifacts you&#x27;re hearing in these clips are coming from the algorithm that is used to go from the synthesized spectrogram to the audio (the Griffin-Lim algorithm).Audio spectrograms have two components: the magnitude and the phase. Most of the information and structure is in the magnitude spectrogram so neural nets generally only synthesize that. If you were to look at a phase spectrogram it looks completely random and neural nets have a very, very difficult time learning how to generate good phases.When you go from a spectrogram to audio you need both the magnitudes and phases, but if the neural net only generates the magnitudes you have a problem. This is where the Griffin-Lim algorithm comes in. It tries to find a set of phases that works with the magnitudes so that you can generate the audio. It generally works pretty well, but tends to produce that sort of resonant artifact that you&#x27;re noticing, especially when the magnitude spectrogram is synthesized (and therefore doesn&#x27;t necessarily have a consistent set of phases).There are other ways of using neural nets to synthesize the audio directly (Wavenet being the earliest big success), but they tend to be much more expensive than Griffin-Lim. Raw audio data is hard for neural nets to work with because the context size is so large.

The first ever recordings had people shouting to get anything to register. They sounded like tin. Fast forward to today.Looking back at image generation just a year or two ago and people would have said similar things.Not hard to imagine the trajectory of synthesized audio taking a similar path.

All the AI music I’ve heard so far has a really unpleasant resonant quality to it. Why is that? Can it be removed?

I can simultaneously burst a bubble and provide fuel for more -- the alignment of the intrinsic manifolds of different domains has been an interesting research topic for zero shot research for a few years. I remember seeing at CVPR 2018 the first zero shot...classifier, I think? That if I recall correctly trained in two domains that were automatically basically aligned with each other enough to provide very good zero shot accuracy.Calling it a Deep Truth might be a bit of an emotional marketing spin but the concept is very exciting nonetheless I believe.

Wow, I am blown away. Some of these clips are really good! I love the Arabic Gospel one. John and George would have loved this so much. And the fact that you can make things that sound good by going through visual space feels to me like the discovery of a Deep Truth, one that goes beyond even the Fourier transform because it somehow connects the aesthetics of the two domains.

is classical music harder? noticed you didn&#x27;t have any classical music tracks. i wonder if it is because it is more structured?

The audio sounds a bit lossy, would it be possible to create high quality spectograms from music, downsample them, and use that as training data for a spectogram upscaler?It might be the last step this AI needs to bring some extra clarity to the output.

&#x2F;u&#x2F;threevox on reddit made a colab for playing with the checkpoint:<a href="https:&#x2F;&#x2F;colab.research.google.com&#x2F;drive&#x2F;1FhH3HlN8Ps_Pr9OR6Qcfbfz7utDvICl0?usp=sharing" rel="nofollow">https:&#x2F;&#x2F;colab.research.google.com&#x2F;drive&#x2F;1FhH3HlN8Ps_Pr9OR6Qc...</a>

Author here: It can certainly be applied to voice, but the model would need deeper training to speak intelligibly. If you want to hear more singing, you can try a prompt like &quot;female voice&quot;, and increase the denoising parameter in the settings of the app.That said, our GPUs are still getting slammed today so you might face a delay in getting responses. Working on it!

Amazing work. Can this be applied to voice?Example prompt: “deep radio host voice saying ‘hello there’”Kind of like a more expressive TTS?

Reach out to the Beatstars CEO. He was looking for an AI play for his music producers marketplace. Probably solid B2B lead there.

Hayk! How smart are you! I loved your work on SymForce and Skydio - totally wasn&#x27;t expecting you to be co-author on this!On a serious note, I&#x27;d really love some advice from you on time management and how you get so much done? I love Skydio and the problems you are solving, especially on the autonomy front, are HARD. You are the VP of Autonomy there and yet also managed to get this done! You are clearly doing something right. Teach us, senpai!

To be honest, we&#x27;re not sure how much value image pre training brings. We have not tried to train from scratch, but it would be interesting.One thing that&#x27;s very important though is the language pre-training. The model is able to do some amazing stuff with terms that do not appear in our data set at all. It does this by associating with related words that do appear in the dataset.

Super! Makes sense since Skydio is also amazing.How much data is used for fine tuning? Since spectrograms are (surely?) very out of distribution for the pre training dataset, how much does value does the pre training really bring?

This is groundbreaking! All other attempts at AI generated music have IMO, fallen flat... These results are actually listenable, and enjoyable! This is almost frightening how powerful this can be

Hello - this is awesome work.  Like other commenters, I think the idea that if you are able to transfer a concept into a visual domain (in this case via fft) it  becomes viable to model with diffusion is super exciting but maybe an oversimplification.  With that in mind, do you think this type of approach might work with panels of time series data?

As a listener, I think you&#x27;re probably still safe. Can you use this to help you though? Maybe.It&#x27;s impressive what it produces, but I think it probably lacks substance in the same way the visual AI art stuff does. For the most part, it passes what I call the at-a-glanceness test. It&#x27;s little better than apophenia (the same thing that makes you see shapes in clouds, faces in rocks, or think you&#x27;ve recognised a familiar word in a foreign language; the last one can happen more often though).So, I think these tools will be used to do background work (ie for visuals maybe help with background tasks in CGI or faraway textures in games). I know less about audio, but I assume it could maybe help a DJ create a transition between two segments they want to combine, as opposed make the whole composition for them, but idk if that example makes sense.Now, onto a more human point: I think that people often listen to music because it means something to them. Similar for people who appreciate visual art.I also love interactive and light art, and I love talking to other artists at light festivals who make them because of the stories and journeys behind their art too. Humans and art are a package deal, IMO.Edit: typos and to add: Also, I think prompt authorship is an art unto itself. I&#x27;m amazed what people can craft with it, but I&#x27;m more impressed by the craft itself than the outputs. Don&#x27;t get me wrong, the outputs are darn cool, but not if you look closer. And it&#x27;s impossible to look beneath the surface altogether, as there is nothing in the output but the pixels.

I love simple generative approaches to get ideas, and go from there. This seems like an extension of that (well, it&#x27;s what I&#x27;m going to try - sample the output, make stems, pull MIDI etc). Will make the creative process more interesting for me, not less.Having said that, it&#x27;s not my job, and I can see where the issues lay there.

Why do you think this will kill your job? To me this looks like an extension of the hip-hop genre.

These are tools. Don&#x27;t think of them as replacements, they aren&#x27;t. But as tools that will help us be creative. As smart as these apps seem, they will still need a  human to decide where and how to use them. They won&#x27;t replace  us but we need to adapt to a new reality.

Isn&#x27;t this just a sampler with extra steps?

I can&#x27;t think of a genre that would embrace it faster. The pay-for-knock-off rap beat market will feel more pressure from this kind of tool, especially as loop-oriented as it already is.

As one of the meatsacks whose job you&#x27;re about to kill... eh, I got nothin, it&#x27;s damn impressive. It&#x27;s gonna hit electronic music like a nuclear bomb, I&#x27;d wager.

Crashes repeatedly on iOS in Firefox (my usual browser), is OK on Safari though, so probably not a webkit thing.

The site isn&#x27;t working for me? Anything I have to fix on my side to make it work?

Hi Hayk, I see that the inference code and the final model are open source. I am not expecting it, but is the training code and the dataset you used for fine-tuning, and process to generate the dataset open source?

I&#x27;ve compiled&#x2F;run a dozen different image to sound programs and none of them produce an acceptable sound. This bit of your code alone would be a great application by itself.It&#x27;d be really cool if you could implement an MS paint style spectrum painting or image upload into the web app for more &quot;manual&quot; sound generation.

Did you have a data set for training the relationship between words and the resulting sound?

This is super awesome.Have you already explored doing the same with voice cloning?

funny that Hayk is an early skydio guy!2 amazing AI projects. Huge respect :)

How many songs did you use for the training data?

What sort of setup do you need to be able to fine tune Stable Diffusion models? Are there good tutorials out there for fine tuning with cloud or non-cloud GPUs?

Obviously this needs a little more polish, but I&#x27;ve wanted this for so long I&#x27;m willing to pay for it now if it helps push the tech forward. Can I give you money?

This is amazing! This is a fantastic concept generator. The verisimilitude with specific composers and techniques is more than a little uncanny. A few thoughts after exploring today…- My strongest suggestion is finding some strategy for smoothing over the sometimes harsh-sounding edge of the sample window
- Perhaps it could be filling in&#x2F;passing over segments of what is sounded to user as a larger loop? Both giving it a larger window to articulate things but maybe also showcasing the interpolation more clearly…
- Tone control may seem challenging but I do wonder if you couldn’t “tune” the output of the model as a whole somehow (given the spectrogram format it could be a translation&#x2F;scale knob potentially?)

Amazing work! Do you plan on open-sourcing the code to train the model?

Super clever idea of course. But leaving aside how it was produced, I’ll be one of those who is underwhelmed by the musicality of this. I am judging this in terms of classical music. I repeatedly tried to get it to just play pure piano music without any other add-ons (cymbals etc). It kept mixing the piano with other stuff.Also the key question is - would something like this ever produce something as hauntingly beautiful and unique as classical music pieces?

Other author here! This got a posted a little earlier than we intended so we didn&#x27;t have our GPUs scaled up yet. Please hang on and try throughout the day!Meanwhile, please read our about page <a href="http:&#x2F;&#x2F;riffusion.com&#x2F;about" rel="nofollow">http:&#x2F;&#x2F;riffusion.com&#x2F;about</a>It’s all open source and the code lives at <a href="https:&#x2F;&#x2F;github.com&#x2F;hmartiro&#x2F;riffusion-app" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;hmartiro&#x2F;riffusion-app</a> --&gt; if you have a GPU you can run it yourselfThis has been our hobby project for the past few months. Seeing the incredible results of stable diffusion, we were curious if we could fine tune the model to output spectrograms and then convert to audio clips. The answer to that was a resounding yes, and we became addicted to generating music from text prompts. There are existing works for generating audio or MIDI from text, but none as simple or general as fine tuning the image-based model. Taking it a step further, we made an interactive experience for generating looping audio from text prompts in real time. To do this we built a web app where you type in prompts like a jukebox, and audio clips are generated on the fly. To make the audio loop and transition smoothly, we implemented a pipeline that does img2img conditioning combined with latent space interpolation.

Incredible stuff, Seth &amp; Hayk. I&#x27;ve been thinking nonstop about new things to build using Stable Diffusion and this is the first one that really took my breath away.

Sounds a bit &quot;clowny&quot; to me, for lack of a better word.

Somewhat unrelated, but given the descriptions, perhaps you should take a listen to the Darktide 40k soundtrack if you haven&#x27;t: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=D4hEOMSjzdo">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=D4hEOMSjzdo</a>

Gregorian death metal folk also seems to have lacked seed tunes but the thing is just in its infancy so soon we&#x27;ll be banging our tonsured heads to the folky beats of ......OK, need to create a band name generator to work in tandem with this thing. Let&#x27;s see what one of its brethren in ML makes of it...- &quot;Echoes of the Past&quot;: This name plays on the idea of Gregorian chanting, which is often associated with the distant past, and combines it with the intense and aggressive sound of death metal.- &quot;The Order of the Black Chant&quot;: This name incorporates elements of both the religious connotations of Gregorian chanting and the dark, heavy sound of death metal, creating a sense of mystery and danger.- &quot;Foretold in Blood&quot;: This name evokes both the ancient, mystical nature of Gregorian chanting and the violent themes of death metal, creating a sense of ancient prophecy coming to pass.- &quot;Crypt of the Silent Choir&quot;: This name brings together the eerie, otherworldly sound of Gregorian chanting with the underground, underground feel of death metal, creating a sense of hidden secrets and forbidden knowledge.&quot;The Order of the Black Chant&quot; it shall be.

Yeah... I just wanted some metal and there&#x27;s no way to get anything gruff or agressive out of this at all :-(

yeah, I also couldn&#x27;t get it to do any folk metal. We shall have to wait for metal AI a short while yet haha

It does seem to lack a lot in heavy music in general. My first attempt was to get it to generate something akin to AC&#x2F;DC and it got fairly close but it still seemed a bit too clean and pop-like. Then I tried to get something closer to nu metal or deathcore and it just kept generating some smooth upbeat jazz. Which is not that bad in its own right, but not at all what I asked for.Edit: After a bit of playing around I at least got some credible results with &quot;electric guitar solo, glam rock&quot;. It also understands grunge.

This looks great and the idea is amazing. I tried with the prompt: &quot;speed metal&quot; and &quot;speed metal with guitar riffs&quot; and got some smooth rock-balad type music. I guess there was no heavy metal in the learning samples haha.Great work!

Stable Diffusion &#x27;only&#x27; has ~1B parameters IIRC.

GPT-3 has 175 billion parameters (says Wikipedia). What is the size of the neural network used in this riffusion project?

@haykmartiros, @seth_, thank you for open sourcing this!Played a bit with the very impressive demos, now waiting in queue for my very own riff to get generate.Great as this is, I&#x27;m imagining what it could do for song crossfades (actual mixing instead of plain crossfade even with beat matching).

Things similar to the “interpolation” part (not the generative part) are already used extensively especially for game and movie sound design. Kyma [1] is the absolute leader (it requires expensive hardware though). IMO later iterations on this approach may lead to similar or better results.FYI, other apps that use more classic but still complex Spectral&#x2F;Granular algos :<a href="https:&#x2F;&#x2F;www.thecargocult.nz&#x2F;products&#x2F;envy" rel="nofollow">https:&#x2F;&#x2F;www.thecargocult.nz&#x2F;products&#x2F;envy</a><a href="https:&#x2F;&#x2F;transformizer.com&#x2F;products&#x2F;" rel="nofollow">https:&#x2F;&#x2F;transformizer.com&#x2F;products&#x2F;</a><a href="https:&#x2F;&#x2F;www.zynaptiq.com&#x2F;morph&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.zynaptiq.com&#x2F;morph&#x2F;</a>[1] <a href="https:&#x2F;&#x2F;kyma.symbolicsound.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;kyma.symbolicsound.com&#x2F;</a>

SD can’t generate coherent images if you increase the output size. They’re basically always unusable unless you don’t need any global architecture to them.

8GB is enough to do 1080p resolution. the UI i use for SD maxes out at 2048x2048. however, it takes a lot longer than 512x512 to generate: 1m40s versus 1.97s.I&#x27;m guessing if one had access to one of those nvidia backplane rackmount devices one could generate 8k or larger resolution images.

I’m curious about the limitations of using spectrograms and transient-heavy sounds like drums.It seems like you’d need very high resolution spectrograms to get a consistently snappy drum sound.

I will try it but at least for the name it deserves praise.

&gt; new tech will allow them to cut costs even more, and increase their marginsHow, when everyone and their dog can generate such music? It&#x27;s gonna be like stock photography in the age of SD.

Was just watching an interview of Billy Corgan (smashing pumpkins) on Rick Beato’s YouTube[1] last night where billy was lamenting the inevitable future where the “psychopaths” in the music biz will use ai and auto tune to churn out three chord non-music mumble rap for the youth of tomorrow, or something to that effect. It was funny because it’s the sad truth. It’s already here but new tech will allow them to cut costs even more, and increase their margins. No need for musicians. Really cool on one hand, in the same way fentanyl is cool — or the cotton gin, but a bit depressing on the other, if you care about musicians. I and a few others will always pay to go the symphony, so good players will find a way get paid, but this is what kids will listen to, because of the profit margin alone.[1]
<a href="https:&#x2F;&#x2F;m.youtube.com&#x2F;watch?v=nAfkxHcqWKI" rel="nofollow">https:&#x2F;&#x2F;m.youtube.com&#x2F;watch?v=nAfkxHcqWKI</a>

Sound is a lot higher fidelity, it&#x27;s harder to make the information available to a computer without serious downsampling or simplification.Consider sounds over 12khz. On a spectrogram during a chorus or drop that area is lit up, with so many things changing from millisecond to millisecond. A lot of AI samples really struggle at high frequencies, or even forgo them entirely.Midi based approaches have been really great though, and an approach like in the OP is fascinating (and impressive).

As someone who works in the audio AI space I think an underappreciated reason for audio lagging behind is that it&#x27;s a lot harder to put impressive audio clips in an academic paper than impressive images.

Can anyone confirm&#x2F;deny my theory that AI audio generation has been lagging behind progress in image generation because it’s way easier to get a billion labeled images than a billion labeled audio clips?

Damn this is insane. I wonder what other things can be encoded as images and generated with SD?

This question has been explored before, see Kologorov Music:
<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Qg3XOfioapI" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Qg3XOfioapI</a>

I was thinking about this - what if someone trained a stable diffusion type model on all of the worlds commercial music? This model would probably produce quite amazing music given enough prompting and I&#x27;m wondering if the music industry would be allowed to claim copyright on works created with such a model. Would it be illegal or is this just like a musician picking up ideas from hearing the world of music? Is it really right to make learning a crime, even if machines are doing it? I&#x27;m conflicted after finding out that for sync licensing the music industry want a percentage of revenue based on your subscription fees, sometimes as high as 15%-20%! I&#x27;m surprised such a huge fee isn&#x27;t considered some kind of protection racket.

A network trained on spectrograms only should do much better.

Perhaps you could do a hierarchical approach somehow, first generating a &quot;zoomed out&quot; structure, then copying parts of it into an otherwise unspecified picture to fill in the details.But perhaps plain stable diffusion wouldn&#x27;t work - you might need different neural networks trained on each &quot;zoom level&quot; because the structure would vary: music generally isn&#x27;t like fractals and doesn&#x27;t have exact self-similarity.

I think there has to be a better way to make long songs...For example, you could take half the previous spectrogram, shift it to the left, and then use the inpainting algorithm to make the next bit... Do that repeatedly, while smoothly adjusting the prompt, and I think you&#x27;d get pretty good results.And you could improve on this even more by having a non-linear time scale in the spectrograms. Have 75% of the image be linear, but the remaining 25% represent an exponentially downsampled version of history. That way, the model has access to what was happening seconds, minutes, and hours ago (although less detail for longer time periods ago).

Cant wait to see this in karaoke, you just sing lyrics and it jams along with music.

Interesting. I experimented a bit with the approach of using diffusion on whole audio files, but I ultimately discarded it in favor of generating various elements of music separately. I&#x27;m happy with the results of my project of composing melodies (<a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;playlist?list=PLoCzMRqh5SkFPG0-RIAR8jYRaICWubUdx" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;playlist?list=PLoCzMRqh5SkFPG0-RIAR8...</a>) and I still think this is the way to go and but that was before Stable Diffusion came out. These are interesting results though, maybe it can lead to something more.

I propose that while you are GPU limited, you make these changes:* Don&#x27;t do the alpha fade to the next prompt - just jump straight to alpha=1.0.* Pause the playback if the server hasn&#x27;t responded in time, rather than looping.

LOL. Yes we had to upgrade our Vercel tier: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;sethforsgren&#x2F;status&#x2F;1603425188401467392" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;sethforsgren&#x2F;status&#x2F;1603425188401467392</a>

I got an actual `HTTP 402: PAYMENT_REQUIRED` response (never seen one of those in the wild, according to Mozilla it is experimental). Someone&#x27;s credit card stopped scaling?

If it can do music, can we train better models for different kinds of music? Or different models for different instruments makes more sense? For different instruments we can get better resolution by making the spectrogram represent different frequency ranges. This is terribly exciting, what a time to be alive.

it seems that SD does cover everything in terms of generative ai. Speaking of music, very interesting paper and demo. Just wondering in terms of license and commercialization, what kind of mess are we expecting here?

This is one of the most ingenious thing I&#x27;ve seen in my life

I can’t help but see parallels to synesthesia. It’s amazing how capable these models are at encoding arbitrary domain knowledge as long as you can represent it visually w&#x2F; reasonable noise margins.

Very cool, but the music still has a very &quot;rough&quot;, almost abrasive tinge to it. My hunch, is that it has to do with the phase estimates being off.Who&#x27;s going to be first to take this approach and use it to generate human speech instead?

I think you&#x27;ll find plenty of people who find that DAWs and music theory help them better find self-expression and celebrate life through their music. Any tool or framework that opens up new modes of achieving that self-expression should be celebrated, not shunned because it isn&#x27;t as &quot;pure&quot; as more time and labor intensive methods. Would you rather someone be forced to dedicate a significant amount of time to studying music and art creation just to be able to find that self-expression?

Generated contents without filtering&#x2F;validation are worthless.I predict some kind of testing, validation or ranking be developed to filter out generated contents. Each domain has its own rules - you need to implement validation for code and math, fact checks for text, contrasting the results from multiple solutions for problem solving, and aesthetic scoring for art.But validation is probably harder than learning to generate in the first place, probably a situation similar to closing the last percent in self driving.

This is what I&#x27;ve been talking about all year. It is such a relief to see it actually happen.In summary: The search for AGI is dead. Intelligence was here and more general than we realized this whole time. Humans are not special as far as intelligence goes. Just look how often people predict that an AI cannot X or Y or Z. And then when an AI does one of those things they say, &quot;well it cannot A or B or C&quot;.What is next: This trend is going to accelerate as people realize that AI&#x27;s power isn&#x27;t in replacing human tasks with AI agents, but letting the AI operate in latent spaces and domains that we never even thought about trying.

Riffusion – Stable Diffusion fine-tuned to generate music