For me as a deaf person the current state of art (in terms of speed &amp; usability) is the Recorder app on a Google Pixel phone (4a&#x2F;6 Pro is what I&#x27;ve used)

I&#x27;m not in the Speech Recognition circles and am looking for open source speech recognition I can play around with - would this be the new state of the art?

They have an example in the post with a very thick Scottish accent. You should listen to it. It&#x27;s pretty impressive.

Looking forward to see if this works well with foreign accents

Are there any published benchmarks available outlining how this compares to other open source ASR software, such as Coqui.ai?

Interesting, I&#x27;m a non-native French speaker, the original French piece struck me as being entirely normal (but maybe it was just the perfect French accent that swayed me). Can you please point out what he said which wasn&#x27;t idiomatic or naturally-worded French?

I&#x27;m interested in building something with this to aid my own French learning. Would love to read your findings if you end up posting it somewhere like twitter&#x2F;blog!

The French version is a little contrived. The speaker is a native speaker, but the text is obviously the result of a translation from English to French, not idiomatic French.I will try to put the code to the test, see how it goes.

&gt;~97% accuracy over hour-long conversations. I&#x27;m sure it&#x27;s been an absolute godsend for law enforcement97% accuracy means roughly three or four errors per minute of speech. That seems potentially extremely problematic for something like law enforcement use where decisions with significant impact on people&#x27;s day and&#x2F;or life might be made on the basis of &quot;evidence&quot;.

Any recommendations for particular services?

That is an exciting possibility. Being able to fix bad setups and missed takes automagically. It’s always been possible, just expensive and time consuming for moderate improvements.

There&#x27;s already software that can imitate a person&#x27;s voice, so we have all the pieces already to do speech-to-text, clean up with GPT-3, and back to text-to-speech in the original person&#x27;s voice. Maybe with a style transfer to keep the person&#x27;s inflections etc the same?

I&#x27;ve not found that to be the case.For technical content, I use Rev.com and provide a glossary and real humans do the transcript. Other AI transcription services get lots wrong because the context often matters. Words like &quot;TCP&#x2F;IP&quot; or &quot;FAT disk format&quot; or &quot;Big Endian&quot; I&#x27;ve never found AI so far to handle well.I&#x27;m interested to test out whisper on this one.<a href="https:&#x2F;&#x2F;corecursive.com&#x2F;063-apple-2001&#x2F;" rel="nofollow">https:&#x2F;&#x2F;corecursive.com&#x2F;063-apple-2001&#x2F;</a>

Since you work on podcasts, do any open source transcription tools currently identity the speaker in the output? This would be particularly helpful for interviews.

I&#x27;m not sure if you&#x27;ve tried Descript, but their ML-based &quot;Studio Sound&quot; filter makes bad audio sound like it was recorded and edited nicely.

It was already better. I edit a podcast and have &gt; a decade of pro audio editing experience in the film industry, and I was already using a commercial AI transcription service to render the content to text and sometimes edit it as such (outputting edited audio).Existing (and affordable) offerings are so good that they can cope with shitty recordings off a phone speaker and maintain ~97% accuracy over hour-long conversations. I&#x27;m sure it&#x27;s been an absolute godsend for law enforcement other people who need to gather poor-quality audio at scale, though much less great for the targets of repressive authority.Having this fully open is a big deal though - now that level of transcription ability can be wrapped as an audio plugin and just used wherever. Given the parallel advances in resynthesis and understanding idiomatic speech, in a year or two I probably won&#x27;t need to cut out all those uuh like um y&#x27;know by hand ever again, and every recording can be given an noise reduction bath and come out sounding like it was recorded in a room full of soft furniture.

More of this is welcome, they should live up their name and original purpose and share other models (code, weights, dataset) in the open source community as well.

Make hay before Google gives away free hay.That said there is value in integration of this into other things.

Can&#x27;t wait to see twelve new $49.99&#x2F;mo speech parser services pop up in the next few weeks.

Some music is hard for even people to make out the lyrics to.

It seems far from good with mixed language content, especially with English and Japanese together. The timestamps are far from perfect. It&#x27;s far from perfect. It&#x27;s nowhere close to human for the more ambiguous translations that depend on context of word. It&#x27;s far below what anyone that spoke either language would consider acceptable. Maybe it&#x27;s unfair to use music, but music is the most realistic test of whether it&#x27;s superior to the average human.

&gt; Neat, <a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper</a> - they have open-sourced it, even the model weights, so they are living up to their name in this instance.Perhaps it will encourage people to add voice command to their apps, which can be sent to gpt3

Is the training dataset and code open too?

Neat, <a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper</a> - they have open-sourced it, even the model weights, so they are living up to their name in this instance.The 4 examples are stunningly good (the examples have speakers with heavy accents, speaking in foreign language, speaking with dynamic background noise, etc.), this is far and away better than anything else I&#x27;ve seen. Will be super curious to see other folks trying it out and seeing if it&#x27;s as robust as it seems, including when confronted with audio speech with natural tics and uhhh&#x27;s and uhmm&#x27;s and everything in-between.I think it&#x27;s fair to say that AI-transcription accuracy is now decidedly superior to the average human&#x27;s, what the implications of this are I&#x27;m not sure.

Personally, I would rather type than talk when interacting with a computer. The only time I use voice interfaces are when the physical interface is so poor it&#x27;s just easier to use voice. Apple TV devices are an example of this.

I’ve been experimenting with voice-interfaces where typing is replaced by talking, but I find it hard to transition users to voice - we ‘seem’ to prefer typing to talking.I wonder if this will change.

Also, based on their demo, this model seems like it might have comprehension well above the level of a typical human.Anyway, it&#x27;s out there now. No way to turn back.

I think it&#x27;s a new capability to add good speech to text, search, and models that can understand and process text.  You have microphones recording speech everywhere, models turning that speech into easily searchable text, and something like GPT-3 reading all the speech and raising red flags for any transgressive idea you please.

I&#x27;d argue that cheap, pervasive, always-on surveillance with a backlog of searchable transcriptions is a qualitatively different capability.

&quot;Secret listening machines everywhere&quot; was a pretty big thing in East Germany. It&#x27;s also the central theme of the movie The Lives of Others.Of course, the ability to scale this more cheaply (throwing more compute at it, instead of more people) is somewhat scary, but it&#x27;s not really introducing a new capability. Especially since you still have to do something with the transcript. An AirBnB landlord who reads the transcript of what you said could as well have listened to the recording.

“Good” for ensuring you’re a compliant consumer - bad if you’re an individual person

We will see an explosion of AI capabilities in the next couple of years. This will have a huge impact on our lives, much of it good but some of it also bad.

AI speech recognition FN scares the heck out of me...for so many reasons.But one that really pisses me off is not being able to turn it off on the iphone, and the fact that aside from &quot;hidden cameras in my airBnB&quot; -- soon we will have to worry about secret listening machines EVERYWHERE

It&#x27;s a micromachines ad from the &#x27;80s. He talked like that in all of them!As for speed, to a computer we don&#x27;t talk very fast, not even that guy.I wonder if it could handle Rap God by Eminem....Let&#x27;s find out!

That example at the top of the page (speed talking) blew me away. He started talking, I was stunned for a minute, then realised yes, it really was English, and I just burst out laughing.That&#x27;s so, so far beyond the previous state-of-the-art, it&#x27;s absurd.

Large is 3GB to save everyone a click. Tiny is 72MB.

(Model weights from <a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper&#x2F;blob&#x2F;main&#x2F;whisper&#x2F;__init__.py" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper&#x2F;blob&#x2F;main&#x2F;whisper&#x2F;__init__...</a> )&quot;tiny.en&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;d3dd57d32accea0b295c96e26691aa14d8822fac7d9d27d5dc00b4ca2826dd03&#x2F;tiny.en.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;d3dd5...</a>&quot;&quot;tiny&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9&#x2F;tiny.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;65147...</a>&quot;&quot;base.en&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;25a8566e1d0c1e2231d1c762132cd20e0f96a85d16145c3a00adf5d1ac670ead&#x2F;base.en.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;25a85...</a>&quot;&quot;base&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;ed3a0b6b1c0edf879ad9b11b1af5a0e6ab5db9205f891f668f8b0e6c6326e34e&#x2F;base.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;ed3a0...</a>&quot;&quot;small.en&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;f953ad0fd29cacd07d5a9eda5624af0f6bcf2258be67c92b79389873d91e0872&#x2F;small.en.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;f953a...</a>&quot;&quot;small&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;9ecf779972d90ba49c06d968637d720dd632c55bbf19d441fb42bf17a411e794&#x2F;small.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;9ecf7...</a>&quot;&quot;medium.en&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;d7440d1dc186f76616474e0ff0b3b6b879abc9d1a4926b7adfa41db2d497ab4f&#x2F;medium.en.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;d7440...</a>&quot;&quot;medium&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1&#x2F;medium.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;345ae...</a>&quot;&quot;large&quot;: &quot;<a href="https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;e4b87e7e0bf463eb8e6956e646f1e277e901512310def2c24bf0e11bd3c28e9a&#x2F;large.pt" rel="nofollow">https:&#x2F;&#x2F;openaipublic.azureedge.net&#x2F;main&#x2F;whisper&#x2F;models&#x2F;e4b87...</a>&quot;

&gt; I can understand not releasing GPT-3, even if I disagree with the decision.Why do you disagree?

True. The potential of GPT-3 to cause internet mayhem was&#x2F;is significant. I would argue that the mere act of announcing it was still a catalyst for an eventual GPT-3-like model being released. In revealing it, they established a target for what open source models could aim to achieve, and simultaneously got bad actors thinking about ways to abuse it.

&gt; This kind of model is harder to abuse, so I guess it passed their internal checks much more easily.The version I choose to believe: stability.ai ate DALL-E for lunch, and that woke them up.

This kind of model is harder to abuse, so I guess it passed their internal checks much more easily.I can understand not releasing GPT-3, even if I disagree with the decision.

&gt; It&#x27;s one model and in a non-strategic area where there are existing open source projects (Kaldi, DeepSpeech, ...).I can already tell this is much better than any of the existing open source projects with the exception of the wav2* sequence of projects and potentially nvidia&#x27;s nemo.

Yes. The same is true of many products from many companies.I feel bad about GPT-3 and DALL-E being released under the terms they were, but I don&#x27;t feel bad about this. I&#x27;m not going to condemn OpenAI for the good things they did, but I will hold them accountable for bad things or good ones they didn&#x27;t do.I&#x27;d given up on OpenAI being open or ethical, but this is a start. It took them down from &quot;evil super-villain&quot; status to mere villain.

It&#x27;s one model and in a non-strategic area where there are existing open source projects (Kaldi, DeepSpeech, ...).For a company that raised $1B, that&#x27;s not exactly living up to their name and original mission.

It seems like OpenAI are finally living up to their name for once with this release? Anything I&#x27;m missing?From what I can gather:1. Includes model weights. I can&#x27;t find the URL, but they reference them enough and have a CLI tool, so I presume I just haven&#x27;t found them yet.2. Includes code: <a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper</a>3. Released under MIT License: <a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper&#x2F;blob&#x2F;main&#x2F;LICENSE" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper&#x2F;blob&#x2F;main&#x2F;LICENSE</a>

I just tested it myself. Its fast enough on colab, couple of seconds but not sure if its fast enough to transcribe realtime audio yet.

Their models range from 70mb to 3gb. The largest model is smaller than the optimised stable diffusion. Not sure what the inference speed is like, haven&#x27;t tried it myself yet.

On my ancient desktop it happily fell back to running on CPU just fine.

Is there a list of system requirements somewhere ? Can it run on cheaper low memory GPUs ? maybe CPUs ?

My own experience agrees: the generally available &quot;SOTA&quot; models are not especially robust, and can be _extremely_ bad (&gt;50% absolute error rate) at some tasks. I&#x27;ll post some preliminary numbers in a sibling comment and look into running my full set of tests on Whisper.It looks like Whisper is probably leaving a lot of accuracy on the table, but initially it does seem to be a lot more robust than general &quot;SOTA&quot; models.For a quick comparison, Silero&#x27;s accuracy charts are kind of nice because they post results for a large variety of datasets. Scroll down to the EN V6 xlarge EE model (not the xlarge CE) [1][1] <a href="https:&#x2F;&#x2F;github.com&#x2F;snakers4&#x2F;silero-models&#x2F;wiki&#x2F;Quality-Benchmarks#en-v6" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;snakers4&#x2F;silero-models&#x2F;wiki&#x2F;Quality-Bench...</a>

One of the things they point out is that the SoTA on e.g. LibriSpeech is only good at LibriSpeech, and doesn&#x27;t generalise as well.&gt; Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper’s zero-shot performance across many diverse datasets we find it is much more robust and makes 50% fewer errors than those models.

It is interesting how they compare with wav2vec2 instead of nemo conformer (which is more accurate) in Table 2.

I&#x27;m looking forward to your comparison. It&#x27;s really hard to make sense of how good this model actually is without being an expert in the area.

Talon was the first thing that came to my mind when I saw this news. Would be nice if it could benefit from Whisper. (Big fan of your work on Talon!)

I suspect Whisper is more robust than other &quot;SOTA&quot; models, but this release is likely leaving a fair bit of accuracy on the table considering the amount of resources OpenAI is capable of throwing at training it.Comparing the readily available test sets from the paper to some of my personal robust models (for the Talon models, this is greedy decoding, no language model):<pre><code> Talon Talon Talon Whisper wav2vec 2.0
 28M 300M 1B Large 960h
 librispeech clean 3.21 2.52 2.40 2.7 2.7
 librispeech other 8.21 6.56 5.63 5.6 6.2
 common voice 13.88 11.65 8.86 9.5 29.9
 tedlium 7.51 6.55 5.47 4.0 10.5
</code></pre>
I have a battery of more difficult tests on hand (including adversarial tests, and diverse accent-specific metrics). I&#x27;ll look at running these tests on each of the Whisper model sizes and following up with a larger comparison.

Comparing this model&#x27;s word error rates to the state of the art [1] on a few common test sets:<pre><code> Whisper SoTA
 LibriSpeech test-clean 2.7% 1.8%
 LibriSpeech test-other 5.6% 2.9%
 Switchboard 13.1% 4.9%
 CallHome 15.8% 9.5%
</code></pre>
The authors do explicitly state that they&#x27;re trying to do a lot of fancy new stuff here, like be multilingual, rather than pursuing just accuracy.[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;syhw&#x2F;wer_are_we" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;syhw&#x2F;wer_are_we</a>

&gt; About a third of Whisper’s audio dataset is non-English, and it is alternately given the task of transcribing in the original language or translating to English. We find this approach is particularly effective at learning speech to text translation and outperforms the supervised SOTA on CoVoST2 to English translation zero-shot.That&#x27;s intriguing. You can just set the model to transcribe everything into English, no matter which language the speaker is using, and it just works. Given that many people are much better at understanding English than at speaking it, this might make voice interfaces much more accessible without much work.

Naively, training the same model on multiple languages has interesting implications.On one hand, it may capture something &quot;deeper&quot; about language.On the other hand, it&#x27;s likely to do great in general, but miss particularities of some language.Understanding the coverage of the training model seems a perennial problem. Is there any (shorthand) way to compare language model training corpora?Clearly if they use common subsets we have a literal comparison. I&#x27;m more interested in whether there&#x27;s progress in characterizing corpora by speech styles, fluency, vocabulary sets, (noise) environment, emotionality, proposition types, etc.(btw: 25 minutes for a 9-minute segment on a 12-thread x86. Lots of jargon spelled as it sounds. Sentences capitalized but no punctuation. Overall good.)

Siri until ios 15 was done in the cloud IIRC.

Good point about realtime or not, however with ML I have found the weaknesses get addressed pretty fast by someone. There is a big step between proof of concept and practical application though, so we shall see.

Someone else in this thread[0] said Whisper was running at 17x real time for them. So, even a weak machine might be able to do an acceptable approximation of real time with Whisper.Also, I feel like shipping to the cloud and back has been shown to be just as fast as on device transcription in a lot of scenarios. Doing it on device is primarily a benefit for privacy and offline, not necessarily latency. (Although, increasingly powerful smartphone hardware is starting to give the latency edge to local processing.)Siri&#x27;s dictation has had such terrible accuracy for me (an American English speaker without a particularly strong regional accent) and everyone else I know for so many years that it is just a joke in my family. Google and Microsoft have much higher accuracy in their models. The bar is so low for Siri that I automatically wonder how much Whisper is beating Siri in accuracy... because I assume it has to be better than that.I really wish there was an easy demo for Whisper that I could try out.[0]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=32928207" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=32928207</a>

Siri and Cortana have to run at least in real time, with reasonable compute resources. Probably faster than real time when the audio gets shipped off to the cloud and transcribed there. This model can&#x27;t do that (in the &quot;large&quot; version, which the examples use).Also, you are comparing Whisper&#x27;s highlight reel with everyday performance of other models. Nobody shows their weaknesses in their highlight reel.

I agree they have the best compared to Apple, Amazon, Microsoft. However I don&#x27;t think it is as good as what is being shown here by OpenAI.

I tried feeding the four examples from this announcement into Google as dictation inputs and it just sits there blankly. On the JFK speech test file in the repo, Google understands perfectly. The samples in the announcement are clearly outside the capabilities of anything Google has launched publicly, but I don&#x27;t know how that translates to overall utility in every day applications.

In my unmeasured empirical observation Google has amazing speech recognition

Is it? Googling suggests that Microsoft invested in OpenAI but doesn’t actually own it.

A lot of models we currently use seem to do the same thing. The model will transcribe a &quot;best effort&quot; interpretation in real time, then as you can continue speaking, you&#x27;ll see it go back and make corrections. I&#x27;m sure you can feed the first X seconds you have into the model, followed by (30-X) seconds of silence, and it will do real time transcription just fine... it would be weird if this broke anything. Then, as you get more speech, you continue getting better transcription of the first 30 seconds, then you switch to a 30 second sliding window.Maybe I&#x27;m missing something, but I don&#x27;t see the problem here.

This AI has a 30 second delay on the audio processing because it needs to be able to &quot;look into the future&quot; to get these good results. That 30s delay would be unacceptable for Siri&#x2F;Google&#x2F;Cortana.

How is it Apple, Google, or Microsoft are not further ahead of the game on speech recognition like this? They have the resources to hire the best ML researchers and throw tons of computing hours at it, yet Siri, Google, and Cortana continue to struggle to get anywhere near this level of comprehension.

Would a multilingual modal perhaps also be better at understanding non-natives speech?

It seems these days that language-oriented models are commonly becoming multilingual by default. There are a lot of common threads when understanding sentence construction between different languages. French and English have different rules but they will still have things like nouns, adjectives, subjects, prepositions, etc. It seems that by training models on many languages you get both a more robust understanding of language, and it saves you the trouble of having to make many more localized models for every language. I also believe that the other languages help the models construct sentences in languages which have very small training sets. If it has a few examples in a rare language as well as good translations to a better-known language, then it can provide good support for the rare language.We also see in image generation models that multi-modal networks are more powerful than single purpose networks. As we move towards more advanced AI systems I suspect we will see more and more generalizable networks with distinct advantages over separate networks that get plugged together.

My understanding is that multi-modal models are the primary focus of OpenAI right now, due to their stated goal of achieving AGI. This product is probably better thought of as an offshoot of their work to create a fully generalizable model, rather than a specific attempt to provide translation&#x2F;transcription services.

It sounds useful to me because you can use tone information to help with the translation, which text-to-text translation can&#x27;t do. But I&#x27;m not sure if that&#x27;s how this model actually works.

Judging from the chart in their github README, Whisper performs much better in parsing Spanish audio than any other language and that in particular blows my mind. I would have expected English to be at the top of any such model, it being such an IT lingua franca.Now I wonder if it works equally well with Spanish from Spain (and its different regions) and Spanish from the New World (and in its myriads of different flavours).

Hold on, it does not only speech recognition, but also language translation, in the same model?What an interesting approach. What benefits does this have over having two dedicated models, one for speech-to-text, and another for translation?It just seems so odd, given the problems of speech-to-text and Spanish-to-English seems so different from one another (in terms of the problem domain). Seems so unusual to have both handled by one model!Does knowledge of speech-to-text carry over into knowledge of translation? Does knowledge of translation carry over into knowledge of speech-to-text? So weird.

Represent a given set of audio inputs as a numeric vector, which can then for example be finetuned for other ML&#x2F;AI problems or placed in an embeddings database for easy ANN search with similar audio clips. In the extreme case it could facilitate better AI audio generation similar to how CLIP can guide a VQGAN.Although the 30 second minimum input is a bit of a bummer since it may not allow much granularity in the resulting embeddings.

What do you mean exactly by audio embeddings?

The model output can be tweaked to produce audio embeddings (akin to BERT for text embeddings and CLIP for image embeddings), which can lead to some interesting applications as the previous two examples have demonstrated.

Oh cool! Yeah I have stopped doing it lately as I was not really using them (I would like to use them for making rough notes for future youtube video scripts), though in general it does seem like good self care too even if I don&#x27;t review them. That said I just tried the base model on one of my voice logs and it was pretty good! Trying the medium model now and it seems basically perfect. So I will have to start doing these logs more!Anyway I am pretty terrible with email but short exchanges can work for me, or maybe we can connect over signal. Send me a message to my email in my profile and I would be happy to sync up!

I do this too, and I’ve built some software for it just for myself.I’d love to chat and hear about how you use this! My email is in my profile, or I’m @tekacs on Twitter (and everywhere). :)

Count me in!! Working on tools actually to turn these transcriptions into something more social

I do this too!  I have been doing it for about a year now, and haven&#x27;t ever run into someone else that does this kind of audio-journaling.  Would you be up for comparing notes sometime about how it is working out for you? I am finding that it is extremely effective form of self-care, but with lots of personal caveats.  I would be so interested to hear your experience.

I just tested it and it was pretty mediocre at least with my accent. I can definitely benefit from a decent app for quick note recording with a button press-&gt;transcribe-&gt;upload to gdrive&#x2F;good UI app for later grepping.

Is that application actually doing on-device transcription? Under &quot;Data safety&quot; on the Google Play page it says &quot;This app may share these data types with third parties: Audio&quot; which doesn&#x27;t exactly instill confidence that my audio will 100% always stay on my device. It also says &quot;Data is encrypted in transit&quot; but if data stays on the device, why it has to be &quot;encrypted in transit&quot;? There should be no transit at all.

Google&#x27;s recorder app is NOT available for most phones. Only Pixels and a couple of other selected handsets

Google&#x27;s recorder app for android will let you record audio files and make some transcriptions, right on the device.

I&#x27;ll probably explore using this, but I&#x27;ve used an app called Just Press Record to do what you say. Runs on Apple Watch too, so you can tap a complication at any time in the day, speak, and you get a transcript on your phone, etc.

Hey this looks great! I like to record audio notes while driving in my car after work, to kind of decompress my thoughts from the day. But I never go back and listen as they can be long and meandering. Sometimes in the audio log I will sum up my thoughts, but this might be 20 minutes in and hard to find. I really wish I had transcriptions so I could easily scan the full contents. I have tried Mozilla Deepspeech (I don&#x27;t want a cloud solution) and I was surprised to find that I could not get Deepspeech to reliably transcribe them. There is a bit of road noise, though I think for a human listener they are easy to understand. It looks like this one might actually do the trick!EDIT: Tried it and it worked great! It is very easy to use. I just did the pip install line in the readme and was ready to go. You literally just run the one pip install line, and then you run the program in the format &quot;whisper my_audio.wav&quot; and it goes. Really nice job OpenAI!

Both, the blog covers it in detail. Pass in audio in any language, and get an English transcription out.

It can do both - I&#x27;ve edited my original post to show the translation task.

Is it translation or transcription? Or both?Both, wow. This is really interesting.

Was this with the `base` model? `large` is running ok on a P100 in colab, but is about 4% the speed of `base.en`. Certainly seems like some of these models will be fast enough for real-time.

It also runs well on a CPU and seems to have proper memory management. Wonderful timing because I was using DeepSpeech for some audio recordings and it required me to script up a splitter to make the files into .wav and then do snippets of 10 seconds each. Everything about this just works out of the box. On a core i5 I&#x27;m getting about 30 seconds every minute. Transcriptionist jobs just turned into editor jobs. I love how it drops the inflections in the audio as well, because it was trained on transcription work, and that is one of the first things you learn to do (drop the uhs and ums and huhs etc, unless it is a strictly verbose transcription).

&gt; dans son ensemble&gt; in sound togetherThat&#x27;s hilarious and honestly, incredibly bad. &quot;Dans son ensemble&quot; is a very common idiom (meaning &quot;as a whole&quot;) while &quot;in sound together&quot; has to be pretty rare. &quot;Son&quot; means &quot;his&#x2F;hers&#x2F;its&quot; as well as &quot;sound&quot;, and the former meaning is probably more common in general so I have no idea how this result could arise.&quot;Termo&quot; also doesn&#x27;t exist in French, it&#x27;s &quot;thermo&quot;, so the transcript even makes orthographic errors.And I forgot about &quot;couplisser&quot; which is also a hilarious made-up word that sounds like it could mean something, but doesn&#x27;t! Edit Google finds exactly one reference of this, in a patent with a typo on the word &quot;coulisser&quot;.I&#x27;m still impressed by the transcript quality since it covers many languages, but the translation part is quite poor.

How did you get it to use the GPU?I have it running right now and it&#x27;s not touching the GPU.

I just tested the model [1] using an RTX3090, trying to translate a french text I found here [2].Some observations:- The full translation of the 6:22 minute video takes about 22 seconds (17x real time)- It recognizes the language by default (and did a good job to recognize it was french audio)- MIT License [3]!- The quality of the transcription is good, but not perfect.- The quality of the translation (if you don&#x27;t consider transcription errors as a translation error) is generally very good.---The transcription:&gt; Bonjour à tous, &lt;error&gt;j&#x27;suis&lt;&#x2F;error&gt; espère que vous allez bien, c&#x27;&#x27;est ENTI. Et aujourd&#x27;, &lt;error&gt;aujourd&#x27;,&lt;&#x2F;error&gt; on se retrouve &lt;error&gt;un peu physique&lt;&#x2F;error&gt; pour parler de la termo dynamique. Vous ne vous inquiétez pas, ça va bien se passer. On va y aller ensemble, &lt;error&gt;être à par exemple,&lt;&#x2F;error&gt; je vous accompagne à travers une série de vidéos pour vous expliquer les principes de base en termo dynamique. Et bah, c&#x27;&#x27;est parti, on va y aller tranquillement. Lidée, c&#x27;&#x27;est vous puissiez comprendre la termo dynamique dans son ensemble. Donc, je vais vraiment prendre mon temps pour &lt;error&gt;couplisser&lt;&#x2F;error&gt; bien comprendre les notions,The translation:&gt; Hello everyone, I hope you&#x27;re doing well, it&#x27;s NT and today we find ourselves a little physical to talk about the thermo dynamic. Don&#x27;t worry, it&#x27;s going well, we&#x27;re going to go together and be the same. I&#x27;m going to accompany you through a series of videos to explain the basic principles in thermo dynamic. Well, let&#x27;s go, &lt;error&gt;we&#x27;re going to go quietly&lt;&#x2F;error&gt;. The idea is that you can understand the thermo dynamic &lt;error&gt;in sound together&lt;&#x2F;error&gt;. So I&#x27;m really going to take my time to understand the notions,---All in all very happy that OpenAI is publishing their models. If Stable Diffusion is any guide, people will hack some crazy things with this.[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper</a>
[2] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=OFLt-KL0K7Y" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=OFLt-KL0K7Y</a>
[3] <a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper&#x2F;blob&#x2F;main&#x2F;LICENSE" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper&#x2F;blob&#x2F;main&#x2F;LICENSE</a>

This is even slightly more direct: access to WSJ data requires paying LDC for the download, and the pricing varies depending on what institution &#x2F; license you&#x27;re from. The cost may be a drop in the bucket compared to compute, but I don&#x27;t know that these licenses are transferable to the end product. We might be a couple court cases away from finding out but I wouldn&#x27;t want to be inviting one of those cases :)

I think it might be even less problematic with something like Whisper than with DALLE&#x2F;SD? Merely consuming data to train a system or create an index is not usually contrary to the law (otherwise Google wouldn&#x27;t exist) – it&#x27;s the publication of copyright content that&#x27;s thorny (and is something you can begin to achieve with results from visual models that include Getty Photos logo, etc.)I think it&#x27;d be a lot harder to make a case for an accurate audio to text transcription being seen to violate the copyright of any of the training material in the way a visual could.

&gt; models learning from data does not make the output of the models a derivative work of that dataMost of the debate seems to be happening on the question of whether everything produced by models trained on copyrighted work represents a derivative work. I argue that at the very least some of it does; so the claim said to be made by the AI companies (see quote above) is clearly a false one.We&#x27;re in a weird place now where AI is able to generate &quot;near verbatim&quot; work in a lot of cases, but I don&#x27;t see an obvious case for treating this any differently than a human reproducing IP with slight modifications. (I am not a lawyer.)For example, copyright law currently prevents you from selling a T-shirt with the character Spider-Man on it. But plenty of AI models can give you excellent depictions of Spider-Man that you could put on a T-shirt and try to sell. It&#x27;s quite silly to think that any judge is going to take you seriously when you argue that your model, which was trained on a dataset that included pictures of Spider-Man, and was then asked to output images using &quot;Spider-Man&quot; as a search term, has magically circumvented copyright law.(I think there&#x27;s a valid question about whether models represent &quot;derivative work&quot; in the GPL sense specifically, but I&#x27;m using the idea more generally here.)

This is a big dispute right now: OpenAI and other AI companies generally take the position that models learning from data does not make the output of the models a derivative work of that data.  For example, GitHub Co-pilot uses all publicly available GitHub code regardless of license, and DALLE-2&#x2F;StableDiffusion&#x2F;etc use lots of non-free images.  I don&#x27;t think this has been challenged in court yet, and I&#x27;m very curious to see what happens when it is.

I think they didn&#x27;t use WSJ for training, only for evaluation. Paper includes WSJ under &quot;Evaluation datasets&quot;

Are there any AI&#x2F;ML models that don&#x27;t use sketchy licensed datasets? Everything seems to be &quot;downloaded from the internet, no license&quot; or more explicitly proprietary. The only exception I can think of would be coqui&#x2F;DeepSpeech?

Be wary of using this model - the licensing of this model seems sketchy. Several of the datasets used for training like WSJ and TED-LIUM have clear non-commercial clauses. I&#x27;m not a lawyer but releasing a model as &quot;MIT&quot; seems dubious, and hopefully OpenAI has paid for the appropriate licenses during training as they are no longer a research-only non profit.

pretty sure such a tool made HN front page a few months ago

Does this work with multiple speakers?I want to build a tool that takes a video and generates subtitles for it, then I want to index the subtitles and let people search for a specific quote to scrub to that part of the video using automatically generated urls.This is for a specific fandom of a ton of content, lots of dirty audio mostly recorded in a gym setting with multiple people speaking.

I&#x27;d love to find a way to test this with longer audio but I don&#x27;t have GPU resources and not exactly sure how to load that into the Colab. Is anyone planning on hosting or sharing a model that can be used by others to test longer form audio (for podcast transcription)?

It&#x27;s actually better than Google Meet subtitle system.

Combine the translation + transcription with voice synthesis, and once compute power allows for this to be miniaturized we will be able to have babel-fish technology in real life.

Whisper performed a lot better than I would&#x27;ve expected it to!

Did respectably with some mumble rap: <a href="https:&#x2F;&#x2F;controlc.com&#x2F;d353dafb" rel="nofollow">https:&#x2F;&#x2F;controlc.com&#x2F;d353dafb</a>(some NSFW words in the lyrics obv)

Oh nice - I have an immediate use case for this. This looks accessible enough that the sci-fi dream of instantaneous audio translation is suddenly within reach.

This is so cool! I was just speaking to a non-technical family member about privacy concerns around using &quot;OK Google&quot; and the like. They responded inquiring about &quot;private&quot; alternatives, to which my answer was &quot;I&#x27;m not aware of good ones that give you that level of accuracy and convenience.&quot;Perhaps this development along with continued optimization and device compute power increases will lead us into a near-future where things like Mycroft devices and cellphones could have local-only speech-to-text and translation capabilities which are accurate even with environmental background noise variations encountered IRL.Great work OpenAI team!

You could already do the speech recognition in a fully open source way with vosk easily, although Whisper may be more accurate

You still interested in this? I&#x27;d be keen to chat to you, worked on a searchable transcript provider for educational youtube videos (likewise, unfortunately pre-whisper, so I did a lot of work with sentence completion perplexity and rpunct to try and improve transcript quality from youtube automatic transcriptions). Can be contacted at revision.ai and demo what we were able to do till now, would be great to hear your thoughts.

I really wish I had this about half a year ago when I was building a tool to automatically turn online school lectures into searchable, clickable transcripts (kind of like YouTube or EdX transcripts).I was originally using Adobe Premiere Pro&#x27;s speech to text to do it, and wrote Python to convert its output to the Hyperaudio format on GitHub. With this, I can totally skip all of that step and this is fully open source, too.App idea:Build an app that takes a video and uses Hyperaudio or a similar project to add a clickable and searchable transcript (clicking in transcript seeks video)

I know a manual transcription company, which is still seeing modest growth from existing clients who also use ASR, so it&#x27;s not quite there yet

It would be exceptional to get a healthy competitor to microsoft&#x2F;nuance&#x27;s dragon monopoly on voice recognition in healthcare.  At a couple thousand bucks a license and the more recent SaaS subscription trend there is a lot of money to be made in that space.

Exactly what I was planning to do! Want to share yours with me?

I just wrote a script with Hazel to automatically transcribe my voice notes to txt. It handles punctuation extremely well. What a wonderful contribution!

DeepSpeech got spun out of Mozilla to coqui.ai and they are continuing the open nature of the project.

Oh this is a relief to have something opensource in this field. I had using Mozilla Deepspeech for transcribing my voice notes , often with hilarious to incomprehensible results.  DeepSpeech is dead ; so I will be sure to check this out.

Great to see OpenAI finally being open :)

I&#x27;m really new to pip, but does this look ok?(after running the command for setuptools)
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in &#x2F;Users&#x2F;xxx&#x2F;Library&#x2F;Python&#x2F;3.9&#x2F;lib&#x2F;python&#x2F;site-packages (22.2.2)
Requirement already satisfied: setuptools in &#x2F;Users&#x2F;xxx&#x2F;Library&#x2F;Python&#x2F;3.9&#x2F;lib&#x2F;python&#x2F;site-packages (65.3.0)----
after trying whisper installation:
 × Getting requirements to build wheel did not run successfully.
 │ exit code: 1
 ╰─&gt; [20 lines of output]
 Traceback (most recent call last):
 File &quot;&#x2F;Users&#x2F;xxx&#x2F;Library&#x2F;Python&#x2F;3.9&#x2F;lib&#x2F;python&#x2F;site-packages&#x2F;pip&#x2F;_vendor&#x2F;pep517&#x2F;in_process&#x2F;_in_process.py&quot;, line 363, in &lt;module&gt;
 main()
 File &quot;&#x2F;Users&#x2F;xxx&#x2F;Library&#x2F;Python&#x2F;3.9&#x2F;lib&#x2F;python&#x2F;site-packages&#x2F;pip&#x2F;_vendor&#x2F;pep517&#x2F;in_process&#x2F;_in_process.py&quot;, line 345, in main
 json_out[&#x27;return_val&#x27;] = hook(*hook_input[&#x27;kwargs&#x27;])
 File &quot;&#x2F;Users&#x2F;xxx&#x2F;Library&#x2F;Python&#x2F;3.9&#x2F;lib&#x2F;python&#x2F;site-packages&#x2F;pip&#x2F;_vendor&#x2F;pep517&#x2F;in_process&#x2F;_in_process.py&quot;, line 130, in get_requires_for_build_wheel
 return hook(config_settings)
 File &quot;&#x2F;Library&#x2F;Developer&#x2F;CommandLineTools&#x2F;Library&#x2F;Frameworks&#x2F;Python3.framework&#x2F;Versions&#x2F;3.9&#x2F;lib&#x2F;python3.9&#x2F;site-packages&#x2F;setuptools&#x2F;build_meta.py&quot;, line 154, in get_requires_for_build_wheel
 return self._get_build_requires(
 File &quot;&#x2F;Library&#x2F;Developer&#x2F;CommandLineTools&#x2F;Library&#x2F;Frameworks&#x2F;Python3.framework&#x2F;Versions&#x2F;3.9&#x2F;lib&#x2F;python3.9&#x2F;site-packages&#x2F;setuptools&#x2F;build_meta.py&quot;, line 135, in _get_build_requires
 self.run_setup()
 File &quot;&#x2F;Library&#x2F;Developer&#x2F;CommandLineTools&#x2F;Library&#x2F;Frameworks&#x2F;Python3.framework&#x2F;Versions&#x2F;3.9&#x2F;lib&#x2F;python3.9&#x2F;site-packages&#x2F;setuptools&#x2F;build_meta.py&quot;, line 150, in run_setup
 exec(compile(code, __file__, &#x27;exec&#x27;), locals())
 File &quot;setup.py&quot;, line 2, in &lt;module&gt;
 from setuptools_rust import Binding, RustExtension
 File &quot;&#x2F;private&#x2F;var&#x2F;folders&#x2F;lj&#x2F;7x6d3dxd3cbdtt484k6xsmyh0000gn&#x2F;T&#x2F;pip-build-env-ieaydl8r&#x2F;overlay&#x2F;lib&#x2F;python3.9&#x2F;site-packages&#x2F;setuptools_rust&#x2F;__init__.py&quot;, line 1, in &lt;module&gt;
 from .build import build_rust
 File &quot;&#x2F;private&#x2F;var&#x2F;folders&#x2F;lj&#x2F;7x6d3dxd3cbdtt484k6xsmyh0000gn&#x2F;T&#x2F;pip-build-env-ieaydl8r&#x2F;overlay&#x2F;lib&#x2F;python3.9&#x2F;site-packages&#x2F;setuptools_rust&#x2F;build.py&quot;, line 23, in &lt;module&gt;
 from setuptools.command.build import build as CommandBuild # type: ignore[import]
 ModuleNotFoundError: No module named &#x27;setuptools.command.build&#x27;
 [end of output]<pre><code> note: This error originates from a subprocess, and is likely not a problem with pip.</code></pre>
error: subprocess-exited-with-error

Yep, I had this too. `pip3 install -U pip setuptools` took care of it. (If you get an error about pip3, try `pip` instead)

There&#x27;s a --device flag you can pass. I&#x27;ve been trying to get `--device cuda` to work on my Windows machine and it&#x27;s saying that torch wasn&#x27;t compiled with CUDA. Trying to figure out what&#x27;s going on there.And on the M1, supposedly PyTorch has support for hardware acceleration using MPS (Metal Performance Shaders, announced here <a href="https:&#x2F;&#x2F;pytorch.org&#x2F;blog&#x2F;introducing-accelerated-pytorch-training-on-mac&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pytorch.org&#x2F;blog&#x2F;introducing-accelerated-pytorch-tra...</a>) but when I tried `--device mps` it blew up with an error &quot;input types &#x27;tensor&lt;1x1280x3000xf16&gt;&#x27; and &#x27;tensor&lt;1xf32&gt;&#x27; are not broadcast compatible&quot;.

I&#x27;m still not successfully using the GPU, but it&#x27;s working decently quickly (with the base model - it&#x27;s incredibly slow to use the Large model) using just the CPU. I&#x27;m going to have to check what magic stable-diffusion is doing to enable the GPU :(

Probably need to pass some kind of options when initializing. The command itself works fine, just shows a warning: warnings.warn(&quot;FP16 is not supported on CPU; using FP32 instead&quot;)

I got requirements installed, but then when running the Python example, I get:RuntimeError: &quot;slow_conv2d_cpu&quot; not implemented for &#x27;Half&#x27;

I got it working inside a docker container on my M1 MBP. FWIW, I&#x27;m having my $180 tinyminimicro PC run a translation task while my M1 MBP runs a transcription task with the same audio input. So far, the PC is actually outputting results a lot faster than the MBP. Interesting results.

Anyone get it running on m1 mac?I keep getting `ModuleNotFoundError: No module named &#x27;setuptools.command.build&#x27;`

Hoping to see this out to use in open source voice assistants, eg. mycroft

Wow that is incredibly impressive. At 0:53 is it translating as well? Didn&#x27;t sound like English to me.

Detected language: english[00:00.000 --&gt; 00:05.400] Gordy and County Kerry are investigating the theft of up to 60 sheep on Mount Brandon.[00:05.400 --&gt; 00:10.400] One of the farmers is offering a reward for information leading to the return of the use,[00:10.400 --&gt; 00:12.200] which are worth thousands of euro.[00:12.200 --&gt; 00:14.200] Well, I&#x27;m fine with that.[00:14.200 --&gt; 00:15.200] That&#x27;s right.[00:15.200 --&gt; 00:16.200] Do you own them?[00:16.200 --&gt; 00:17.200] Anyone can say it.[00:17.200 --&gt; 00:18.200] Fine with that.[00:18.200 --&gt; 00:22.720] Last Saturday, Mikey Joe O&#x27;Shea brought his flock of Scotch sheep down from the mountain[00:22.720 --&gt; 00:25.320] commonage ahead of lambing.[00:25.320 --&gt; 00:29.840] He discovered over 50 were missing, allowing for a number of deaths and[00:29.840 --&gt; 00:30.840] strays.[00:30.840 --&gt; 00:34.600] Mikey is convinced over 45 sheep have been stolen.[00:34.600 --&gt; 00:35.600] It was a good night.[00:35.600 --&gt; 00:36.600] It would be a full moon there.[00:36.600 --&gt; 00:37.600] It would be a good night.[00:37.600 --&gt; 00:38.600] It would be bright out.[00:38.600 --&gt; 00:40.600] There could be anyone going up in the mountains.[00:40.600 --&gt; 00:41.600] It would be a good night.[00:41.600 --&gt; 00:43.600] Well, that was 45 sheep missing.[00:43.600 --&gt; 00:49.600] Mikey and the lambs and everything in the sheep, they counted out a nice bit of money.[00:49.600 --&gt; 00:52.200] They&#x27;ve been doing the boat in Nassan.[00:52.200 --&gt; 00:53.200] It&#x27;s a big one.
[00:53.200 --&gt; 00:54.200] It&#x27;s a big one.
[00:54.200 --&gt; 00:55.200] It&#x27;s a big one.[00:55.200 --&gt; 00:59.000] Mikey&#x27;s next door neighbor says some of his sheep have also been stolen.[00:59.000 --&gt; 01:00.000] Come back.
[01:00.000 --&gt; 01:01.000] Come back.
[01:01.000 --&gt; 01:02.000] Come back.[01:02.000 --&gt; 01:03.000] I&#x27;ve been missing about 10 years.[01:03.000 --&gt; 01:04.000] It&#x27;s not all that difficult.[01:04.000 --&gt; 01:06.320] All they&#x27;ve got to do is have a good dog.[01:06.320 --&gt; 01:10.560] Have a good dog and go at night, some moonshine night.[01:10.560 --&gt; 01:11.560] Just put the dog around him.[01:11.560 --&gt; 01:14.120] Put him on a trailer and walk him.[01:14.120 --&gt; 01:18.360] And then probably somebody else to pick him up.[01:18.360 --&gt; 01:29.960] Everybody&#x27;s doing it north, but he&#x27;s doing it.

Are you sure? I just ran some of Kimmy&#x27;s sketches through it and ... The results are garbage.

Their Scottish accent example is pretty good, I&#x27;d like to see it work on some very strong English accents like this one: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=nJ7QB3om-QY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=nJ7QB3om-QY</a>

Measuring performance in rounds of successful Chinese whisper(irony)

Given this, are there good (and available&#x2F;open source) models for text to speech? Last time I tried everything still sounded extremely robotic, and&#x2F;or were a pain to set up and run. It would be fun to set up a pipeline where the two processes &#x27;communicate&#x27;.

I knew there was a reason why I kept my MP3 library even after subscribing to Spotify. Now piping everything through whisper. So far the generated lyrics are reasonable, though it thinks the REM song says &quot;Linnie Bruce is not afraid.&quot;No surprise that it appears to have successfully transcribed all the recordings of Harvard Sentences I could find. <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Harvard_sentences" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Harvard_sentences</a>

CUDA worked fine with large on my 2080Ti FWIW. The speedup is ridiculous, as expected. My Ryzen 3800X used almost an hour transcribing a minute worth of speech, while the 2080Ti does it in like 10-20 seconds.

This should, in theory, work with CUDA; my GPU doesn&#x27;t have enough RAM to do it (it runs out at 2.9GiB allocated, I have 4GiB, but am running a compositing desktop, which chews up about 600MiB; not sure where the other ~400MiB went)[edit]I confirmed CUDA worked with the &quot;small&quot; model, which used 3.3GB of GPU ram, and resulted in much poorer recognition than the &quot;medium&quot; model on my CPU (but it ran at least two orders of magnitude faster).<pre><code> {
 description = &quot;Python 3.9 development environment&quot;;
 outputs = { self, nixpkgs }:
 let
 system = &quot;x86_64-linux&quot;;
 pkgs = import nixpkgs {
 inherit system;
 config.allowUnfree = true;
 config.cudaSupport = true;
 };
 in {
 devShells.${system}.default = pkgs.mkShell {
 buildInputs = with pkgs; [
 cudatoolkit linuxPackages.nvidia_x11
 cudaPackages.cudnn
 libGLU libGL
 xorg.libXi xorg.libXmu freeglut
 xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib 
 ncurses5 stdenv.cc binutils
 ffmpeg
 python39
 python39Packages.pip
 python39Packages.numpy
 python39Packages.pytorch-bin
 python39Packages.virtualenv
 ];

 shellHook = &#x27;&#x27;
 export LD_LIBRARY_PATH=&quot;${pkgs.linuxPackages.nvidia_x11}&#x2F;lib&quot;
 &#x27;&#x27;; 
 };
 };
 }</code></pre>

For those on NixOS, here&#x27;s a quick and dirty flake.nix that will let you make a venv in which to &quot;pip install&quot;&#x27;Just put it in a flake.nix, and &quot;nix develop&quot; followed by &quot;virtualenv .&#x2F;venv; . .&#x2F;venv&#x2F;bin&#x2F;activate; pip install git+<a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper.git" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper.git</a>&quot;<pre><code> {
 description = &quot;Python 3.9 development environment&quot;;

 outputs = { self, nixpkgs }:
 let
 system = &quot;x86_64-linux&quot;;
 pkgs = import nixpkgs { inherit system; };
 in {
 devShells.${system}.default = pkgs.mkShell {
 buildInputs = [
 pkgs.ffmpeg
 pkgs.python39
 pkgs.python39Packages.pip
 pkgs.python39Packages.numpy
 pkgs.python39Packages.pytorch
 pkgs.python39Packages.virtualenv
 ];
 };
 };
 }</code></pre>

Rev AI will also create a transcription separated by multiple speakers, which it doesn&#x27;t appear Whisper can do (yet). I expect that Whisper will overtake the alternatives soon, given that it&#x27;s open source, but today it&#x27;s not there yet.

&gt; you can also get speaker labels with existing open source software.Hello Nickolay :)Diarization has always been the hard part for me, especially since it is very difficult to do comparisons within your domain. The evaluation metrics are not descriptive enough imo.Would you say Titanet or EcapaTDNN are decent for use in production alongside, say, Whisper, or any other ASR output, if given the timestamps, so as to bypass running VAD? I&#x27;m just about to run experiments to try pyannote&#x27;s diarization model and google&#x27;s uis-rnn to test out how well they work, but it&#x27;s a tad beyond my ability to evaluate.I also wonder if Whisper architecture would be good for generating embeddings, but I feel it&#x27;s focused so much on what is said rather than how it&#x27;s said that it might not transfer over well to speaker tasks.

You can apply public punctation model from Vosk on top of Kaldi output, you can also get speaker labels with existing open source software.On quick video transcription test this model is more accurate than AssemblyAI and Rev AI. It will be harder for them to sell pure ASR now. Some more business-oriented applications will still be important though, for example ASR as part of callcenter analytics solution or as a part of medical ERP system.The value of automatic summarization is small, without AI it is very hard to make it right, you need to be an expert in the field to understand what is important.

Any opinions on what this means for speech-to-text companies like rev.ai and assmembly.ai ?We&#x27;ve tested open source solutions for s2t, like kaldi, but the quality was not good enough. However, one of the main advantages of a service like assembly.ai to me was that they offer sentence splitting in form of punctuation and speaker detection, which Kaldi does not.So I guess I answered my own question to some degree: A S2T service is more than just S2T. We already see assembly.ai add more and more features (like summarisation, PID redaction ect.) that are a value-add to plain S2T.Still, curious to hear what your take on that is.

Really incredible to see that their multilingual audio-to-English approach is viable. I&#x27;m super excited about this, and great to see that openai actually open up about something, for once.Skimming the codebase I can&#x27;t immediately see code to do additional training.Being able to fine-tune the model to a specific language or case (eg. teach it specifically about some technical topic that might not be so prevalent in the current train set) would be majorly disruptive to current SOTA in &quot;callcenter analytics&quot; tech. Especially when combining Whisper with GPT3.

Ran it on Juicy by The Notorious B.I.G and results were considerably worse than my mix of prog-rock and british invasion music I had tried before, though at least some of that is due to the number of proper-nouns in that song.It took about 1000 CPU-minutes for this 5 minute song on my Ryzen 2700 with 12 OpenMP threads (about 100 minutes wall-clock).

For what it&#x27;s worth, even the large model balks on Easy (Aesop Rock), eg.&quot;Fountainheads spittle sniglets quicker than quidditch seekers snatch golden snitches.&quot;becomes&quot;Stirred up out mids bittles, snicklets, cricket and quidditch seekers net golden snitches.&quot;¯\_(ツ)_&#x2F;¯

Ran a few other songs through it and found one obvious mistranscription:&quot;He&#x27;s the bedroom cosmic rocker&quot; (should be &quot;He&#x27;s the veteran cosmic rocker&quot; in Veteran Cosmic Rocker by The Moody Blues)I also noticed that it&#x27;s a little on the conservative side for detecting speech; all songs were missing at least part of one line.

I just threw a random rock MP3 at it, and a first readthrough shows no transcription errors; this is quite good.Now I just want OCR that&#x27;s even 50% as good as this...

Sorry, I don&#x27;t have anything available.One item I remember was that I said &quot;Dr Kemeny&quot; in relation to Dartmouth College (he was a famous mathematician, invented the BASIC programming language and was president of the college). It replaced those instances with &quot;Jack Kennedy&quot;.In another instance, I said that &quot;Evidently, you have a reading comprehension problem.&quot;. It replaced it with &quot;Evidently, I have a ...&quot;, completely reversing the meaning.There was zero problems with the microphones or audio, and it was not rushed or mumbled talk. There were 80+ other examples over a few hours of talking, and some from other speakers. And those were just the obvious ones I could catch.Another massive problem with this technology is that a human stenographer can notice when s&#x2F;he missed something and didn&#x27;t hear and ask the speaker to repeat or clarify what was said, and will often during a pause request clarification on spelling of names, addresses, etc. In contrast, this &quot;AI&quot; technology just barges ahead ASSuming that it knows what it is doing and inserts literally whatever sounds good in the transcript, completely silent that it doesn&#x27;t have a clue.Having seen this up close, I&#x27;m of the strong opinion that anyone foisting this software on the market without huge warnings that this is not usable for any critical functions is, basically a fraud. They know or certainly should know that these failures not only exist but are common and systemic, yet they barge along like it is OK. It is not.

Do you have a demo audio clip for this? I&#x27;d be interested to see how it looks in practice.

&gt;&gt;Current &quot;AI&quot; algorithms are fundamentally flawed because they rely on a statistical approach.YES! The old joke about &quot;Artificial Stupidity&quot; is actually more true than anyone realized.These statistical so-called-AI systems actually work to actively REMOVE or sanitize out any unexpected information, making it all conform with the EXPECTED results from the training set.This not only REMOVES the most high-information &#x27;surprising&#x27; or unexpected nuggets, it actively HIDES them. When something unexpected comes up, it gets force fit into the expected prediction algorithms and output as if it were good.I&#x27;m not saying that there are no useful things that can be done with this technology — there is a LOT of mundane work out there to be done.But, we will never get this type of &quot;AI&quot; saying &quot;Huh, that&#x27;s odd, I wonder why that is?&quot;, which is exactly the kind of observation that leads a prepared and fertile mind to great discoveries.

I&#x27;ve been saying this for years. Current &quot;AI&quot; algorithm are fundamentally flawed because they rely on a statistical approach. This works moderately well for some use cases but it will rarely give you 100% confidence.
Good luck with self-flying planes or self-running nuclear power plants.

Like every model I&#x27;ve seen there is something like this:&gt;&gt;A decoder is trained to predict the corresponding text...Prediction of expected text in the context of the previous text.While this is valuable in casual transcription, it can be extremely dangerous in serious contexts.From personal experience, having given a deposition with an &quot;AI&quot; transcription, it will literally reverse the meanings of sentences.This is because it produces the EXPECTED output in a context, and NOT THE ACTUAL OUTPUT.Like a speaker that clips the output, these types of systems &#x27;clip&#x27; the really valuable information out of a transcription. Worse yet, this is a completely silent failure, as the transcript LOOKS really good.Basic info theory shows that there is more information contained in &#x27;surprising&#x27; chunks of data than in expected ones. These systems actively work to substitute &#x27;expected&#x27; speech to overwrite &#x27;surprising&#x27; speech.The transcript I got was utter trash, multiple pages of errata I had to submit when the normal is a couple of lines. And as I said, some literally reversed the meaning in a consequential way, and yet completely silently.This kind of silent active failure mode is terrifying. Unless it is solved, and I see no way to solve it without removing ALL predictive algos from the system, these types of systems must not be used in any situation of serious consequence, at least not without real redundancy and backup.

&gt; goffifor 2), it&#x27;s actually written in the description: &quot;phrase-level timestamps&quot;, so it should be possible (phrase level is neat for skipping to a special location on a video, but maybe not for audio editing).

You properly mentioned timestamps. There are many other important properties of good ASR system like vocabulary adaptability (if you can introduce new words) or streaming. Or confidences. Or latency of the output. Compared to Vosk models this model can not work in streaming manner, so not very suitable for real-time applications.But in general the model is robust and accurate and trained on the amount of speech we never dreamed about in Vosk. We will certainly benefit from this model as a teacher (together with others like gigaspeech models). I recently wrote about it <a href="https:&#x2F;&#x2F;alphacephei.com&#x2F;nsh&#x2F;2022&#x2F;06&#x2F;14&#x2F;voting.html" rel="nofollow">https:&#x2F;&#x2F;alphacephei.com&#x2F;nsh&#x2F;2022&#x2F;06&#x2F;14&#x2F;voting.html</a>

Really interesting, I can see ton of potential uses.2 questions:1) how does it compare to state of the art FOSS solutions? I&#x27;m seeking about DeepSpeech or Vosk2) would it be somehow possible to associate timestamp to the words recognized? That would be amazing for things such as audio editing or skipping to a particular location on a video

Anyone know if it is possible to output IPA using this?International Phonetic Alphabet (IPA)- <a href="https:&#x2F;&#x2F;wikipedia.org&#x2F;wiki&#x2F;International_Phonetic_Alphabet" rel="nofollow">https:&#x2F;&#x2F;wikipedia.org&#x2F;wiki&#x2F;International_Phonetic_Alphabet</a>_________EDIT: Based on list of languages in the tokenizer code here, doesn’t appear IPA is supported:<a href="https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper&#x2F;blob&#x2F;5f8d4bcc254d4f3e833d353b4cd4454004f5b51a&#x2F;whisper&#x2F;tokenizer.py" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;openai&#x2F;whisper&#x2F;blob&#x2F;5f8d4bcc254d4f3e833d3...</a>

Looks like it defaults to the model called &quot;small&quot;.I just ran some benchmarks - M1 Max, pytorch, with a 1.29 second flac (looks like the matrix math was running on a single thread):<pre><code> tiny
 146.522ms detect_lang
 549.131ms decode_one
 0.057ms tokenizer

 base
 354.885ms detect_lang
 1046.679ms decode_one
 0.011ms tokenizer

 small
 803.892ms detect_lang
 3194.503ms decode_one
 0.017ms tokenizer

 medium
 2279.689ms detect_lang
 10128.255ms decode_one
 0.023ms tokenizer

 large
 3656.478ms detect_lang
 17249.024ms decode_one
 0.016ms tokenizer</code></pre>

For more benchmarks on an rtx 2060 (6gb), the &quot;small&quot; model for me is roughly 10x real-time and the tiny model is 30x real-time.

Okay this is super impressive. I just downloaded Whisper and fed it a random flac file I had handy and it did a really good job. Also impressive that it works on my weak CPU:A 3m07s flac took 5m to transcribe:<pre><code> $ whisper --device cpu &#x27;BLACKPINK - BORN PINK&#x2F;01 Pink Venom.flac&#x27;
 Detecting language using up to the first 30 seconds. Use `--language` to specify the language
 Detected language: korean
 [00:00.000 --&gt; 00:10.000] Blackpink
 [00:11.000 --&gt; 00:14.000] Kick in the door, wave in the coco
 [00:14.000 --&gt; 00:16.000] 팝콘이는 친게 껴들 생각 말고
 [00:16.000 --&gt; 00:19.000] I talk to talk, run ways I walk walk
 [00:19.000 --&gt; 00:21.000] 힘 감고 팝 팝 안 봐도 척
 [00:21.000 --&gt; 00:24.000] By one and two by two
 [00:24.000 --&gt; 00:26.000] 내 손끝 두 하나에 타면 아지은 중
 [00:26.000 --&gt; 00:30.000] 갓 자쇼 지금 화려해 T makes no sense
 [00:30.000 --&gt; 00:32.000] You couldn&#x27;t get a dollar out of me
 [00:33.000 --&gt; 00:38.000] 자 오늘 밤이야 눈톱을 품고
 [00:38.000 --&gt; 00:41.000] 미혼을 뺏음 down
 [00:41.000 --&gt; 00:43.000] Look what you made us do
 [00:43.000 --&gt; 00:47.000] 천천히 널 잠재울 파이어
 [00:48.000 --&gt; 00:52.000] 잠이 날 만큼 아름다워
 [00:52.000 --&gt; 00:53.000] I bring the pain like
 [00:53.000 --&gt; 00:57.000] 디스탑, 팽팽, 디스탑, 팽팽, 디스탑, 팽팽, 팽팽
 [00:57.000 --&gt; 00:58.000] Get em, get em, get em
 [00:58.000 --&gt; 01:00.000] Straight till you don&#x27;t like
 [01:00.000 --&gt; 01:01.000] Whoa, whoa, whoa
 [01:01.000 --&gt; 01:03.000] Straight till you don&#x27;t like
 [01:03.000 --&gt; 01:04.000] Ah, ah, ah
 [01:04.000 --&gt; 01:05.000] Taste that, pink venom
 [01:05.000 --&gt; 01:06.000] Taste that, pink venom
 [01:06.000 --&gt; 01:08.000] Taste that, pink venom
 [01:08.000 --&gt; 01:09.000] Get em, get em, get em
 [01:09.000 --&gt; 01:11.000] Straight till you don&#x27;t like
 [01:11.000 --&gt; 01:12.000] Whoa, whoa, whoa
 [01:12.000 --&gt; 01:13.000] Straight till you don&#x27;t like
 [01:13.000 --&gt; 01:14.000] Ah, ah, ah
 [01:14.000 --&gt; 01:15.000] Blackpink and Amo
 [01:15.000 --&gt; 01:17.000] Got it by the smack ram
 [01:17.000 --&gt; 01:18.000] But rest in peace
 [01:18.000 --&gt; 01:19.000] Please light up a candle
 [01:19.000 --&gt; 01:20.000] This the knife of a vando
 [01:20.000 --&gt; 01:22.000] Messed up and I&#x27;m still in saline
 …SNIP…</code></pre>

Was wondering the same.I really wish I would have been paying attention in Unix class...Something like `microphone | chunk 3s | whisper | stdout` would be SO COOL!!! I think that&#x27;s possible but too lazy to look more.

Would also like to know this. It looks like they&#x27;re processing the audio file in 30 second chunks, so a naive approach of keeping a buffer of 30-second input stream chunks and just continually writing to an output .mp3 could work...

How can I use this (or something similar) for live translation? I don&#x27;t mind if there&#x27;s a 30s delay.As in I don&#x27;t want to input a file, I want to input the microphone sound.

Given how robust it seems to be with fast speech, I wonder if you could save cycles by speeding up the audio before feeding it in.

Good to see them releasing model weights - hopefully now that Stable Diffusion is out they will release Dall-E 2 source and weights as well.

Maybe, a number of speech recognition engines have been integrated into <a href="https:&#x2F;&#x2F;github.com&#x2F;dictation-toolbox&#x2F;dragonfly" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dictation-toolbox&#x2F;dragonfly</a>

If the Whisper models provide any benefits over the existing Talon models, and if it&#x27;s possible to achieve any kind of reasonable interactive performance, I will likely integrate Whisper models into Talon.Talon&#x27;s speech engine backend is modular, with Dragon, Vosk, the WebSpeech API, and Talon&#x27;s own engine all used in different ways by users.

Is it feasible to use this for Talon-like voice-driven computer usage?

True. The test should only be done on the material released after the model.

No transcripts, no. And recent episodes, within the past couple of weeks, so probably not part of the training either.

Did these podcasts have transcripts? You might be inadvertently evaluating it on data that it was trained on, which is basically cheating. Even if not, it might be trained on similar podcasts. Judging how good these kinds of models are is really hard.

Just tested this on some developer podcasts which usually fail hard given they&#x27;re full of technical jargon, brand names, etc. Whisper is a revolution! It&#x27;s picking up terms like Heroku, DigitalOcean, GitHub, ECS, AWS, etc. and capitalizing properly - something nothing else did unless you provided a whole pile of guiding vocabulary.

How well does it do for technical and domain oriented speech? For example I have audio recordings of a senior explaining some very technical aspects of our software. Will it understand the technical terms in that speech?I guess I will need to download and run on it to see how correct it is.

As far as TTS goes, Mycroft.ai[0] has released a decent offline one.[0]<a href="https:&#x2F;&#x2F;mycroft.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;mycroft.ai&#x2F;</a>

This is only one side of the coin, you still need really good models for Speech Synthesis and then be able to have it all working in almost real time, ideally locally on device.

I&#x27;d be interested to see how well it performs on something like an RPi. M1 is pretty beefy.

For an offline (non-streaming) model, 1x realtime is actually kind of bad, because you need to wait for the audio to be available before you can start processing it. So if you wait 10 seconds for someone to finish speaking, you won&#x27;t have the result until 10 seconds after that.You could use really small chunk sizes and process them in a streaming fashion, but that would impact accuracy, as you&#x27;re significantly limiting available context.

To be more precise the original comment said &quot;M1 Max&quot; which in itself is significantly beefier a bare &quot;M1&quot;

I really want all this too. The smallest model is ~80mb and the largest is 3gb. Not sure about system requirements yet; but models that small suggest this may be doable locally on a single board computer.Edit: According to this comment[0] the base model runs in real time on an M1 CPU. The tiny model apparently decodes an audio file twice as fast. These are promising results.[0] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=32927360#32929739" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=32927360#32929739</a>

hah, of course someone had the idea already and executed on it. But yeah, basically that but without the screen (probably would go a long way to decrease the cost, $299 is pretty steep for such a device)

Are you thinking about reimplementing Mycroft?The Mycroft has done a lot of cool and important work in the field to ship an actual personal assistant product (stuff like wake word detection).

This really makes me want to build a Amazon Echo&#x2F;Google Nest&#x2F;etc replacement that&#x27;s open hardware, open source and most importantly recognises voice completely offline. I find that I don&#x27;t use these smart devices for much more than setting timers anyway so this seems like an easy project.I just wonder what system requirements Whisper has and whether there are open source voice recognition models that are specifically built for embedded devices.

Depends if you&#x27;re trying to run it offline or over the cloud.

If your family uses Apple devices, Apple offers free on-device speech recognition. Only caveat is that it needs to be restarted every minute due to whatever stupid limitation (or bug) they&#x27;ve introduced.<a href="https:&#x2F;&#x2F;developer.apple.com&#x2F;documentation&#x2F;speech&#x2F;recognizing_speech_in_live_audio" rel="nofollow">https:&#x2F;&#x2F;developer.apple.com&#x2F;documentation&#x2F;speech&#x2F;recognizing...</a>Also, see `requiresOnDeviceRecognition`

Out of interest, did you try Nemo? <a href="https:&#x2F;&#x2F;github.com&#x2F;NVIDIA&#x2F;NeMo" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;NVIDIA&#x2F;NeMo</a>

It might require too much work for what you are looking for, but the wav2letter library is the best real-time transcription OSS I have found by a considerable margin.

Try <a href="https:&#x2F;&#x2F;github.com&#x2F;alphacep&#x2F;vosk-api&#x2F;blob&#x2F;master&#x2F;csharp&#x2F;demo&#x2F;VoskDemo.cs#L10" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;alphacep&#x2F;vosk-api&#x2F;blob&#x2F;master&#x2F;csharp&#x2F;demo...</a>

The base model seems to run faster than real time on my machine. The “medium” model is larger and runs more slowly - roughly real time or maybe slightly slower.

I tried it out and it&#x27;s way too slow on my machine that is no slouch (Ryzen 9 5950&#x2F;GTX 3080).It&#x27;s doing seconds of translation per minute for me at least.

thuttinger posted in this thread: <a href="https:&#x2F;&#x2F;github.com&#x2F;tobiashuttinger&#x2F;openai-whisper-realtime" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;tobiashuttinger&#x2F;openai-whisper-realtime</a>

Can this be used as a real-time transcription or is it too slow for that?Curious what anyone is using these days for a real-time transcription. It doesn&#x27;t have to be perfect, but just good enough.My kids watch some youtube vidoes where people will make a mod where it converts them talking to text then look for keywords and spawn a boss in Terraria if you say the wrong keyword etc.I made a clone of that with the .NET System.Speech.Recognition library. It... works.. but my biggest problem is that #1 it waits until you are done speaking to translate to text on the callback, so there was too much of a delay for it to be fun.. the point is that it will be checking a stream of chatter. #2 is the recognition is pretty crap, I mean it&#x27;s nearly good enough for my silly purpose but it&#x27;s still pretty bad.

This is dropping right in the middle of Interspeech 2022.I don’t believe OpenAI has anyone presenting at the conference, so presumably this was timed to coincide with that and get buzz at the conference.Curious how this model compares with foss STT from the startup Coqui.

I&#x27;m thinking of releasing a plugin in for Unity to that can be used to  match a phrase to an action. Seeing Whisper is making me think I should include a way to use voice and not just text.

I seem to have worked around it by tweaking the &quot;options&quot; line from the sample code to this:&gt;&gt;&gt; options = whisper.DecodingOptions(fp16=False)

It seems the cmdline script is smart enough to switch over automatically, but invoking it from python just fails if the correct option isn&#x27;t set

I am running on work laptop not using GPU. (I&#x27;m running in docker). I just get<pre><code> warnings.warn(&quot;FP16 is not supported on CPU; using FP32 instead&quot;)
</code></pre>
And it works.

I know this isn&#x27;t a tech support forum but maybe someone here knows. I&#x27;m attempting the sample python code from the github and almost get a transcription running on my work laptop without a GPU, but I run into this error message:&gt;&gt;&gt; result = whisper.decode(model, mel, options)Traceback (most recent call last):[snip]RuntimeError: &quot;slow_conv2d_cpu&quot; not implemented for &#x27;Half&#x27;It looks like a Torch error, is there some twiddling with &quot;options&quot; I can do to get it to run?

This is absolute garbage python as I am neither a python developer, nor a good developer. I was trying to play around with real time transcriptions. However, it does work!&gt; * recording
* done recording
Recording saved to file.wav
Press enter to transcribe&#x2F;Users&#x2F;laptop&#x2F;Development&#x2F;Personal&#x2F;Public&#x2F;pythonProject1&#x2F;venv&#x2F;lib&#x2F;python3.9&#x2F;site-packages&#x2F;whisper&#x2F;transcribe.py:70: UserWarning: FP16 is not supported on CPU; using FP32 instead
 warnings.warn(&quot;FP16 is not supported on CPU; using FP32 instead&quot;)
Detected language: english
 Goodbye, I need to go pick up my wife.
Press enter to start recordingAny improvements welcome here.```
# This is a sample Python script.# Press ⌃R to execute it or replace it with your code.
# Press Double ⇧ to search everywhere for classes, files, tool windows, actions, and settings.def print_hi(name):
 # Use a breakpoint in the code line below to debug your script.
 print(f&#x27;Hi, {name}&#x27;) # Press ⌘F8 to toggle the breakpoint.def record_microphone(seconds):
 import pyaudio
 import wave<pre><code> CHUNK = 1024
 FORMAT = pyaudio.paInt16
 CHANNELS = 1
 RATE = 44100
 RECORD_SECONDS = seconds
 WAVE_OUTPUT_FILENAME = &quot;file.wav&quot;

 p = pyaudio.PyAudio()

 stream = p.open(format=FORMAT,
 channels=CHANNELS,
 rate=RATE,
 input=True,
 frames_per_buffer=CHUNK)

 print(&quot;* recording&quot;)

 frames = []

 for i in range(0, int(RATE &#x2F; CHUNK * RECORD_SECONDS)):
 data = stream.read(CHUNK)
 frames.append(data)

 print(&quot;* done recording&quot;)

 stream.stop_stream()
 stream.close()
 p.terminate()

 wf = wave.open(WAVE_OUTPUT_FILENAME, &#x27;wb&#x27;)
 wf.setnchannels(CHANNELS)
 wf.setsampwidth(p.get_sample_size(FORMAT))
 wf.setframerate(RATE)
 wf.writeframes(b&#x27;&#x27;.join(frames))
 wf.close()

 return WAVE_OUTPUT_FILENAME



</code></pre>
if __name__ == &#x27;__main__&#x27;:
 seconds = 5
 while True:
 print(&quot;Press enter to start recording&quot;)
 input()
 filename = record_microphone(seconds)
 print(&quot;Recording saved to &quot; + filename)
 print(&quot;Press enter to transcribe&quot;)
 input()
 import whisper
 model = whisper.load_model(&quot;base&quot;)<pre><code> result = model.transcribe(filename)
 print(result[&quot;text&quot;])

</code></pre>
```

Hmm are there any noteworthy open sourced speech to speech models? Like transform a spoken line to another voice, copying both the words spoken and the inflections?

Whisper – open source speech recognition by OpenAI