I would assume this is 1 request containing 2k pages vs N requests whose total pages add up to 1000.

That makes sense. Idle time is nearly free after all.

If only business people I work with would understand 100GB even transfer over the network is not going to return immediately results ;)

Batching likely means the response is not real-time. You set up a batch job and they send you the results later.

Batched often means a higher latency option (minutes&#x2F;hours instead of seconds), which providers can schedule more efficiently on their GPUs.

Usually (With OpenAI, I haven&#x27;t checked Mistral yet) it means an async api rather than a sync api.e.g. you submit multiple requests (pdfs) in one call, and get back an id for the batch. You then can check on the status of that batch and get the results for everything when done.It lets them use their available hardware to it&#x27;s full capacity much better.

You can check the example code on the Mistral documentation, you would _only_ have to change the value of the variable `document_url` to the URL of your uploaded PDF... and you need to change the `MISTRAL_API_KEY` to the value of your specific key that you can get from the Le Platforme webpage.<a href="https:&#x2F;&#x2F;docs.mistral.ai&#x2F;capabilities&#x2F;document&#x2F;#ocr-with-pdf" rel="nofollow">https:&#x2F;&#x2F;docs.mistral.ai&#x2F;capabilities&#x2F;document&#x2F;#ocr-with-pdf</a>

May I ask as a layperson, how would you about using this to OCR multiple hundreds of pages?
I tried the chat but it pretty much stops after the 2nd page.

I thought we stopped -ly company names ~8 years ago?

From my testing so far, it seems it&#x27;s super fast and responded synchronously. But it decided that the entire page is an image and returned `![img-0.jpeg](img-0.jpeg)` with coordinates in the metadata for the image, which is the entire page.Our tool, doctly.ai is much slower and async, but much more accurate and gets you the content itself as an markdown.

Dang. Super fast and significantly more accurate than google, Claude and others.Pricing : $1&#x2F;1000 pages, or per 2k pages if “batched”. I’m not sure what batching means in this case: multiple pdfs? Why not split them to halve the cost?Anyway this looks great at pdf to markdown.

Not really. When you go 60 mph (or km&#x2F;h) you don&#x27;t specify the 1.00 for the hours either. pages&#x2F;$ is the unit, 1000 is the value.

Ya, presumably it is missing the number `1.00`.

Yeah you can read it as &quot;pages per dollar&quot; or as a unit &quot;pages&#x2F;$&quot;, it all comes out the same meaning.

Dollar signs go before the number, not after it like units.  It needs to be 1000 pages&#x2F;$1 to make sense, whereas 10km and 10h and 10&#x2F;h all make sense so 10km&#x2F;h does.  I imagine you would be confused by km&#x2F;h 10 but not $10.

Great example of how information is sometimes compartmentalized arbitrarily in the brain: I imagine you have never been confused by sentences such as “I’m running at 10 km&#x2F;h”.

They say: &quot;releasing the API mistral-ocr-latest at 1000 pages &#x2F; $&quot;I had to reread that a few times. I assume this means 1000pg&#x2F;$1 but I&#x27;m still not sure about it.

It outperforms the competition significantly AND can extract embedded images from the text. I really like LLMs for OCR more and more. Gemini was already pretty good at it

6 years ago I was working with a very large enterprise that was struggling to solve this problem, trying to scan millions of arbitrary forms and documents per month to 
clearly understand key points like account numbers, names and addresses, policy numbers, phone numbers, embedded images or scribbled notes, and also draw relationships between these values on a given form, or even across forms.I wasn&#x27;t there to solve that specific problem but it was connected to what we were doing so it was fascinating to hear that team talk through all the things they&#x27;d tried, from brute-force training on templates (didn&#x27;t scale as they had too many kinds of forms) to every vendor solution under the sun (none worked quite as advertised on their data)..I have to imagine this is a problem shared by so many companies.

Mathpix is ace. That’s the best results I got so far for scientific papers and reports. It understands the layout of complex documents very well, it’s quite impressive. Equations are perfect, figures extraction works well.There are a few annoying issues, but overall I am very happy with it.

You can get bounding boxes from our pdf api at Mathpix.comDisclaimer, I’m the founder

Just tested with a multilingual (bidi) English&#x2F;Hebrew document.The Hebrew output had no correspondence to the text whatsoever (in context, there was an English translation, and the Hebrew produced was a back-translation of that).Their benchmark results are impressive, don&#x27;t get me wrong. But I&#x27;m a little disappointed. I often read multilingual document scans in the humanities. Multilingual (and esp. bidi) OCR is challenging, and I&#x27;m always looking for a better solution for a side-project I&#x27;m working on (fixpdfs.com).Also, I thought OCR implied that you could get bounding boxes for text (and reconstruct a text layer on a scan, for example). Am I wrong, or is this term just overloaded, now?

hence my comment about privacy and need for legislation :)It isn&#x27;t the tech that&#x27;s the problem but the people that will abuse it.

Good lord, I dearly hope not. That sounds like a coddled hellscape world, something you&#x27;d see made fun of in Disney&#x27;s Wall-E.

&gt; I wondered how soon will I be able to ask the video player &quot;What am I looking at here, describe the equations&quot; and it will OCR the frames, analyze them and explain them to me.Seems like <a href="https:&#x2F;&#x2F;aiscreenshot.app" rel="nofollow">https:&#x2F;&#x2F;aiscreenshot.app</a> might fit the bill.

Yeah, all these things are possible today, but getting them well polished and integrated is another story. Imagine all this being supported by &quot;HTML6&quot; lol. When apple gets around to making this part of safari, then we know it&#x27;s ready.

Now? OK, you need to screencap and upload to LLM, but that&#x27;s well established tech by now. (Where by &quot;well established&quot;, I mean at least 9 months old ;)Same goes for &quot;navigating HTTP sites via LLM prompts&quot;. Most LLMs have web search integration, and the &quot;Deep Research&quot; variants do more complex navigation.Video chat is there partially, as well. It doesn&#x27;t really pay much attention to gestures &amp; expressions, but I&#x27;d put the &quot;earliest possible&quot; threshold for that a good chunk closer than 5 years.

I was just watching a science-related video containing math equations. I wondered how soon will I be able to ask the video player &quot;What am I looking at here, describe the equations&quot; and it will OCR the frames, analyze them and explain them to me.It&#x27;s only a matter of time before &quot;browsing&quot; means navigating HTTP sites via LLM prompts. although, I think it is critical that LLM input should NOT be restricted to verbal cues. Not everyone is an extrovert that longs to hear the sound of their own voices. A lot of human communication is non-verbal.Once we get over the privacy implications (and I do believe this can only be done by worldwide legislative efforts), I can imagine looking at a &quot;website&quot; or video, and my expressions, mannerisms and gestures will be considered prompts.At least that is what I imagine the tech would evolve into in 5+ years.

If you know the font in advance (which you often do in these cases) you can do insane reconstructions. Also keep in mind that it doesn&#x27;t have to be a perfect match, with the help of the color and other facts (such as likely location) about the car you can narrow it down significantly.

Maybe if you had multiple frames, and used something very clever?

There are photo enhancers online. But your picture is way too pixelated to get any useful info from it.

Looks like a paper temp tag. Other than that, I&#x27;m not sure much can be had from it.

Finding the right subreddit and asking there is probably a better approach if you want to maximize the chances of getting the plate &#x27;decrypted&#x27;.

Its in CA, looks like paper plates which follow a specific format and the last two seem to be the numbers &#x27;64&#x27;. Police should be able to search for temp tag with partial match and match the make&#x2F;model. Was curious to see if any software could help though

To even get started on this you&#x27;d also need to share some contextual information like continent, country etc. I&#x27;d say.

If it’s a video, sharing a few frames can help as well

Bit unrelated but is there anything that can help with really low resolution text? My neighbor got hit and run the other day for example, and I&#x27;ve been trying every tool I can to make out some of the letters&#x2F;numbers on the plate<a href="https:&#x2F;&#x2F;ibb.co&#x2F;mr8QSYnj" rel="nofollow">https:&#x2F;&#x2F;ibb.co&#x2F;mr8QSYnj</a>

* Character recognition on monolingual text in a narrow domain is solved

One of my hobby projects while in University was to do OCR on book scans. Doing character recognition was solved, but finding the relationship between characters was very difficult. I tried &quot;primitive&quot; neural nets, but edge cases would often break what I built. Super cool to me to see such an order of magnitude in improvement here.Does it do hand written notes and annotations? What about meta information like highlighting? I am also curious if LLMs will get better because more access to information if it can be effectively extracted from PDFs.

I wonder how it compares to USPS workers at deciphering illegible handwriting.

Document processing is where b2b SAAS is at.

Not OP, but it sounds like the kind of project I’d undertake.Happiness for me is about exploring the problem within constraints and the satisfaction of building the solution. Brittleness is often of less concern than the fun factor.And some kinds of brittleness can be managed&#x2F;solved, which adds to the fun.

You&#x27;ll be happier finding a replacement meter that has an interface to monitor it directly or a second meter. An old phone and OCR will be very brittle.

What software-tools do you usw to Programm the APP?

4o transcribes it perfectly. You can usually root an old Android and write this app in ~2h with LLMs if unfamiliar. The hard part will be maintaining camera lens cleanliness and alignment etc.The time cost is so low that you should give it a gander. You&#x27;ll be surprised how fast you can do it. If you just take screenshots every minute it should suffice.

I use this for a watermeter. Works quite well as long as you have a good SD card

Wow. I was looking at hooking my water meter into home assistant, and was going to investigate just counting an optical pulse (it has a white portion on the gear that is in a certain spot every .1 gal) This is like the same meter I use, and perfect.(It turns out my electric meter, though analog, blasts out it&#x27;s reading on RF every 10 seconds unencrypted. I got that via my RTL-SDR reciever :) )

This[1] is something I&#x27;ve come across but not had a chance to play with, designed for reading non-smart meters that might work for you. I&#x27;m not sure if there&#x27;s any way to run it on an old phone though.[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;jomjol&#x2F;AI-on-the-edge-device">https:&#x2F;&#x2F;github.com&#x2F;jomjol&#x2F;AI-on-the-edge-device</a>

<a href="https:&#x2F;&#x2F;www.home-assistant.io&#x2F;integrations&#x2F;seven_segments&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.home-assistant.io&#x2F;integrations&#x2F;seven_segments&#x2F;</a><a href="https:&#x2F;&#x2F;www.unix-ag.uni-kl.de&#x2F;~auerswal&#x2F;ssocr&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.unix-ag.uni-kl.de&#x2F;~auerswal&#x2F;ssocr&#x2F;</a><a href="https:&#x2F;&#x2F;github.com&#x2F;tesseract-ocr&#x2F;tesseract">https:&#x2F;&#x2F;github.com&#x2F;tesseract-ocr&#x2F;tesseract</a><a href="https:&#x2F;&#x2F;community.home-assistant.io&#x2F;t&#x2F;ocr-on-camera-image-for-water-electricity-meter&#x2F;124548" rel="nofollow">https:&#x2F;&#x2F;community.home-assistant.io&#x2F;t&#x2F;ocr-on-camera-image-fo...</a><a href="https:&#x2F;&#x2F;www.google.com&#x2F;search?q=home+assistant+ocr+integration" rel="nofollow">https:&#x2F;&#x2F;www.google.com&#x2F;search?q=home+assistant+ocr+integrati...</a><a href="https:&#x2F;&#x2F;www.google.com&#x2F;search?q=esphome+ocr+sensor" rel="nofollow">https:&#x2F;&#x2F;www.google.com&#x2F;search?q=esphome+ocr+sensor</a><a href="https:&#x2F;&#x2F;hackaday.com&#x2F;2021&#x2F;02&#x2F;07&#x2F;an-esp-will-read-your-meter-for-you&#x2F;" rel="nofollow">https:&#x2F;&#x2F;hackaday.com&#x2F;2021&#x2F;02&#x2F;07&#x2F;an-esp-will-read-your-meter-...</a>...start digging around and you&#x27;ll likely find something. HA has integrations which can support writing to InfluxDB (local for sure, and you can probably configure it for a remote influxdb).You&#x27;re looking at 1xRaspberry PI, 1xUSB Webcam, 1x&quot;Power Management &#x2F; humidity management &#x2F; waterproof electrical box&quot; to stuff it into, and then either YOLO and DIY to shoot over to your influxdb, or set up a Home Assistant and &quot;attach&quot; your frankenbox as some sort of &quot;sensor&quot; or &quot;integration&quot; which spits out metrics and yadayada...

Related, does anyone know of an app that can read gauges from an image and log the number to influx? I have a solar power meter in my crawlspace, it is inconvenient to go down there. I want to point an old phone at it and log it so I can check it easily. The gauge is digital and looks like this:<a href="https:&#x2F;&#x2F;www.pvh2o.com&#x2F;solarShed&#x2F;firstPower.jpg" rel="nofollow">https:&#x2F;&#x2F;www.pvh2o.com&#x2F;solarShed&#x2F;firstPower.jpg</a>

<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;echo840&#x2F;ocrbench-leaderboard" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;echo840&#x2F;ocrbench-leaderboard</a>

Came here to see if you all had run a benchmark on it yet :)

Update: Just ran our benchmark on the Mistral model and results are.. surprisingly bad?Mistral OCR:- 72.2% accuracy- $1&#x2F;1000 pages- 5.42s &#x2F; pageWhich is pretty far cry from the 95% accuracy they were advertising from their private benchmark. The biggest thing I noticed is how it skips anything it classifies as an image&#x2F;figure. So charts, infographics, some tables, etc. all get lifted out and returned as [image](image_002). Compared to the other VLMs that are able to interpret those images into a text representation.<a href="https:&#x2F;&#x2F;github.com&#x2F;getomni-ai&#x2F;benchmark">https:&#x2F;&#x2F;github.com&#x2F;getomni-ai&#x2F;benchmark</a><a href="https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;getomni-ai&#x2F;ocr-benchmark" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;getomni-ai&#x2F;ocr-benchmark</a><a href="https:&#x2F;&#x2F;getomni.ai&#x2F;ocr-benchmark">https:&#x2F;&#x2F;getomni.ai&#x2F;ocr-benchmark</a>

We published this benchmark the other week. We&#x27;ll can update and run with Mistral today!<a href="https:&#x2F;&#x2F;github.com&#x2F;getomni-ai&#x2F;benchmark">https:&#x2F;&#x2F;github.com&#x2F;getomni-ai&#x2F;benchmark</a>

It’s interesting that none of the existing models can decode a Scrabble board screen shot and give an accurate grid of characters.I realize it’s not a common business case, came across it testing how well LLMs can solve simple games. On a side note, if you bypass OCR and give models a text layout of a board standard LLMs cannot solve Scrabble boards but the thinking models usually can.

Its Mistral, they are the only homegrown AI Europe has, so people pretend they are meaningful.I&#x27;ll give it a try, but I&#x27;m not holding my breath. I&#x27;m a huge AI Enthusiast and I&#x27;ve yet to be impressed with anything they&#x27;ve put out.

&quot;World&#x27;s best OCR model&quot; - that is quite a statement. Are there any well-known benchmarks for OCR software?

Simon Willison linked to an impressive demo of Qwen2-VL in this area: I haven&#x27;t found a version of it that I could run locally yet to corroborate. <a href="https:&#x2F;&#x2F;simonwillison.net&#x2F;2024&#x2F;Sep&#x2F;4&#x2F;qwen2-vl&#x2F;" rel="nofollow">https:&#x2F;&#x2F;simonwillison.net&#x2F;2024&#x2F;Sep&#x2F;4&#x2F;qwen2-vl&#x2F;</a>

Is there a reliable handwriting OCR benchmark out there (updated, not a blog post)? Despite the gains claimed for printed text, I found (anecdotally) that trying to use Mistral OCR on my messy cursive handwriting to be much less accurate than GPT-4o, in the ballpark of 30% wrong vs closer to 5% wrong for GPT-4o.Edit: answered in another post: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;echo840&#x2F;ocrbench-leaderboard" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;echo840&#x2F;ocrbench-leaderboard</a>

I actually think you&#x27;re onto something there. The &quot;MicroLLMs Architecture&quot; could mirror how microservices revolutionized web architecture.Instead of one massive model trying to do everything, you&#x27;d have specialized models for OCR, code generation, image understanding, etc. Then a &quot;router LLM&quot; would direct queries to the appropriate specialized model and synthesize responses.The efficiency gains could be substantial - why run a 1T parameter model when your query just needs a lightweight OCR specialist? You could dynamically load only what you need.The challenge would be in the communication protocol between models and managing the complexity. We&#x27;d need something like a &quot;prompt bus&quot; for inter-model communication with standardized inputs&#x2F;outputs.Has anyone here started building infrastructure for this kind of model orchestration yet? This feels like it could be the Kubernetes moment for AI systems.

We’ll just stick LLM Gateway LLM in front of all the specialized LLMs. MicroLLMs Architecture.

&gt; Has anyone tried this on specialized domains like medical or legal documents?I’ll try it on a whole bunch of scientific papers ASAP. Quite excited about this.

What do you mean by &quot;free&quot;? Using the OpenAI vision API, for example, for OCR is quite a bit more expensive than $1&#x2F;1k pages.

Also interesting to see that parts of the training infrastructure to create frontier models is itself being monetized.

I think for regulated fields &#x2F; high impact fields $1&#x2F;1000 is well-worth the price; if the accuracy is close to 100% this is way better than using people, who are still error-prone

I&#x27;d love to try it for my domain (regulation), but $1&#x2F;1000 pages is significantly more expensive than my current local Docling based setup that already does a great job of processing PDF&#x27;s for my needs.

I have done OCR on leases. It’s hard. You have to be accurate and they all have bespoke formatting.It would almost be easier to switch everyone to a common format and spell out important entities (names, numbers) multiple times similar to how cheques do.The utility of the system really depends on the makeup of that last 5%. If problematic documents are consistently predictable, it’s possible to do a second pass with humans. But if they’re random, then you have to do every doc with humans and it doesn’t save you any time.

As someone who has had a home built, and nearly all my friends and acquaintances report the same thing, having a 1% error on information in this business would mean not a 10x but a 50x improvement over the current practice in the field.If nobody is supervising building documents all the time during the process, every house would be a pile of rubbish. And even when you do stuff stills creeps in and has to be redone, often more than once.

At my client we want to provide an AI that can retrieve relevant information from documentation (home building business, documents detail how to install a solar panel or a shower, etc) and we&#x27;ve set up an entire system with benchmarks, agents, etc, yet the bottleneck is OCR!We have millions and millions of pages of documents and an off by 1 % error means it compounds with the AI&#x27;s own error, which compounds with documentation itself being incorrect at times, which leads it all to be not production ready (and indeed the project has never been released), not even close.We simply cannot afford to give our customers incorrect informatiinWe have set up a backoffice app that when users have questions, it sends it to our workers along the response given by our AI application and the person can review it, and ideally correct the ocr output.Honestly after an year of working it feels like AI right now can only be useful when supervised all the time (such as when coding). Otherwise I just find LLMs still too unreliable besides basic bogus tasks.

By optimistic, do you mean &#x27;tweaked&#x27;? :)

Excited to test this our on our side as well. We recently built an OCR benchmarking framework specifically for VLMs[1][2], so we&#x27;ll do a test run today.From our last benchmark run, some of these numbers from Mistral seem a little bit optimistic. Side by side of a few models:model | omni | mistral |gemini | 86% | 89% |azure | 85% | 89% |gpt-4o | 75% | 89% |google | 68% | 83% |Currently adding the Mistral API and we&#x27;ll get results out today![1] <a href="https:&#x2F;&#x2F;github.com&#x2F;getomni-ai&#x2F;benchmark">https:&#x2F;&#x2F;github.com&#x2F;getomni-ai&#x2F;benchmark</a>[2] <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;getomni-ai&#x2F;ocr-benchmark" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;datasets&#x2F;getomni-ai&#x2F;ocr-benchmark</a>

re: real world implications, LLMs and VLMs aren&#x27;t magi, and anyone who goes in expecting 100% automation is in for a surprise (especially in domains like medical or legal).IMO there&#x27;s still a large gap for businesses in going from raw OCR outputs —&gt; document processing deployed in prod for mission-critical use cases.e.g. you still need to build and label datasets, orchestrate pipelines (classify -&gt; split -&gt; extract), detect uncertainty and correct with human-in-the-loop, fine-tune, and a lot more. You can certainly get close to full automation over time, but it&#x27;s going to take time and effort.But for RAG and other use cases where the error tolerance is higher, I do think these OCR models will get good enough to just solve that part of the problem.Disclaimer: I started a LLM doc processing company to help companies solve problems in this space (<a href="https:&#x2F;&#x2F;extend.app&#x2F;">https:&#x2F;&#x2F;extend.app&#x2F;</a>)

Is the model Open Source&#x2F;Weight? Else the cost is for the model, not GPU.

$1 for 1000 pages seems high to me. Doing a google searchRent and Reserve NVIDIA A100 GPU 80GB - Pricing Starts from $1.35&#x2F;hourI just don&#x27;t know if in 1 hour and with a A100 I can process more than a 1000 pages. I&#x27;m guessing yes.

&gt; 94.89% overall accuracyThere are about 47 characters on average in a sentence. So does this mean it gets around 2 or 3 mistakes per sentence?

The new Mistral OCR release looks impressive - 94.89% overall accuracy and significantly better multilingual support than competitors. As someone who&#x27;s built document processing systems at scale, I&#x27;m curious about the real-world implications.Has anyone tried this on specialized domains like medical or legal documents? The benchmarks are promising, but OCR has always faced challenges with domain-specific terminology and formatting.Also interesting to see the pricing model ($1&#x2F;1000 pages) in a landscape where many expected this functionality to eventually be bundled into base LLM offerings. This feels like a trend where previously encapsulated capabilities are being unbundled into specialized APIs with separate pricing.I wonder if this is the beginning of the componentization of AI infrastructure - breaking monolithic models into specialized services that each do one thing extremely well.

Curious to see how this performance against more real world usage of someone taking a photo of text (which the text then becomes slightly blurred) and performing OCR on it.I can&#x27;t exactly tell if the &quot;Mistral 7B&quot; image is an example of this exact scenario.

Looks to be API only for now. Documentation here: <a href="https:&#x2F;&#x2F;docs.mistral.ai&#x2F;capabilities&#x2F;document&#x2F;" rel="nofollow">https:&#x2F;&#x2F;docs.mistral.ai&#x2F;capabilities&#x2F;document&#x2F;</a>

I asked LeChat this question:If I upload a small PDF to you are you able to convert it to markdown?LeChat said yes and away we went.

Le chat doesn’t seem to know about this change despite the blog post stating it. Can anyone explain how to use it in Le Chat?

Pretty cool, would love to use this with paperless, but I just can&#x27;t bring myself to send a photo of all my documents to a third party, especially legal and sensitive documents, which is what I use Paperless for.Because of that I&#x27;m stuck with crappy vision on Ollama (Thanks to AMDs crappy ROCm support for Vllm)

It did better than any other solution out there. However, I can only validate by the logic of the text. It&#x27;s a recipe book.

Ohhh. Gonna test it out with some 100+ year old scribbles :)

&gt; There is an OCR page on the link you provided.I don’t see it either. There might be some caching issue.

How is it out of their control to document what they mean by chunking in their parameters?

There is an OCR page on the link you provided. It includes a very, very simple curl command (like most of their docs).I think the friction here exists outside of Mistral&#x27;s control.

1. There’s no simple page &#x2F; sandbox to upload images and try it. Fine, I’ll code it up.2. “Explore the Mistral AI APIs” (<a href="https:&#x2F;&#x2F;docs.mistral.ai" rel="nofollow">https:&#x2F;&#x2F;docs.mistral.ai</a>) links to all apis except OCR.3. The docs on the api params refer to document chunking and image chunking but no details on how their chunking works?So much unnecessary friction smh.

CNN-based OCR also have &quot;hallucinations&quot; and Transformers aren&#x27;t that much different in that respect. This is a problem solved with domain specific post-processing.

well already in 2013 ocr systems used in xerox scanners (turned on by default!) randomly altered numbers, so its not an issue only occuring in llms.

Absolutely right - we tried estimating LLM confidence and the results are not great. Any process that requires reliability will struggle with LLM OCR.<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43350816">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43350816</a>

LLM based OCR is a disaster, great potential for hallucinations and no estimate of confidence. Results might seem promising but you’ll always be wondering.

It&#x27;s not guessing if the form is known and you can read the information directly.This is a common scenario at many banks. You can expect nearly perfect metadata for anything pushed into their document storage system within the last decade.

&gt; Only if the PDFs are unknown or were created by way of a cellphone camera, multifunction office device, etc should you need to reach for OCR.It&#x27;s always safer to OCR on every file. Sometimes you&#x27;ll have a &quot;clean&quot; pdf that has a screenshot of an Excel table. Or a scanned image that has already been OCR&#x27;d by a lower quality tool (like the built in Adobe OCR). And if you rely on this you&#x27;re going to get pretty unpredictable results.It&#x27;s way easier (and more standardized) to run OCR on every file, rather than trying to guess at the contents based on the metadata.

&gt; It takes images and PDFs as inputIf you are working with PDF, I would suggest a hybrid process.It is feasible to extract information with 100% accuracy from PDFs that were generated using the mappable acrofields approach. In many domains, you have a fixed set of forms you need to process and this can be leveraged to build a custom tool for extracting the data.Only if the PDFs are unknown or were created by way of a cellphone camera, multifunction office device, etc should you need to reach for OCR.The moment you need to use this kind of technology you are in a completely different regime of what the business will (should) tolerate.

&quot;Yes&quot; but if a human could do it &quot;AI&quot; should be able to do it too.

No, not a captcha solver. When I worked in education, I was given a 90s paper document that a teacher needed OCRd but it was completely warped. It was my job to remediate those type of documents for Accessibility reasons. I had to scan and OCR it but the result was garbage. Mind you I had access to Windows, Linux and MacOS tools but still difficult to do. I had to guess what it said, which was not impossible but it was time-consuming, not doable in the time-frame I was given, so I had no option but to manually retype all the information into a new document and convert it that way. Document remediation and accessibility should be a good use case for A.I., in education.

Are you trying to build a captcha solver?

I use gemini to solve textual CAPTCHAS with those kind of distortions and more: 60% of the time it works every time.

I would like to see how it performs with massively warped and skewed scanned text images, basically a scanned image where the text lines are wavy as opposed as straight horizontal, where the letters are elongated. One where the line widths are different depending on the position on the scanned image. I once had to deal with such a task that somebody gave me with OCR software, Acrobat, and other tools could not decode the mess so I had to recreate the 30 pages myself, manually. Not a fun thing to do but that is a real use case.

The hard ones are things like contracts, leases, and financial documents which 1) don’t have a common format 2) are filled with numbers proper nouns and addresses which it’s really important not to mess up 3) cannot be inferred from context.Typical OCR pipeline would be to pass the doc through a character-level OCR system then correct errors with a statistical model like an LLM. An LLM can help correct “crodit card” to “credit card” but it cannot correct names or numbers. It’s really bad if it replaces a 7 with a 2.

Perusing the web site, it&#x27;s depressing how much behind Mistral is on basic &quot;how can I make this a compelling hook for customers&quot; for the page.The notebook link? An ACL&#x27;d docThe examples don&#x27;t even include a small text-to-markdown sample.The before&#x2F;after slider is cute, but useless - SxS is a much better way to compare.Trying it in &quot;Le Chat&quot; requires a login.It&#x27;s like an example of &quot;how can we implement maximum loss across our entire funnel&quot;. (I have no doubt the underlying tech does well, but... damn, why do you make it so hard to actually see it, Mistral?)If anybody tried it and has shareable examples - can you post a link? Also, anybody tried it with handwriting yet?

Is this burying the lede? OCR is a solved problem, but structuring document data from scans isn&#x27;t.

Has anyone tried it for handwriting?So far Gemini is the only model I can get decent output from for a particular hard handwriting task

I&#x27;m experiencing the same. Maybe the sentence &quot;Mistral OCR capabilities are free to try on le Chat.&quot; was a hallucination.

It worked perfectly for me with a simple 2 page PDF that contained no graphics or formatting beyond headers and list items. Since it was so small I had the time to proof-read it and there were no errors. It added some formatting, such as bolding headers in list items and putting tics around file and function names. I won&#x27;t complain.

I tried with both PDFs and PNGs in Le Chat and the results were the worst I&#x27;ve ever seen when compared to any other model (Claude, ChatGPT, Gemini).So bad that I think I need to enable the OCR function somehow, but couldn&#x27;t find it.

A bit of a tangent, but aren’t CNNs still dominating over ViTs among computer vision competition winners?

It certainly isn&#x27;t the same cost if expressed as a non-subsidized $$$ one needs for the Transformers compute aka infra.CNNs trained specifically for OCR can run in real time on as small compute as a mobile device is.

Making Transformers the same cost as CNN&#x27;s (which are used in character-level ocr, as opposed to image-patch-level) is a good thing. The problem with CNN based  character-level OCR is not the recognition models but the detection models. In a former life, I found a way to increase detection accuracy, and, therefore, overall OCR accuracy, and used that as an enhancement on top of Amazon and Google OCR. It worked really well. But the transformer approach is more powerful and if it can be done for $1 per 1000 pages, that is a game changer, IMO, at least of incumbents offering traditional character-level OCR.

Given the fact that multi-modal LLMs are getting so good at OCR these days, is it a shame that we can&#x27;t do local OCR with high accuracy in the near-term?

This is $1 per 1000 pages.For comparison, Azure Document Intelligence is $1.5&#x2F;1000 pages for general OCR and $30&#x2F;1000 pages for “custom extraction”.

Great question. The language models are definitely beating the old tools. Take a look at Gemini for example.Doctly runs a tournament style judge. It will run multiple generations across LLMs and pick the best one. Outperforming single generation and single model.

I did not try doctly, but AWS Textract does not support in my case Russian, so the output is completely useless

Why pay more for doctly than an AWS Textract?

would be glad to see benchmarking results

We need to update the examples on the front page. Currently for things that are considered charts&#x2F;graphs&#x2F;figures we convert to a description. For things like logos or images we do an image tag. You can also choose to exclude them.The difference with this is that it took the entire page as an image tag (it&#x27;s just a table of text in my document). rather than being more selective.I do like that they give you coordinates for the images though, we need to do something like that.Give the actual tool a try. Would love to get your feedback for that use case. It gives you 100 free credits initially but if you email me (ali@doctly.ai), I can give you an extra 500 (goes for anyone else here also)

Interestingly I’m currently going through and scanning the hundreds of journal papers my grandfather authored in medicine and thinking through what to do about graphs. I was expecting to do some form of multiphase agent based generation of LaTeX or SVG rather than a verbal summary of the graphs. At least in his generation of authorship his papers clearly explained the graphs already. I was pretty excited to see your post naturally but when I looked at the examples what I saw was, effectively, a more verbose form of``` ![img-0.jpeg](img-0.jpeg) ```I’m assuming this is partially because your use case is targeting RAG under various assumptions bur also partially because multimodal models aren’t near what I would need to be successful with?

We&#x27;ll definitely be doing more tests, but the results I got on the complex tests would result in a lower score and might not be worth the extra cost of the judgement itself.In our current setup Gemini wins most often. We enter multiple generations from each model into the &#x27;tournament&#x27;, sometimes one generation from gemini could be at the top while another in the bottom, for the same tournament.

If you have a judge system, and Mistral performs well on other tests, wouldn&#x27;t you want to include it so if it scores the highest by your judges ranking it would select the most accurate result? Or are you saying that mistral&#x27;s image markdown would score higher on your judge score?

Yes I used the API. They have examples here:<a href="https:&#x2F;&#x2F;docs.mistral.ai&#x2F;capabilities&#x2F;document&#x2F;" rel="nofollow">https:&#x2F;&#x2F;docs.mistral.ai&#x2F;capabilities&#x2F;document&#x2F;</a>I used base64 encoding of the image of the pdf page. The output was an object that has the markdown, and coordinates for the images:[OCRPageObject(index=0, markdown=&#x27;![img-0.jpeg](img-0.jpeg)&#x27;, images=[OCRImageObject(id=&#x27;img-0.jpeg&#x27;, top_left_x=140, top_left_y=65, bottom_right_x=2136, bottom_right_y=1635, image_base64=None)], dimensions=OCRPageDimensions(dpi=200, height=1778, width=2300))] model=&#x27;mistral-ocr-2503-completion&#x27; usage_info=OCRUsageInfo(pages_processed=1, doc_size_bytes=634209)

Where did you test it? At the end of the post they say:&gt; Mistral OCR capabilities are free to try on le Chatbut when asked, Le Chat responds:&gt; can you do ocr?&gt; I don&#x27;t have the capability to perform Optical Character Recognition (OCR) directly. However, if you have an image with text that you need to extract, you can describe the text or provide details, and I can help you with any information or analysis related to that text. If you need OCR functionality, you might need to use a specialized tool or service designed for that purpose.Edit: Tried anyway by attaching an image; it said it could do OCR and then output... completely random text that had absolutely nothing to do with the text in the image!... Concerning.Tried again with a better definition image, output only the first twenty words or so of the page.Did you try using the API?

We&#x27;ve been getting great results with those aswell. But ofcourse there is always some chance of not getting it perfect, specially with different handwritings.Give it a try, no credit cards needed to try it. If you email me (ali@doctly.ai) i can give you extra free credits for testing.

Does doctly do handwritten forms like dates?I have a lot of &quot;This document filed and registered in the county of ______ on ______ of _____ 2023&quot; sort of thing.

Customers are willing to pay for accuracy compared to existing solutions out there. We started out in need of an accurate solution for a RAG product we were building, but none of the solutions we tried were providing the accuracy we needed.

How do you stay competitive with $2&#x2F;100 pages pricing as compared to mistral and others offering 1000 pages for $1 approx?

Co-founder of doctly.ai here (OCR tool)I love mistral and what they do. I got really excited about this, but a little disappointed after my first few tests.I tried a complex table that we use as a first test of any new model, and Mistral OCR decided the entire table should just be extracted as an &#x27;image&#x27; and returned this markdown:```
![img-0.jpeg](img-0.jpeg)
```I&#x27;ll keep testing, but so far, very disappointing :(This document I try is the entire reason we created Doctly to begin with. We needed an OCR tool for regulatory documents we use and nothing could really give us the right data.Doctly uses a judge, OCRs a document against multiple LLMs and decides which one to pick. It will continue to run the page until the judge scores above a certain score.I would have loved to add this into the judge list, but might have to skip it.

The output includes images from the input. You can see that on one of the examples where a logo is cropped out of the source and included in the result.

Zotero&#x27;s PDF viewer also does this now. Being able to annotate PDFs and having a reference manager has been a life saver.

Thanks for the link! Good to know someone is working on something similar.

&gt;a UI that solves the annoying problem of looking for a referenced figure on another page, which breaks up the flow of reading.A tangent but this exact issue is what I was frustrated for a long time with pdf reader and reading science papers. Then I found sioyek that pops up a small window when you hover over links (references and equations and figures) and it solved it.Granted, the pdf file must be in right format, so OCR could make this experience better. Just saying the UI component of that already exist<a href="https:&#x2F;&#x2F;sioyek.info&#x2F;" rel="nofollow">https:&#x2F;&#x2F;sioyek.info&#x2F;</a>

This is incredibly exciting. I&#x27;ve been pondering&#x2F;experimenting on a hobby project that makes reading papers and textbooks easier and more effective. Unfortunately the OCR and figure extraction technology just wasn&#x27;t there yet. This is a game changer.Specifically, this allows you to associate figure references with the actual figure, which would allow me to build a UI that solves the annoying problem of looking for a referenced figure on another page, which breaks up the flow of reading.It also allows a clean conversion to HTML, so you can add cool functionality like clicking on unfamiliar words for definitions, or inserting LLM generated checkpoint questions to verify understanding. I would like to see if I can automatically integrate Andy Matuschak&#x27;s Orbit[0] SRS into any PDF.Lots of potential here.[0] <a href="https:&#x2F;&#x2F;docs.withorbit.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;docs.withorbit.com&#x2F;</a>

I don&#x27;t need AGI just give me superhuman OCR so we can turn all existing pdfs into text* and cheaply host it.Feels like we are almost there.*: <a href="https:&#x2F;&#x2F;annas-archive.org&#x2F;blog&#x2F;critical-window.html" rel="nofollow">https:&#x2F;&#x2F;annas-archive.org&#x2F;blog&#x2F;critical-window.html</a>

Given the wide variety of pricing on all of these providers, I keep wondering how the economics work. Do they have fantastic margin on some of these products or is it a matter of subsidizing the costs, hoping to capture the market? Last I heard, OpenAI is still losing money.

This is $1 per 1000 pages.
For comparison, Azure Document Intelligence is $1.5&#x2F;1000 pages for general OCR and $30&#x2F;1000 pages for “custom extraction”.

I&#x27;m using gemini to solve textual CAPTCHA with some good results (better than untrained OCR).I will give this a shot

Where do you see open source benchmark results that confirm Mistral’s performance?

I am all for open source, but where do you see benchmarks that conclude that it&#x27;s just equivalent?

Back in my days Mistral used to torrent models.

Congrats to Mistral for yet again releasing another closed source thing that costs more than running an open source equivalent:<a href="https:&#x2F;&#x2F;github.com&#x2F;DS4SD&#x2F;docling">https:&#x2F;&#x2F;github.com&#x2F;DS4SD&#x2F;docling</a>

It&#x27;s not about harder but about what error you can tolerate. Here if you have accuracy 99% for many applications it&#x27;s enough. If you have 99% accuracy per trip of no crash during self driving then you gonna be dead within a year very likely.For cars we need accuracy at least 99.99% and that&#x27;s very hard.

I never thought driving a car is harder than editing a pdf.

I&#x27;ve been able to edit PDFs (95%+ of them) accurately for the past 10 years...

I never thought I&#x27;d see the day where technology finally advanced far enough that we can edit a PDF.

For general use this will be good.But I bet that simple ML will lead to better OCRs when you are doing anything specialized, such as, medical documents, invoices etc.

Does anything handle dual columns well? Despite being the academic standard, it seemingly throws off every generic tool.

Looks good but in the first hover&#x2F;slider demo one can see how it could lead to confusion when handling side by side content.Table 1 is referred to in section `2 Architectural details` but before `2.1 Multimodal Decoder`. In the generated markdown though it is below the latter section, as if it was in&#x2F;part of that section.Of course I am nitpicking here but just the first thing I noticed.

gemini flash is notorious for hallucinating the output of the OCR, be careful with it. For straight forward, semi-structured, low page count (under 5) it should perform well, but the more the context window is stretched the more the output gets more unreliable

This is cool! With that said for anyone looking to use this in RAG, the downside to specialized models instead of general VLMs is you can&#x27;t easily tune it to your use specific case. So for example, we use Gemini to add very specific alt text to images in the extracted Markdown. It&#x27;s also 2 - 3X the cost of Gemini Flash - hopefully the increased performance is significant.Regardless excited to see more and more competition in the space.Wrote an article on it: <a href="https:&#x2F;&#x2F;www.sergey.fyi&#x2F;articles&#x2F;gemini-flash-2-tips" rel="nofollow">https:&#x2F;&#x2F;www.sergey.fyi&#x2F;articles&#x2F;gemini-flash-2-tips</a>

Dupe of an hour previous post
<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43282489">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43282489</a>

Wonder how it does with table data in pdfs &#x2F; page-long tabular data?

A similar but different product that was discussed on HN is OlmOCR from AI2, which is open source:<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43174298">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43174298</a>

They will offer integrations into enterprise systems, just like they do today.Lots of big companies don&#x27;t like change. The existing document processing companies will just silently start using this sort of service to up their game, and keep their existing relationships.

I 100% agree with this, I think you can even extend this to any AI, in the end, IMO, as the llm is more commoditized, the surface of which the value is delivered will matter more

It will be interesting to see how all the companies in the document processing space adapt as OCR becomes a commodity.The best products will be defined by everything &quot;non-AI&quot;, like UX, performance and reliability at scale, and human-in-the loop feedback for domain experts.

Probably they are overfitting the benchmarks, since other users also complain of the low accuracy

It fits the &quot;use cases&quot; mentioned in the article&gt; Preserving historical and cultural heritage: Organizations and nonprofits that are custodians of heritage have been using Mistral OCR to digitize historical documents and artifacts, ensuring their preservation and making them accessible to a broader audience.

HTR ( Handwritten Text Recognition ) is a completely different space than OCR. What were you expecting exactly?

Optical Character Recognition (OCR) and Handwritten Text Recognition (HTR) are different tasks

Semi-OT (similar language): The national archives in Sweden and Finland published a model for OCR:ing handwritten Swedish text from the 1600s to the 1800s with what to me seems like a very level of accuracy given the source material. (4% character error rate)<a href="https:&#x2F;&#x2F;readcoop.eu&#x2F;model&#x2F;the-swedish-lion-i&#x2F;" rel="nofollow">https:&#x2F;&#x2F;readcoop.eu&#x2F;model&#x2F;the-swedish-lion-i&#x2F;</a><a href="https:&#x2F;&#x2F;www.transkribus.org&#x2F;success-story&#x2F;creating-the-swedish-lion-%e2%85%b0-model" rel="nofollow">https:&#x2F;&#x2F;www.transkribus.org&#x2F;success-story&#x2F;creating-the-swedi...</a><a href="https:&#x2F;&#x2F;huggingface.co&#x2F;Riksarkivet" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;Riksarkivet</a>They have also published a fairly large volume of OCR:ed texts (IIRC birth&#x2F;death notices from church records) using this model online. As a beginner genealogist it&#x27;s been fun to follow.

my recommendation is to train a custom model

Also working with historical handwritten German documents. So far Gemini seems to be the least wrong of the ones I&#x27;ve tried - any recommendations?

For this task, general models will always perform poorly. My company trains custom gen ai models for document understanding. We recently trained a VLM for the German government to recognize documents written in old German handwriting, and it performed with exceptionally high accuracy.

Tried with a few historical handwritten German documents, accuracy was abysmal.

I am pretty sure it added a kasrah not present in the input on the 2nd to last line.  (Not saying it&#x27;s not super impressive, and also that almost certainly is the right word, but I think that still means not quite &quot;perfect&quot;?)

He means the space between the wāw (و) and the word

Arabic speaker here. No, it&#x27;s perfect.

And here I thought after reading the headline: finally a reliable Arabic OCR. I&#x27;ve never in my life found a good that does the job decently especially for a scanned document. Or is there something out there I don&#x27;t know about?

I noticed on the Arabic example they lost a space after the first letter on the third to last line, can any native speakers confirm? (I only know enough Arabic to ask dumb questions like this, curious to learn more.)Edit: it looks like they also added a vowel mark not present in the input on the line immediately after.Edit2: here&#x27;s a picture of what I&#x27;m talking about, the before&#x2F;after: <a href="https:&#x2F;&#x2F;ibb.co&#x2F;v6xcPMHv" rel="nofollow">https:&#x2F;&#x2F;ibb.co&#x2F;v6xcPMHv</a>

A great question for people wanting to use OCR in business is... Which digits in monetary amounts can you tolerate being incorrect?

They&#x27;re going to become commoditized unless they add value elsewhere. Good news for customers.

&gt;&gt; Any legacy vendors providing pure OCR are going to get steamrolled by these VLMs.-OR- they can just use these APIs, and considering that they have a client base - which would prefer to not rewrite integrations to get the same result - they can get rid of most code base, replace it with llm api and increase margins by 90% and enjoy good life.

Yup definitely, and this is exactly why I built my startup. I&#x27;ve heard this a bunch across startups &amp; large enterprises that we work with. 100% automation is an impossible target, because even humans are not 100% perfect. So how we can expect LLMs to be?But that doesn&#x27;t mean you have to abandon the effort. You can still definitely achieve production-grade accuracy! It just requires having the right tooling in place, which reduces the upfront tuning cost. We typically see folks get there on the order of days or 1-2 weeks (it doesn&#x27;t necessarily need to take months).

It really depends on their fault tolerance. I think there&#x27;s a ton of useful applications where OCR would be 99.9%, 99%, and even 98% reliable. Skillful product manager can keep these limitations in mind and work around them.

... unavoidable &quot;human in the loop&quot; - depends imo.From the comments here, it certainly seems that for general OCR it&#x27;s not up to snuff yet. Luckily, I don&#x27;t have great ambitions.I can see this working for me with just a little care upfront preprocessing now that I know where it falls over. It casually skips portions of the document, and misses certain lines consistently. Knowing that I can do a bit massaging, and feed it what I know it likes, and then reassemble.I found in testing that it failed consistently at certain parts, but where it worked, it worked extremely well in contrast to other methods&#x2F;services that I&#x27;ve been using.

One problem I’ve encountered at my small startup in evaluating OCR technologies is precisely convincing stakeholders that the “human-in-the-loop” part is both unavoidable, and ultimately beneficial.PMs want to hear that an OCR solution will be fully automated out-of-the-box. My gut says that anything offering that is snake-oil, and I try to convey that the OCR solution they want is possible, but if you are unwilling to pay the tuning cost, it’s going to flop out of the gate. At that point they lose interest and move on to other priorities.

yeah that&#x27;s a fun challenge — what we&#x27;ve seen work well is a system that forces the LLM to generate citations for all extracted data, map that back to the original OCR content, and then generate bounding boxes that way. Tons of edge cases for sure that we&#x27;ve built a suite of heuristics for over time, but overall works really well.

AWS Textract works pretty well for this and is much cheaper than running LLMs.

I&#x27;m working on a projet that uses PaddleOCR to get bounding boxes. It&#x27;s far from perfect, but it&#x27;s open source and good enough for our requirements. And it can mostly handle a 150 MB single-page PDF (don&#x27;t ask) without completely keeling over.

The challenge I have is how to get bounding boxes for the OCR, for things like redaction&#x2F;de-identification.

An LLM with billions of parameters for extracting text from a PDF (which isn&#x27;t even a rasterized image) really does not &quot;solve OCR&quot;.

btw - what &#x27;dark patterns&#x27; does portkey contain?

Your customer includes Checkr? Impressive. Are they referencable?

We&#x27;re approaching the point where OCR becomes &quot;solved&quot; — very exciting! Any legacy vendors providing pure OCR are going to get steamrolled by these VLMs.However IMO, there&#x27;s still a large gap for businesses in going from raw OCR outputs —&gt; document processing deployed in prod for mission-critical use cases. LLMs and VLMs aren&#x27;t magic, and anyone who goes in expecting 100% automation is in for a surprise.You still need to build and label datasets, orchestrate pipelines (classify -&gt; split -&gt; extract), detect uncertainty and correct with human-in-the-loop, fine-tune, and a lot more. You can certainly get close to full automation over time, but it&#x27;s going to take time and effort. But the future is on the horizon!Disclaimer: I started a LLM doc processing company to help companies solve problems in this space (<a href="https:&#x2F;&#x2F;extend.app&#x2F;">https:&#x2F;&#x2F;extend.app&#x2F;</a>)

I&#x27;ve been searching relentlessly for something like this! I wonder why it&#x27;s been so hard to find... is it the Chinese?In any case, thanks for sharing.

Yes, i have. The problem with using just an LLM is that while it reads and understands text, but it cannot reproduce it accurately. Additionaly the textbooks I&#x27;ve mentioned have many diagrams and illustrations in them (e.g. books on anatomy or biochemistry). I don&#x27;t really care about extracting text from them, I just need them extracted as images alongside the text, and no LLM does that.

Have you had a chance to compare results from MinerU vs LLM such a Gemini 2.0 or anthropic&#x27;s native PDF tool?

Great progress, but unfortunately, for our use case (converting medical textbooks from PDF to MD), the results are not as good as those by MinerU&#x2F;PDF-Extract-Kit [1].Also the collab link in the article is broken, found a functional one [2] in the docs.[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;opendatalab&#x2F;MinerU">https:&#x2F;&#x2F;github.com&#x2F;opendatalab&#x2F;MinerU</a>
[2] <a href="https:&#x2F;&#x2F;colab.research.google.com&#x2F;github&#x2F;mistralai&#x2F;cookbook&#x2F;blob&#x2F;main&#x2F;mistral&#x2F;ocr&#x2F;structured_ocr.ipynb#scrollTo=svaJGBFlqm7_" rel="nofollow">https:&#x2F;&#x2F;colab.research.google.com&#x2F;github&#x2F;mistralai&#x2F;cookbook&#x2F;...</a>

Is this free in LeChat? I uploaded a handwritten text and it stopped after the 4th word.

Someone working there has good taste to include a Nizar Qabbani poem.

&gt; &quot;Fastest in its category&quot;Not one mention of the company that they have partnered with and that is Cerebras AI and that is the reason they have fast inference [0]Literally no-one here is talking about them and they are about to IPO.[0] <a href="https:&#x2F;&#x2F;cerebras.ai&#x2F;blog&#x2F;mistral-le-chat" rel="nofollow">https:&#x2F;&#x2F;cerebras.ai&#x2F;blog&#x2F;mistral-le-chat</a>

It&#x27;s useful to have the plain text down the line for operations not involving a language model (e.g. search).  Also if you have a bunch of prompts you want to run it&#x27;s potentially cheaper, although perhaps less accurate, to run the OCR once and save yourself some tokens or even use a smaller model for subsequent prompts.

Tons of uses: Storage (text instead of images), search (user typing in a text box and you want instant retrieval from a dataset), etc.  And costs: run on images once - then the rest of your queries will only need to run on text.

Fun, but LLMs would follow them post OCR anyways ;)I see OCR much like phonemes in speech, once you have end to end systems, they become latent constructs from the past.And that is actually good, more code going into models instead.

The biggest risk of vision LLMs for OCR is that they might accidentally follow instructions is the text that they are meant to be processing.(I asked Mistral if their OCR system was vulnerable to this and they said &quot;should be robust, but curious to see if you find any fun examples&quot; - <a href="https:&#x2F;&#x2F;twitter.com&#x2F;simonw&#x2F;status&#x2F;1897713755741368434" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;simonw&#x2F;status&#x2F;1897713755741368434</a> and <a href="https:&#x2F;&#x2F;twitter.com&#x2F;sophiamyang&#x2F;status&#x2F;1897719199595720722" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;sophiamyang&#x2F;status&#x2F;1897719199595720722</a> )

Getting PDFs into #$@ Confluence apparently. Just had to do this and Mistral saved me a ton of hassle compared to this: <a href="https:&#x2F;&#x2F;community.atlassian.com&#x2F;forums&#x2F;Confluence-questions&#x2F;Can-I-upload-a-PDF-to-Confluence&#x2F;qaq-p&#x2F;2914330" rel="nofollow">https:&#x2F;&#x2F;community.atlassian.com&#x2F;forums&#x2F;Confluence-questions&#x2F;...</a>

But what&#x27;s the need exactly for OCR when you have multimodal LLMs that can read the same info and directly answer any questions about it ?For a VLLM, my understanding is that OCR corresponds to a sub-field of questions, of the type &#x27;read exactly what&#x27;s written in this document&#x27;.

La Plateforme agent builder - <a href="https:&#x2F;&#x2F;console.mistral.ai&#x2F;build&#x2F;agents&#x2F;new" rel="nofollow">https:&#x2F;&#x2F;console.mistral.ai&#x2F;build&#x2F;agents&#x2F;new</a>

I feel like i can&#x27;t create an agent with their OCR model yet ? Is it something planned or it&#x27;s only API?

Fwiw, they have an example of a parking receipt in a cookbook: <a href="https:&#x2F;&#x2F;colab.research.google.com&#x2F;github&#x2F;mistralai&#x2F;cookbook&#x2F;blob&#x2F;main&#x2F;mistral&#x2F;ocr&#x2F;structured_ocr.ipynb" rel="nofollow">https:&#x2F;&#x2F;colab.research.google.com&#x2F;github&#x2F;mistralai&#x2F;cookbook&#x2F;...</a>

Agreed. In general I&#x27;ve had such bad performance for complex table based invoice parsing, that every few months I try the latest models to see if its better. It does say &quot;96.12&quot; on top-tier benchmark under the Table category.

We find CV models to be better (higher midpoint on an ROC curve) for the types of docs you mention.

It&#x27;s shocking how much our industry fails to see past its own nose.Not a single example on that page is a Purchase Order, Invoice etc. Not a single example shown is relevant to industry at scale.