are highly effective fashions that take photos as inputs, as a substitute of textual content like conventional LLMs. This opens up numerous prospects, contemplating we will immediately course of the contents of a doc, as a substitute of utilizing OCR to extract textual content, after which feeding this textual content into an LLM.
On this article, I’ll focus on how one can apply imaginative and prescient language fashions (VLMs) for lengthy context doc understanding duties. This implies making use of VLMs to both very lengthy paperwork over 100 pages or very dense paperwork that include numerous data, equivalent to drawings. I’ll focus on what to contemplate when making use of VLMs, and what sort of duties you possibly can carry out with them.
Why do we’d like VLMs?
I’ve mentioned VLMs so much in my earlier articles, and lined why they’re so vital to grasp the contents of some paperwork. The primary purpose VLMs are required is that numerous data in paperwork, requires the visible enter to grasp.
The choice to VLMs is to make use of OCR, after which use an LLM. The issue right here is that you just’re solely extracting the textual content from the doc, and never together with the visible data, equivalent to:
- The place completely different textual content is positioned relative to different textual content
- Non-text data (basically all the things that isn’t a letter, equivalent to symbols, or drawings)
- The place textual content is positioned relative to different data
This data is usually essential to actually perceive the doc, and also you’re thus usually higher off utilizing VLMs immediately, the place you feed within the picture immediately, and may subsequently additionally interpret the visible data.
For lengthy paperwork, utilizing VLMs is a challenges, contemplating you want numerous tokens to characterize visible data. Processing hundres of pages is thus an enormous problem. Nonetheless, with numerous latest developments in VLM know-how, the fashions have gotten higher and higher and compressing the visible data into affordable context lengths, making it attainable and usable to use VLMs to lengthy paperwork for doc understanding duties.
OCR utilizing VLMs
One good choice to course of lengthy paperwork, and nonetheless embrace the visible data, is to make use of VLMs to carry out OCR. Conventional OCR like Tesseract, solely extracts the textual content immediately from paperwork along with the bounding field of the textual content. Nonetheless, VLMs are additionally skilled to carry out OCR, and may carry out extra superior textual content extraction, equivalent to:
- Extracting Markdown
- Explaining purely visible data (i.e. if there’s a drawing, clarify the drawing with textual content)
- Including lacking data (i.e. if there’s a field saying Date and a clean area after, you possibly can inform the OCR to extract Date
)
Lately, Deepseek launched a strong VLM based mostly OCR mannequin, which has gotten numerous consideration and traction recently, making VLMs for OCR extra fashionable.
Markdown
Markdown could be very highly effective, since you extract formatted textual content. This enables the mannequin to:
- Present headers and subheaders
- Symbolize tables precisely
- Make daring textual content
This enables the mannequin to extract extra consultant textual content, will extra precisely depicts the textual content contents of the paperwork. If you happen to now apply LLMs to this textual content, the LLMs will carry out manner higher than if you happen to utilized then to easy textual content extracted with conventional OCR.
LLMs carry out higher on formatted textual content like Markdown, than on pure textual content extracted utilizing conventional OCR.
Clarify visible data
One other factor you need to use VLM OCR for is to elucidate visible data. For instance, if in case you have a drawing with no textual content in it, conventional OCR wouldn’t extract any data, because it’s solely skilled to extract textual content characters. Nonetheless, you need to use VLMs to elucidate the visible contents of the picture.
Think about you may have the next doc:
That is the introduction textual content of the doc
That is the conclusion of the doc If you happen to utilized conventional OCR like Tesseract, you’ll get the next output:
That is the introduction textual content of the doc
That is the conclusion of the docThat is clearly a problem, because you’re not together with details about the picture displaying the Eiffel tower. As a substitute, you must use VLMs, which might output one thing like:
That is the introduction textual content of the doc
This picture depicts the Eiffel tower throughout the day
That is the conclusion of the docIf you happen to used an LLM on the primary textual content, it after all wouldn’t know the doc accommodates a picture of the Eiffel tower. Nonetheless, if you happen to used an LLM on the second textual content extracted with a VLM, the LLM would naturally be higher at responding to questions in regards to the doc.
Add lacking data
You can even immediate VLMs to output contents if there may be lacking data. To grasp this idea, take a look at the picture under:
If you happen to utilized conventional OCR to this picture, you’ll get:
Deal with Street 1
Date
Firm GoogleNonetheless, it might be extra consultant if you happen to used VLMs, which if instructed, may output:
Deal with Street 1
Date
Firm Google That is extra informative, as a result of we’re data any downstream mannequin, that the date area is empty. If we don’t present this data, it’s inconceivable to know late if the date is just lacking, the OCR wasn’t capable of extract it, or some other purpose.
Nonetheless, OCR utilizing VLMs nonetheless endure from a few of the points that conventional OCR struggles with, as a result of it’s not processing visible data immediately. You’ve most likely heard the saying that a picture is value a thousand phrases, which frequently holds true for processing visible data in paperwork. Sure, you possibly can present a textual content description of a drawing with a VLM as OCR, however this article will by no means be as descriptive because the drawing itself. Thus, I argue you’re in numerous circumstances higher off immediately processing the paperwork utilizing VLMs, as I’ll cowl within the following sections.
Open supply vs closed supply fashions
There are numerous VLMs obtainable. I follw the HuggingFace VLM leaderboard to concentrate to any new excessive performing fashions. In accordance with this leaderboard, you must go for both Gemini 2.5 Professional, or GPT-5 if you wish to use closed supply fashions by way of an API. From my expeirence, these are nice choices, which works effectively for lengthy doc understanding, and dealing with advanced paperwork.
Nonetheless, you may also need to make the most of open-source fashions, attributable to privateness, value, or to have extra management over your individual software. On this case, SenseNova-V6-5-Professional tops the leaderboard. I havn’t tried this mannequin personally, however I’ve used Qwen 3 VL so much, which I’ve good expertise with. Qwen has additionally launched a particular cookbook for lengthy doc understanding.
VLMs on lengthy paperwork
On this part I’ll speak about making use of VLMs to lengthy paperwork, and concerns it’s important to make when doing it.
Processing energy concerns
If you happen to’re operating an open-source mannequin, certainly one of your most important concerns is how giant of a mannequin you possibly can run, and the way lengthy it takes. You’re relying on entry to a bigger GPU, atleast an A100 normally. Fortunately that is broadly obtainable, and comparatively low-cost (sometimes value 1.5 – 2 USD per hour an numerous cloud suppliers now). Nonetheless, you need to additional think about the latency you possibly can settle for. Runing VLMs require numerous processing, and it’s important to think about the next elements:
- How lengthy is appropriate to spend processing one request
- Which picture decision do you want?
- What number of pages do you might want to course of
If in case you have a stay chat for instance, you want fast course of, nevertheless if you happen to’re merely processing within the background, you possibly can enable for longer processing occasions.
Picture decision can also be an vital consideration. If you happen to want to have the ability to learn the textual content in paperwork, you want high-resolution photos, sometimes over 2048×2048, although it naturally relies on the doc. Detailed drawings for instance with small textual content in them, would require even increased decision. Improve decision, significantly will increase processing time and is a crucial consideration. It’s best to goal for the bottom attainable decision that also means that you can carry out all of the duties you need to carry out. Moreover, the variety of pages is the same consideration. Including extra pages is usually essential to have entry to all the knowledge in a doc. Nonetheless, usually, an important data is contained early within the doc, so you might get away with solely processing the primary 10 pages for instance.
Reply dependent processing
One thing you possibly can attempt to decrease the required processing energy, is to start out of easy, and solely advance to heavier processing if you happen to don’t get the specified solutions.
For instance, you might begin of solely trying on the first 10 pages, and seeing if you happen to’re capable of correctly remedy the duty at hand, equivalent to extracting a bit of knowledge from a doc. Provided that we’re not capable of extract the piece of information, we begin taking a look at extra pages. You possibly can apply the identical idea to the decision of your photos, beginning with decrease decision photos, and transferring to increased decision of required.
This will of hierarchical processing reduces the required processing energy, since most duties could be solved solely trying on the first 10 pages, or utilizing decrease decision photos. Then, provided that crucial, we transfer on to course of extra photos, or increased decision photos.
Value
Value is a crucial consideration when utilizing VLMs. I’ve processed numerous paperwork, and I sometimes see round a 10x enhance in variety of tokens when utilizing photos (VLMs) as a substitute of textual content (LLMs). Since enter tokens are sometimes the motive force of prices in lengthy doc duties, utilizing VLMs often considerably will increase value. Word that for OCR, the purpose about extra enter tokens than output tokens doesn’t apply, since OCR naturally produces numerous output tokens when outputting all textual content in photos.
Thus, when utilizing VLMs, is extremely vital to maximise your utilization of cached tokens, a subject I mentioned in my latest article about optimizing LLMs for value and latency.
Conclusion
On this article I mentioned how one can apply imaginative and prescient language fashions (VLMs) to lengthy paperwork, to deal with advanced doc understanding duties. I mentioned why VLMs are so vital, and approaches to utilizing VLMs on lengthy paperwork. You possibly can for instance use VLMs for extra advanced OCR, or immediately apply VLMs to lengthy paperwork, although with precautions about required processing energy, value and latency. I believe VLMs have gotten increasingly vital, highlighted by the latest launch of Deepseek OCR. I thus suppose VLMs for doc understanding is a subject you must get entangled with, and you must discover ways to use VLMs for doc processing purposes.
👉 Discover me on socials:
📩 Subscribe to my e-newsletter
🧑💻 Get in contact
✍️ Medium
You can even learn my different articles:
