How I Fantastic-Tuned Granite-Imaginative and prescient 2B to Beat a 90B Mannequin — Insights and Classes Discovered

Growing a customized analysis metric like that is essential for reliably monitoring mannequin enhancements and benchmarking efficiency in opposition to reference fashions.

Coaching Setup

To fine-tune the mannequin effectively on my NVIDIA RTX 4070 Ti Tremendous, which has 16 GB VRAM, I used LoRA (Low-Rank Adaptation). This allowed me to replace solely a small variety of parameters, considerably decreasing GPU reminiscence utilization. In truth, throughout coaching, the mannequin used solely about half of the out there VRAM — with sufficient headroom to mess around with longer sequences, however not sufficient to deal with a couple of batch. Moreover, LoRA is mostly quicker to coach than approaches like QLoRA.

LoRA Setup

I used the next LoRA configuration:

# Setup LoRA
target_modules = []
for layer_type in layers_to_tune:
    target_modules.prolong(
        identify for identify, _ in mannequin.named_modules()
        if (layer_type in identify) 
        and '_proj' in identify
    )
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=target_modules,
    use_dora=True,
    init_lora_weights="gaussian"
)

Key factors:

• r=16: This low-rank dimension gives a superb stability between mannequin capability and GPU reminiscence utilization.
• use_dora=True: DoRA (Weight-Decomposed Low Rank Adaptation) improves the training capability and stability of LoRA by decomposing the pretrained weights into magnitude and course parts, serving to the mannequin higher resemble the capability of full fine-tuning — all with out including inference overhead. Carried out barely higher than the default setting.
• init_lora_weights="gaussian": No specific purpose, I didn’t need to experiment with this parameter.
• target_modules: This versatile setup permits selectively concentrating on imaginative and prescient layers, language layers, or each, relying on the experiment. In observe, imaginative and prescient layers remained unaffected — even with use_dora=False— since DoRA at present helps solely embedding, linear, and Conv2d layers. In consequence, I fine-tuned solely the language layers.

Dataset Setup

Throughout my preliminary experiments, I stored operating into out-of-memory (OOM) errors — although there was nonetheless loads of out there GPU VRAM after loading mannequin, LoRA layers and optimizer parameters (round 4GB nonetheless free). There have been no reminiscence spikes throughout coaching, however the crashes constantly occurred on the identical coaching step.

After some investigation, I spotted that the issue was attributable to massive tables, which resulted in extraordinarily lengthy token sequences. To handle this, I adjusted the max_seq_length parameter and filtered out samples that exceeded this restrict. After experimentation, I discovered that utilizing max_seq_length = 1024 allowed me to fine-tune the mannequin reliably with out triggering OOM errors.

To filter out outsized tables, I wrote a easy knowledge processing operate that:

• Filters out samples whose HTML token size exceeds max_seq_length
• Robotically balances the variety of coaching and take a look at samples
• Makes use of streaming to keep away from loading your complete dataset into reminiscence (PubTabNet-HTML is sort of massive, round 10 GB on disk)

.

def load_process_filter_dataset(dataset, max_seq_length, num_train_images, num_test_images, system_message):
    international processor
    ds = load_dataset(dataset, cut up='prepare', streaming=True)
    max_html_tokens = max_seq_length - len(processor.tokenizer.tokenize(system_message))
    num_total_needed = num_train_images + num_test_images
    filtered_samples = []
    p_bar = tqdm(complete=num_total_needed, desc="Filtering dataset samples")
    for pattern in ds:
        processed = process_and_filter_example(pattern, max_html_tokens)
        if processed:
            filtered_samples.append(processed)
            p_bar.replace(1)
        if len(filtered_samples) >= num_total_needed:
            break
    p_bar.shut()
    # Convert to in-memory dataset
    ds_filtered = Dataset.from_list(filtered_samples)
    # Cut up into prepare/take a look at
    ds_train = ds_filtered.choose(vary(num_train_images))
    ds_test = ds_filtered.choose(vary(num_train_images, num_total_needed))
    return ds_train, ds_test

def process_and_filter_example(instance, max_html_tokens):
    international processor
    extracted_table = extract_html_table(instance['html_table'])
    token_count = len(processor.tokenizer.tokenize(extracted_table))
    if token_count < max_html_tokens:
        instance['html_table'] = extracted_table
        return instance
    return None

The ultimate configuration included num_train_images=10000 and num_test_images=250 to compute the analysis loss.

Fantastic-Tuning Configuration

For coaching, I used the Transformers SFTTrainer to fine-tune the mannequin:

# Coaching arguments
    training_args = SFTConfig(
        output_dir=f"src/fashions/{model_name.cut up('/')[-1].exchange('-', '_', 1).cut up('-')[0]}/checkpoints/{experiment_name}",
        num_train_epochs=1,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        gradient_accumulation_steps=gradient_accumulation_steps,
        max_seq_length=max_seq_length,
        warmup_steps=10,
        learning_rate=3e-4,
        weight_decay=0.01,
        logging_strategy="steps",
        eval_strategy='steps',
        logging_steps=25,
        save_strategy="steps",
        save_steps=50,
        save_total_limit=1,
        greater_is_better=False,
        load_best_model_at_end=True,
        optim="adamw_torch_fused",
        bf16=True,
        push_to_hub=False,
        report_to="wandb" if not debug else "none",
        remove_unused_columns=False,
        gradient_checkpointing=True,
        dataset_text_field="",
        dataset_kwargs={"skip_prepare_dataset": True},
        dataset_num_proc=8
    )

# Setup Coach
    coach = SFTTrainer(
        mannequin=mannequin,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
        data_collator=collate_fn,
        peft_config=peft_config,
        processing_class=processor.tokenizer
    )

Key factors:

• num_train_epochs=1: The dataset may be very massive, and to run a number of experiments effectively, I selected to coach for just one full epoch whereas maximizing studying per pattern and variety of coaching samples.
• per_device_train_batch_size=1: Bigger batch sizes wouldn't slot in GPU reminiscence with out considerably decreasing max_seq_length — which might harm efficiency on massive tables. Maintaining longer sequences was extra vital for this process.
• gradient_accumulation_steps=8: Used to successfully simulate a bigger batch measurement and assist stabilize the training course of, compensating for the small bodily batch. That is the ultimate worth, however experimented with gradient_accumulation_steps=4 as properly.
• optim="adamw_torch_fused" and bf16=True: These settings leverage trendy NVIDIA architectures (Ada Lovelace) to speed up coaching and scale back reminiscence utilization — as really helpful for this {hardware}.

Analysis Loss Workaround

On the time of creating the mission, there's a identified subject within the Transformers + LoRA integration that causes an error when operating analysis with a validation dataset throughout coaching. Thankfully, a community-tested workaround is out there (though not but merged into the primary department), and I efficiently used this repair in my experiments.

Analysis (Inference) Setup

The analysis dataset used for last scoring was fully unbiased from the eval_dataset used throughout coaching. It consists of 500 randomly chosen pictures, none of which have been included in both the train_dataset or the coaching eval_dataset.

As soon as fine-tuning was full, I used the greatest mannequin checkpoint — chosen based mostly on the bottom analysis loss — to run inference on these 500 samples.

Initially, I tried to carry out inference by merely loading the LoRA/DoRA adapter on high of the bottom mannequin. Nevertheless, I discovered that inference with DoRA adapters is extraordinarily sluggish when not merged into the mannequin weights (as defined within the official PEFT docs). In truth, producing one take a look at random pattern took about 90 seconds on this configuration.

To resolve this, I merged the adapter weights into the bottom mannequin — which is the really helpful observe — and after merging, inference pace improved dramatically: right down to ~20 seconds for a similar pattern, making full analysis runs rather more sensible.

The reference fashions used for comparability with my fine-tuned fashions are:

• meta-llama/Llama-3.2–90B-Imaginative and prescient: Meta’s large 90-billion parametermannequin — the primary baseline I aimed to surpass by way of specialization and parameter-efficient fine-tuning of a a lot smaller VLM.
• KennethTM/pix2struct-base-table2html: A a lot smaller mannequin fine-tuned from Google’s pix2struct-base, extremely specialised for precisely the identical dataset I used on this mission. Due to its smaller measurement, the developer(s) was in a position to prepare it for a lot of extra samples and over longer coaching runs — demonstrating the important thing benefit of utilizing smaller, focused fashions for particular duties.

These two baselines allowed me to benchmark each scaling-based efficiency (vs the 90B mannequin) and specialization effectivity (vs the smaller, devoted Pix2Struct mannequin).

Experiments & Outcomes

A complete of 9 experiments have been carried out, iteratively modifying one or two parts at a time. The aim was to grasp the impact of every change on mannequin efficiency, progressively refining the setup to realize the absolute best HTML Similarity rating in comparison with reference fashions.

The experimental course of was incremental: each time a change improved the outcomes, it was integrated into the subsequent spherical of experiments and continued exploring new variations.

The experiments centered on adjusting the next parts:

1. Imaginative and prescient vs. Language Layers

• 1.1 lang_only
• 1.2 vision_only
• 1.3 lang_vision

2. Floor Fact Output Format

3. Coaching Framework

• 3.1 lang_table_unsloth
• 3.2 vision_table_unsloth

4. Gradient Accumulation

5. Immediate Format

6. Gradient Accumulation & Dataset Measurement

Each the analysis loss and the HTML Similarity metric have been used to evaluate mannequin efficiency, and I discovered them to be properly correlated — confirming that HTML Similarity is an effective proxy for the way properly the mannequin is studying the duty.

Earlier than diving into the outcomes of every experiment, let’s first have a look at GPU reminiscence utilization throughout coaching, which is usually probably the most important consider figuring out whether or not a mannequin might be fine-tuned on shopper {hardware}.

As proven within the graph, GPU utilization remained secure all through coaching — averaging round 75% VRAM utilization, or roughly 12 GB on my GPU. Most of VRAM utilization (~5.5 GB) is the frozen mannequin weights. LoRA gradients + optimizer states take little or no (<< 1 GB). Activations + overhead ought to fill the remaining (~5–6 GB), which will depend on batch_size and max_seq_length.

First Run: lang_only

This experiment makes use of the next preliminary parts/parameters:

These have been the beginning values for the primary experiment. In subsequent runs, I modified lots of them as I refined the method. This primary experiment centered solely on tuning language layers, whereas coaching the mannequin to foretell the complete uncooked HTML output — together with all the pieces inside and across the 

 tags.

Since this was the primary run, I’ll embrace the coaching loss curve right here for instance the way it behaves. For later experiments, I’ll omit this graph — because the conduct was comparable throughout runs, with minor variations. In observe, analysis loss is extra helpful for evaluating efficiency throughout experiments.

One vital notice concerning the logging configuration: logging_steps=25 implies that the coaching loss is barely logged after each 25 steps, the place every logged worth is the common over gradient_accumulation_steps=4. In consequence, the biggest drop in loss seems on the second log level — the place a lot of the preliminary studying occurs. After that, the mannequin continues studying extra progressively, with a sluggish lowering development, relying on the problem of the coaching samples.

Now, let’s check out the analysis loss:

Since we're evaluating on the identical set of 250 validation samples, the analysis loss curve offers us a extra secure and significant view of mannequin studying — and can function a baseline for comparisons throughout future runs.

Right here, we observe a transparent and constant downward development all through coaching. The preliminary loss begins near 0.03, with a gradual enchancment as coaching progresses, ultimately stabilizing slightly below 0.015.

The graceful nature of this curve — in comparison with the extra variable coaching loss — displays the common construction of the validation set and confirms that the mannequin is generalizing properly to unseen samples, even with a small batch measurement and a single epoch of coaching.

Now, let’s examine the efficiency of this fine-tuned mannequin in opposition to the reference fashions on the HTML Similarity metric:

As we are able to see, this primary experiment already delivers sturdy efficiency features — bettering the bottom Granite-Imaginative and prescient 2B mannequin by a big margin (+0.18) and clearly outperforming LLaMA 90B Imaginative and prescient on this specialised process. Solely Pix2Struct retains a slight lead at this stage.

Second Run: vision_only

There isn’t a lot to research on this run. I examined a number of variations that would probably unblock studying within the imaginative and prescient layers — together with drastically growing the training fee — however with out success.

Whereas the bottom code means that fine-tuning imaginative and prescient layers ought to be potential, in observe I discovered it was not working on this setup. The next analysis loss curve confirms that no studying occurred — the loss remained fixed all through coaching. To keep away from losing compute assets, I finished the run early:

Moreover, coaching was noticeably quicker on this run in comparison with the earlier lang_only experiment — suggesting that the language layers (which include the majority of the mannequin’s parameters) remained frozen, and solely the small imaginative and prescient layers have been being processed:

Third Run: lang_vision

At this level, it was clear that solely language layers have been being successfully educated. On this lang_vision run — the place each language and imaginative and prescient layers have been chosen — I anticipated outcomes much like lang_only.

Certainly, the analysis loss curve confirmed this expectation, exhibiting almost equivalent conduct to lang_only:

As soon as this was clear, I once more stopped coaching early to preserve assets, and proceeded to check new approaches.

Fourth Run: lang_table_only

This experiment modified the next element:

The aim of this run was to coach the mannequin to foretell solely the desk content material, with none surrounding HTML wrapper code. This method might assist enhance studying — by eradicating pointless tokens — and likewise align the coaching conduct extra carefully with Pix2Struct’s mannequin.

Moreover, by stripping out the wrapper HTML, the goal sequences turned shorter — which allowed longer and extra advanced tables to suit throughout the mannequin’s context window. This transformation might additionally enhance the mannequin’s skill to generalize to bigger or extra detailed tables.

Let’s have a look at the analysis loss in comparison with the primary run:

At first look, the upper analysis loss might sound counterintuitive. Nevertheless, there’s a transparent rationalization: the wrapper HTML code is trivial for the mannequin to study — because it tends to be almost equivalent throughout many coaching samples. These repetitive tokens scale back cross-entropy loss, artificially reducing the common loss in earlier runs. By eradicating them, the mannequin now focuses totally on the tougher and variable desk content material — leading to a better however extra significant loss worth.

Now, let’s see how this variation impacted the HTML Similarity metric:

On this first take a look at, we observe no important acquire or degradation from utilizing this new output format. It's potential that the mannequin would wish extra epochs or bigger coaching samples to completely adapt to this new format. One other concept is to replace the immediate — in order that from the very first step the mannequin understands it ought to focus solely on desk content material, somewhat than having to deduce this conduct by way of coaching alone. This will probably be explored within the subsequent experiments.

Fifth / Sixth Run: lang_table_unsloth, vision_table_unsloth

In these experiments, I explored the next parts:

At this level, I found the promising Unsloth framework — which claims to supply 2x quicker coaching with as much as 70% decrease reminiscence utilization. After all, I needed to check whether or not it might speed up my workflow.

My first concept was to leverage the improved reminiscence dealing with to run longer sequences (max_seq_length=2048), however in my case this rapidly led to Out of Reminiscence (OOM) errors — so I reverted to my earlier configuration.

The coaching pace enhancements, nevertheless, have been simple — nearly 4x quicker than my earlier runs:

Sadly, this got here at a transparent price to loss efficiency:

Given this noticeable drop in high quality, I paused the experiment to research additional — significantly to see if Unsloth would permit me to coach imaginative and prescient layers, which is one in all its marketed benefits. Nevertheless, I encountered precisely the identical conduct as with HuggingFace Transformers — no precise studying in imaginative and prescient layers.

With these leads to thoughts, I made a decision to put aside Unsloth for this missionand proceed utilizing HuggingFace Transformers, which had proven extra dependable studying in earlier runs.

Seventh Run: lang_table_only_2

Listed below are the brand new parameters for this run:

Going again to the earlier configuration, I needed to research the impression of a bigger digital batch measurement (through increased gradient_accumulation_steps).

The outcomes have been promising — the analysis loss turned smoother and trended nearer to the unique lang_only run, although the mannequin was now predicting solely the desk content material:

Primarily based on this constructive consequence, I made a decision to maintain this gradient_accumulation_steps=8 setting for the ultimate experiment.

Evaluating this mannequin on HTML Similarity resulted in a small however significant enchancment — lastly reaching parity with Pix2Struct:

Naturally, the aim isn't just to match Pix2Struct — however to surpass it. Two vital levers remained to discover: dataset measurement and immediate.

Eighth Run: lang_table_only_3

The up to date parameters for this run have been:

I by accident reverted gradient_accumulation_steps again to 4 on this run, solely realizing it as soon as the coaching was almost full — however this truly gave me an extra-chance to watch its impact on studying.

The principle aim right here was to double the coaching measurement (to 10K pictures) and to check the up to date, clearer immediate format. Sadly, a random CUDA error brought on coaching to halt round 80% completion — besides, the advance was clear:

As anticipated, some smoothness was misplaced as a result of smaller digital batch measurement, however the brand new immediate proved very efficient — noticeably boosting mannequin studying.

This set the stage completely for the last experiment, utilizing this improved immediate, 10K coaching samples, and restoring gradient_accumulation_steps to eight.

Ultimate Run: lang_table_only_4

The ultimate set of parameters are:

The analysis loss for this last run:

As anticipated, restoring the gradient_accumulation_steps to eight smoothed the loss curve, decreasing spikes and reaching barely decrease total loss values. With a full epoch of coaching on 10K pictures, this turned the best-performing mannequin throughout all experiments.

Now, let’s have a look at the ultimate outcomes on the HTML Similarity metric:

The aim of this mission was achieved — the fine-tuned mannequin now surpasses each reference fashions on this process. Trying again on the authentic Granite-Imaginative and prescient 2B, the LoRA fine-tuning improved efficiency to 0.77, a +21 share level acquire — all achieved in underneath 8 hours on a consumer-grade GPU.

Qualitative Outcomes

To higher illustrate how a lot the mannequin improved by way of fine-tuning, let’s have a look at a particular instance: Picture ID 618932.

This desk is especially difficult — underneath the Kappa column there are sub-headers (Current research and King et al. 2001). These advanced layouts usually problem generic VLMs, particularly once they haven’t been uncovered to sufficient comparable examples throughout coaching. Fashions can normally perceive these sub-headers and reply questions about them, however producing the complete desk construction in HTML typically requires additional immediate tuning and specialised fine-tuning.

Let’s first see how a base, non-fine-tuned Granite-Imaginative and prescient 2B mannequin performs on this process.

Baseline: Uncooked Granite-Imaginative and prescient 2B

The mannequin can reply questions based mostly on the desk appropriately:

immediate='What's the Kappa worth for the query "Do you talk with this energy?" within the current research?'
res = predict(pattern['image'], immediate=immediate)
print(res)

Out[1]:

74

Nevertheless, when requested to generate the complete HTML desk, the mannequin struggles:

immediate = "Convert desk to HTML ()"
html = predict(pattern['image'], immediate=immediate)
html = '' if '

' not in html else html show(HTML(html))

Out[2]:

And the HTML Similarity metrics for this try:

Fashion similarity: 1.0000
Structural similarity: 0.4091
Lev-Edit Distance: 0.1434
Ultimate HTML Similarity Rating: 0.3619

Fantastic-Tuned Mannequin: lang_table_only_4

Now, let’s strive the very same take a look at utilizing the fine-tuned mannequin:

from src.fashions.granite_vision.transformers_library import LLM as granite_vision

mannequin = granite_vision(
    model_path,
    adapter='lang_table_only_4'
)

Out[4]:

Mannequin loaded
Adapter 'lang_table_only_4' loaded
Adapter 'lang_table_only_4' merged
Utilizing cuda: NVIDIA GeForce RTX 4070 Ti SUPER

And the identical prediction immediate:

immediate = "Convert desk to HTML ()"
html = mannequin.predict(pattern['image'], max_new_tokens=1024, question=immediate)
show(HTML(html))

Out[5]:

The fine-tuned mannequin now produces an output that carefully matches the bottom fact, appropriately capturing the desk construction and sub-headers — one thing the bottom mannequin struggled with.

Ultimate HTML Similarity metrics:

Fashion similarity: 1.0000
Structural similarity: 0.9231
Lev-Edit Distance: 1.0000
Ultimate HTML Similarity Rating: 0.9615

This instance reveals a transparent quantitative enchancment as properly: from a rating of 0.36 to 0.96 on a fancy desk construction — confirming that fine-tuning on this specialised process dramatically boosts the mannequin’s functionality.

Inference Pace

One main benefit of utilizing a smaller mannequin — except for the flexibility to fine-tune on consumer-grade {hardware} — is inference pace. Even when bigger fashions supply aggressive efficiency, latency and throughput stay key components, particularly in manufacturing settings.

Let’s examine the inference pace of the totally different fashions:

As proven within the plot, Pix2Struct is by far the quickest mannequin. For some use instances — resembling batch-processing hundreds of paperwork for desk extraction — this pace benefit might translate into important time financial savings and decrease compute prices.

Nevertheless, the fine-tuned Granite-Imaginative and prescient 2B achieves a superb stability when the quantity of paperwork to course of will not be large, having a superior accuracy on this specialised process and fairly quick inference with out the necessity for very massive compute infrastructure.

Conclusions

This mission demonstrated that with LoRA-based fine-tuning and a focused process (desk extraction → HTML), a small vision-language mannequin (Granite-Imaginative and prescient 2B) can outperform a lot bigger fashions — even Meta’s 90B LLaMA Imaginative and prescient — whereas requiring solely a shopper GPU and fewer than a day of coaching.

Just a few key takeaways:

• Small, specialised fashions matter — you don’t at all times want 70B+ fashions to resolve particular issues with excessive accuracy.
• Parameter-efficient fine-tuning (LoRA) is a game-changer: adapting massive basis fashions turns into accessible for many practitioners.
• Immediate design and coaching targets have an enormous affect — small adjustments (like switching to lang_table_only or refining the immediate) immediately impacted efficiency.
• Having a customized metric (HTML Similarity) was important to trace significant progress past generic text-based metrics.
• Smaller fashions not solely prepare quicker, but additionally infer quicker — preferrred for manufacturing pipelines with excessive quantity.

Lastly — and perhaps most significantly — the sort of experimentation reveals that you may transfer quick and iterate even with restricted {hardware}. Fantastic-tuning highly effective open fashions and adapting them to real-world duties is not reserved to huge labs anymore.

I hope this encourages different AI engineers to experiment with small VLMs and fine-tuning methods for their very own initiatives and options — and to see that highly effective outcomes are potential even with out large compute budgets!

What’s Subsequent?

There are positively some attention-grabbing follow-up concepts that may be explored subsequent:

• Immediate engineering refinements: Ultimate checks (whereas scripting this weblog) confirmed that separating prompts into system message (defining conduct) and person message (offering process directions) considerably improved the bottom mannequin’s efficiency. Making use of this technique throughout fine-tuning might additional improve the mannequin’s skill to constantly generate correct HTML. This will probably be examined in upcoming experiments.
• Coaching imaginative and prescient layers: At present, solely the language layers are fine-tuned, as coaching the imaginative and prescient layers by way of text-only loss proved ineffective. A extra superior method might contain including an auxiliary imaginative and prescient loss — for instance, contrastive studying between imaginative and prescient outputs and HTML construction — to raised adapt the imaginative and prescient spine for desk extraction duties.
• Improved generalization: The present mannequin is fine-tuned on a single dataset. Increasing coaching to incorporate extra various doc layouts, desk types, and noisy OCR situations might enhance robustness and transferability to real-world knowledge.

Hyperlinks

---

In case you preferred this submit, be at liberty to achieve out or share your individual experiments!

---

In the direction of Knowledge Science is a neighborhood publication. Submit your insights to achieve our international viewers and earn by way of the TDS Creator Fee Program.

Write for TDS

Associated Articles

• 

  Step-by-step code information to constructing a Convolutional Neural Community

  August 20, 2024

  6 min learn

• A deep dive on the that means of understanding and the way it applies to LLMs

  August 21, 2024

  31 min learn

• 

  A newbie’s information to forecast reconciliation

  August 20, 2024

  13 min learn

• 

  Characteristic engineering, structuring unstructured knowledge, and lead scoring

  August 21, 2024

  7 min learn

• 

  With demos, our new answer, and a video

  August 16, 2024

  10 min learn

• 

  Discover the knowledge of LSTM main into xLSTMs - a possible competitors to the present-day LLMs

• 

  This sophistication matrix can present you the place you must go