[LLM-deploy] Deploying Qwen VLM for the First Time

Background

Because of a project requirement, I couldn’t use a closed-source model for PDF OCR, and traditional OCR didn’t work well enough — so I had no choice but to use an open-source model. In the OCR space, almost everything at the top of the rankings is some version of Qwen, so I started by testing Qwen’s latest 3.5.

Process

Choosing the tech

First I compared a few options for parsing PDF images — traditional Tesseract / EasyOCR, PaddleOCR, and the multimodal VLM (Qwen) approach — then picked 20 of the more complex pages from the PDF to test:

• Tesseract / EasyOCR: traditional OCR. But it doesn’t include Chinese out of the box — you have to install a language pack separately; even after installing the Traditional Chinese pack (tessdata_best), it still didn’t quite meet the bar — some Traditional Chinese characters got split apart, and glyphs were misread, e.g. 「辯護人」 (defense counsel) parsed as 「拓護人」. Just comparing plain text alone, only about 73% of it matched the VLM’s output.
• PaddleOCR: an OCR model from Baidu. Its plain-text results nearly caught up with the VLM, but its one obvious weak spot is flowcharts, tables, and blurry pages — on a trial-procedure flowchart it read the arrow pixels as 全房品品房品品机, and the reading order of the boxes got scrambled.
• VLM (Qwen): it doesn’t just recognize characters, it “understands the layout” — it can read flowcharts and tables in logical order, and handles blurry spots better; more importantly, it can also take a prompt to adjust its behavior (ignore bleed-through from the back of the page, mark unclear text as 【blurry】 instead of guessing, preserve the article-number structure, and so on).

For the sake of thorough PDF parsing, I chose to deploy a VLM myself to process the project’s documents.

Doing the math

To plan what spec of GPU I’d need, I first had to roughly estimate the model’s footprint:

Model weights

Start with Qwen3.5-35B-A3B, whose weights are 35B:

So with the INT4 (AWQ/GPTQ) quantized version, I need to reserve close to 20GB for the weights.

Turning images into tokens

The vision encoder (ViT) in the Qwen VL series cuts a patch every 14x14 pixels, then does a 2x2 merge — combining 4 adjacent patches into 1 token that gets pushed to the model. So a 1024x1024 image generates 1300+ tokens, which occupy the KV cache just like text tokens do.

KV cache

Once you’ve accounted for the weights, there’s a second thing eating VRAM: the KV cache.

The model spits out one character at a time, and every time it produces a new one it has to “look back” at all the preceding ones. Recomputing every previous character each time would be wasteful, so the model stores what it already computed for each token (the Key and Value) and reuses it directly later — that stored stuff is the KV cache.

The biggest difference from the weights is this: the weights are fixed (that ~18GB, whether you run one page or a hundred), but the KV cache varies — its size depends on “how many tokens you feed in at once × how many requests you handle simultaneously.” So how many requests a single card can serve at once, and how long a context it can take, really comes down to “after the weights take their share, how much space is left for the KV cache.” The formula for the KV cache is:

per token = 2 (Key + Value) × num layers × num KV heads × head dim × bytes per number

From Qwen 3’s config:

"text_config": {
    "head_dim": 128,
    "num_hidden_layers": 64,
    "num_key_value_heads": 8,
  },

Plugging in (64 layers, 8 KV heads, 128 dims per head, KV cache stored in FP16 so 2 bytes — this is set by KV_CACHE_DTYPE, independent of whether the weights are INT4):

2 × 64 × 8 × 128 × 2 ≈ 256 KB / per token

So a rough estimate:

In other words, one OCR request eats roughly 1GB of KV cache. Assuming ~30GB is left after the weights, that’s about 30 concurrent requests.

(Aside: the KV cache can also be quantized separately — set KV_CACHE_DTYPE to fp8 and it becomes 1 byte, cutting it in half. This is a separate knob, independent of weight quantization.)

And those ~1300 vision tokens mentioned earlier count in here too, so the higher the image resolution, the more tokens, the bigger the KV cache, and the slower the prefill.

GPU estimate

Add the two pieces together and you roughly know how big a card to rent:

• Weights: ~18GB after Int4 quantization — this part is fixed
• KV cache + others: leave some room for concurrency, plus activations, the vision encoder, and the framework’s own overhead — call it a dozen-plus GB

Rough total: a GPU with 30GB or more.

The luck problem

Right after deploying, I randomly picked 20 of the more complex PDF pages locally and called the Runpod endpoint to parse them — all succeeded. But once I moved the service to the cloud (upload image -> GPU pod -> callback -> R2 bucket), I found that even parsing just 5 pages, one of them would always hit OOM / timeout. I didn’t want to just bump up the GPU size on a whim — I kept feeling something was off — so I tweaked a few of vLLM’s environment variables:

• GPU_MEMORY_UTILIZATION (0.9 → 0.75): at first I thought too little VRAM was left, so I lowered vLLM’s utilization to leave it more headroom. OOM happened anyway.

• MAX_NUM_SEQS (8 → 2): next I thought too many requests were being stuffed onto the same card and blowing out the KV cache, so I dialed down “how many sequences to process at once.” But digging into the logs showed the KV cache was nowhere near full — not this either.

• GDN_PREFILL_BACKEND=triton: this one is probably closest to the truth — this model’s kernel (Qwen3.5 uses a newer attention architecture) has to be “compiled on the fly” during worker cold start, and the compilation itself hogs memory, so the process gets killed by the system. Setting this did help somewhat, but some workers still didn’t pick up the setting and kept OOM-ing / timing out. From the vLLM log:

  FlashInfer GDN prefill kernel is JIT-compiled; first run may take a while
  to compile. Set `--gdn-prefill-backend triton` to avoid JIT compile time
  

The real cause was inference — so, downgrade

Looking back and going through the logs, I found: different workers were running different kernels — some printed Using FlashInfer, some printed Using Triton. The FlashInfer one has to JIT-compile at cold start (the compile target is sm_90a, i.e. Hopper cards like the H100/H200), and the compilation itself blows out memory and gets killed (exit 137). Having the AI dig through the vLLM source, I learned that which kernel is chosen is decided by the GPU model — only Hopper cards go down the FlashInfer path, and RunPod hands out a different card each time it spins up a worker. So within the same batch of 5 pages, whichever page happened to land on a FlashInfer worker died, and the ones on Triton were fine. The failures looked random, but the root cause was actually “inconsistent GPU / worker,” not the model choosing kernels on its own.

On top of that, this model is a reasoning model: before every answer it runs a big chunk of thinking (<think>), so a single page takes several times longer, and even without OOM it easily hits the execution timeout. It was basically two problems stacked on top of each other — “an unstable kernel” plus “slow inference.”

Once that clicked, I decided: rather than keep fighting this new architecture head-on, switch to a more mature, simpler version. So I tested the previous generation, Qwen3-VL-32B:

• It’s a dense architecture (not that new attention that needs JIT compilation), so no matter which card the worker lands on, there’s no OOM-prone kernel;
• It’s the non-reasoning version — it doesn’t run a <think> block first, so each page is much faster and far less likely to time out.

The moment I switched over, OOM and timeouts both vanished, and it was several times faster to boot.

Running eval on the recognized 20 pages

After switching models, I still had to check whether the “downgrade” hurt recognition quality. I fed the 20-page test set mentioned at the start of this post to Qwen3-VL-32B, and the results were pretty good:

• Overall character consistency ~94%.
• Where they differed, it was mostly layout differences (line breaks, whitespace), or the new model actually doing better — e.g. it uses the standard glyph (為 instead of the old-form 爲), correctly marks superscript footnotes (⁶), and reconstructs tables more neatly.
• Most importantly: the spots that matter in legal text — article numbers (Article X), and words like 「得 / 應」 where a single character flips “may” into “must” — were identical across both models.

So from this it looks like downgrading to Qwen3-VL cost nothing in quality, and I went ahead and ran the full PDF OCR with it.

Conclusion

With an AI coding agent, choosing the tech stack is genuinely much easier — have the AI write a demo and a human just evaluates whether it’s any good. During the VLM deployment, though, not really understanding Qwen3.5-35B-A3B cost me a lot of time wrestling with OOM, and in the end, unexpectedly, dropping down a generation and picking the non-reasoning version got it done quickly.

And for future LLM / VLM tasks, you really don’t always have to pick the newest, strongest model — everything is still shifting anyway. Just find the local optimum based on what your own business logic needs and what’s currently available on the market.

Ref

• Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
• Qwen3-VL-32B config
• vLLM source: GDN prefill kernel backend selection (--gdn-prefill-backend, FlashInfer/Triton chosen by GPU model)

• ← Previous Post