A vision encoder turns the image into embeddings that occupy a fixed set of placeholder token positions in the prompt. The language model then attends over text+image tokens uniformly. vLLM's input processor handles encoding, placeholder expansion, and caching the encoder output so it isn't recomputed each step.

The vision encoder is extra compute/memory before prefill; image tokens inflate sequence length (and KV); and the encoder-cache plus input-processing must be profiled and batched carefully, especially for dynamic-resolution models.