Phase 11 — Exercises: Multi-LoRA

Contents

• Warm-up (explain)
• Core (trace the code)
• Build (your lab)
• Design (staff-level)
• Self-grading

Warm-up (explain)

1. What is a LoRA, in terms of W' = W + ?? Why is it cheap (use the rank r)?
2. What are the shrink and expand steps, and what shapes do they pass through?
3. Why does serving N adapters cost ≈ one base read + N small matmuls, not N model runs?

Core (trace the code)

4. In LoRARequest (request.py:8), what identifies an adapter and what does id 0/-1 mean?
5. In punica_cpu.py, match add_shrink (:166) and add_expand (:197) to shrink/expand.
6. How does SGMV apply different adapters to different rows in one kernel (segments by id)?
7. Where is max_loras enforced — name both the manager and the scheduler spot (Phase 3).

Build (your lab)

8. In lab-01, why is the LoRA delta at most rank r? Prove it with matrix_rank.
9. Add an effective_rank knob: stack two adapters on the same rows (sum of deltas) and verify it equals adding them sequentially.
10. Measure FLOPs: compare base matmul FLOPs to the adapter's shrink+expand FLOPs for r=16, in=out=4096. What's the overhead ratio?

Design (staff-level)

11. A platform serves 5,000 customer fine-tunes. Compare (a) one full deployment per customer vs (b) shared base + multi-LoRA: memory, cost, cold-start. Where does (b) win and where does it hurt?
12. max_loras is hit constantly (lots of distinct adapters per batch). What are your options (raise it, route by adapter, replicate), and the tradeoffs?
13. How does LoRA on MoE expert layers (lora/layers/fused_moe.py) complicate the batched apply, and why?

Self-grading

4–7 and 11–13 are interview-grade. Could you whiteboard shrink/expand and the grouped batched apply? If not, re-read 01-deep-dive.md.