Oct 2026 · LEAF Workshop · IEEE MILCOM 2026
Measure, Then Compress
Scale- and Architecture-Dependent Routing of Compression for Edge AI
Brian Jalaian · Venkat R. Dasari (DEVCOM ARL)
I presented this work with Venkat Dasari of DEVCOM ARL as a poster at the LEAF workshop at IEEE MILCOM 2026, which we co-chaired. The question behind it is simple: when you compress a model for the edge, should you prune, quantize, or search, and does the answer stay the same as models grow? We measured instead of assuming.
No static compression recipe survives contact with model scale, family, and architecture. When you measure, the right route changes, so the router should measure first and compress second.
The problem
Edge accelerators differ in compute, memory, and power, and each compression method wins only in a narrow context. In practice the choice between pruning, quantization, and architecture search is made by hand and defended by assertion, for example that supernet search is intractable for LLMs.
We add no new operator. Instead we measure which route to apply and what it costs.
What we did
We built an instrumented router. A model analyzer takes a pretrained model and device budgets and sends it down one of three routes: search, prune, and recover (Route P); post-training quantization (Route Q); or a width search for vision models. Every route reports its own cost in GPU-seconds, joules, and peak memory, and that measurement is what selects the route.
We ran one unchanged supernet search across an MLP, a CNN, a ViT, and Qwen3 from 0.6B to 8B parameters (a 13.7× range), with a 4% accuracy-loss budget on CommonsenseQA and ARC-Challenge.
Result 1: search is bound by memory, not time
- From Qwen3-0.6B to 8B, search compute grows 5.9×, but peak memory grows 8.9×.
- A full search at Qwen3-8B takes under one GPU-hour (2,888 GPU-seconds), so search is affordable.
- Peak memory scales with an exponent of 0.83 for the LLM versus 0.08–0.28 for vision models. Compute is sublinear everywhere.
Result 2: search pays only on some architectures
- On CNNs and ViTs, searched width allocation beats uniform allocation by 6.6–12.8 points. CNNs won 56 of 63 paired comparisons while being smaller.
- On MLPs the gain is 0.3–0.5 points, a coin flip. We report it as a null result.
Result 3: quantize, but the crossover follows no scale law
- Quantization beats prune-and-recover at every scale: 2.75× compression versus 1.18× at 8B, with zero search.
- 4-bit NF4 becomes admissible between 4B and 8B on Qwen3, but between 1B and 7B on OLMo-2.
- A precision rule calibrated on Qwen3 is wrong at 2 of 3 OLMo-2 sizes.
Routing guidance
- Key the router on peak memory, not GPU-seconds.
- For LLMs, quantize first, and measure int8 versus NF4 per model.
- For vision, search CNN and ViT widths; skip MLPs.
Limits: all runs are on NVIDIA A40 GPUs rather than edge devices, and cover multiple-choice and image-classification tasks only.