Oct 2026 · LEAF Workshop · IEEE MILCOM 2026
A Generalized Optimization Engine (GOE)
For Edge AI Inference Acceleration
Venkat R. Dasari, Jakob A. Adams, Vinod K. Mishra (DEVCOM ARL) · Brian Jalaian
GOE is joint work with Venkat Dasari, Jakob Adams, and Vinod Mishra at DEVCOM ARL, and we presented it as a poster at the LEAF workshop at IEEE MILCOM 2026. It asks a practical question: given a model and a target device, which compression method actually deploys, and still works once it does?
GOE-compressed LLMs run on a GPU-less edge CPU. Proper 8-bit quantization is about 1.9× smaller and 1.7× faster with accuracy essentially unchanged, while a naive 8-bit scheme collapses to chance. The compression method, not just the bit-width, decides what survives deployment.
The problem
The tactical edge is dynamic, heterogeneous, and resource-constrained, and large models are hard to deploy there. Existing compression methods work in a narrow, model- or platform-specific context, often with accuracy decay. The goal of GOE is one model- and hardware-agnostic engine.
The approach
GOE maximizes model performance over choices of quantization, pruning, and hyperparameters, subject to budgets on inference time, memory, and energy on the target device. A model analyzer profiles each model and routes it to the right methods: architecture search, tensor decomposition, structured pruning, and distillation, followed by hardware-targeted quantization and compilation (TVM, TensorRT, OpenVINO).
The engine splits by model scale: supernet architecture search for CNNs, and compression routes for LLMs.
LLMs on an edge CPU
We ran Llama-3.2-1B and Qwen2.5-1.5B on an Intel Core Ultra 7 265U with no GPU, using llama.cpp, and compared 8-bit and 4-bit quantization against structured pruning with recovery.
- 8-bit (Q8_0): about 1.9× smaller and 1.7× faster, with accuracy essentially unchanged.
- 4-bit (Q4_K_M): about 3× smaller and 2.5× faster, for a few accuracy points.
- Pruning plus 4-bit was fastest (72 tokens/s on Llama) but fell toward chance accuracy.
CNNs
- ResNet-50 with TorchAO INT8: more than 25× the throughput, with accuracy unchanged at 81.0%.
- ResNet-50 pruned 75% shrinks from 97.7 to 6.2 MB, and after 5 epochs of recovery matches or beats the base accuracy.
- Architecture search on ResNet-50: the top-10 subnets are all at least 2× compressed, and quantization trims about 10% more.
- Caveat: MobileNetV3-Small, already compact, drops from 68% to 38% under INT8.
Takeaways
- Method beats bit-width: a naive dynamic 8-bit scheme, the same width as Q8_0, collapses to chance because of activation-outlier error.
- Aggressive pruning buys speed but approaches chance even after recovery.
- GOE's job is to route each model to a method that both deploys and works.
Next: multimodal inputs and on-the-fly reconfiguration mid-mission.