Practical guide. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.
For: Developers and teams putting AI into practical use
⚡ TL;DR — the 30-second version
- 70B at four bits needs about 35GB for raw weights alone.
- Active parameters are not the full deployment memory requirement.
- Measure on your hardware; include power and maintenance costs.
Begin with memory arithmetic. A 70-billion-parameter checkpoint at a nominal four bits per weight already needs roughly 35GB for the raw weights, before quantization metadata, runtime buffers and KV cache. It cannot fit entirely in 24GB VRAM on that arithmetic alone. CPU offload is an option, not a promise of cloud-beating speed.
Context length and concurrent requests add runtime state. MoE active-parameter counts describe computation per token, not the total checkpoint you must store and serve. The Qwen3.8-Flash-Next repository and Mistral Medium 3.5 card make good examples of why architecture and deployment requirements must be read together.
“Local inference buys control. It does not suspend memory arithmetic or electricity bills.”
Start with a checkpoint that fits comfortably, measure time to first token and sustained output on your hardware, and test the actual documents or code you need. Keep private workloads local when that meets your threat model, but account for downloaded extensions, telemetry and network tools too.
A hybrid router can send bounded tasks to a local model and escalate harder work. Whether it saves money depends on utilization, hardware cost, quality and maintenance. Run the same acceptance cases on both routes and include offload latency, setup work and failed results in the comparison.
Field guide / 01
Separate decimal GB from usable GiB
A simple weight estimate is parameters multiplied by bits per weight divided by eight. For 70 billion parameters at four bits, that is 35 billion bytes: roughly 32.6 GiB before quantization metadata or runtime state. Advertised memory and operating-system units can differ.
KV cache, activations and concurrency increase the working set. A quantized checkpoint's file size is useful evidence, but the decisive number is peak memory while serving your actual context and request pattern.
Field guide / 02
Offload changes the bottleneck
CPU offload can make a workload possible when VRAM is insufficient, but transfer bandwidth and host memory can dominate latency. Measure the output rate and responsiveness rather than describing the configuration as fast because it loads successfully.
Compare local and hosted inference using accepted-task cost at your real utilization. A machine that sits idle most of the month has different economics from one running steady batch work.
Before you ship
0/5Use this as a review aid. Checks reset when you leave this page.
Questions that come up
Will the Lab estimator tell me that a model definitely fits?
No. It estimates raw weights and configurable headroom. Runtime-specific cache and memory measurements are still required.
Sources & verification
Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.
Editorial policy and correction log →