DROPGoogle launches Gemini 3.8 Flash TTS and Flash-Lite TTS///DROPGemini Omni brings 1080p generation to Google Vids///BREAKClaude Opus 5.5 is here: 1M context, $4 input / $20 output///BREAKOpenAI adds GPT-6 Sol and Luna to the GPT-6 family///PAPEROpus 5.5's system card documents gains—and evaluation limits///BREAKGrok 4.7 launches at $2 input / $6 output per million tokens///DROPGrok Voice Transcribe 2.0 targets difficult real-world audio///TRENDGoogle steers new Gemini API projects to 3.8 Flash or 3.5 Flash-Lite///DROPGoogle launches Gemini 3.8 Flash TTS and Flash-Lite TTS///DROPGemini Omni brings 1080p generation to Google Vids///BREAKClaude Opus 5.5 is here: 1M context, $4 input / $20 output///BREAKOpenAI adds GPT-6 Sol and Luna to the GPT-6 family///PAPEROpus 5.5's system card documents gains—and evaluation limits///BREAKGrok 4.7 launches at $2 input / $6 output per million tokens///DROPGrok Voice Transcribe 2.0 targets difficult real-world audio///TRENDGoogle steers new Gemini API projects to 3.8 Flash or 3.5 Flash-Lite///
DARC.DEV
All guides
TutorialLevel · Build#Open Weights#Local AI#Privacy

Local AI: plan the memory before downloading the weights

Quantization, KV cache, CPU offload and hardware costs. What a 24GB GPU can—and cannot—promise for large models.

DCDariusz Czechowicz23 Sept 20263 min readPractical guide
Local AI: plan the memory before downloading the weights
Source-checked Sources & corrections
2 practical sections · checklist · sources

Practical guide. Provider claims are attributed to the sources below. This page does not claim a new independent DARC benchmark.

For: Developers and teams putting AI into practical use

⚡ TL;DR — the 30-second version

  • 70B at four bits needs about 35GB for raw weights alone.
  • Active parameters are not the full deployment memory requirement.
  • Measure on your hardware; include power and maintenance costs.

Begin with memory arithmetic. A 70-billion-parameter checkpoint at a nominal four bits per weight already needs roughly 35GB for the raw weights, before quantization metadata, runtime buffers and KV cache. It cannot fit entirely in 24GB VRAM on that arithmetic alone. CPU offload is an option, not a promise of cloud-beating speed.

Context length and concurrent requests add runtime state. MoE active-parameter counts describe computation per token, not the total checkpoint you must store and serve. The Qwen3.8-Flash-Next repository and Mistral Medium 3.5 card make good examples of why architecture and deployment requirements must be read together.

“Local inference buys control. It does not suspend memory arithmetic or electricity bills.”
— Editorial interpretation

Start with a checkpoint that fits comfortably, measure time to first token and sustained output on your hardware, and test the actual documents or code you need. Keep private workloads local when that meets your threat model, but account for downloaded extensions, telemetry and network tools too.

A hybrid router can send bounded tasks to a local model and escalate harder work. Whether it saves money depends on utilization, hardware cost, quality and maintenance. Run the same acceptance cases on both routes and include offload latency, setup work and failed results in the comparison.

Field guide / 01

Separate decimal GB from usable GiB

A simple weight estimate is parameters multiplied by bits per weight divided by eight. For 70 billion parameters at four bits, that is 35 billion bytes: roughly 32.6 GiB before quantization metadata or runtime state. Advertised memory and operating-system units can differ.

KV cache, activations and concurrency increase the working set. A quantized checkpoint's file size is useful evidence, but the decisive number is peak memory while serving your actual context and request pattern.

Field guide / 02

Offload changes the bottleneck

CPU offload can make a workload possible when VRAM is insufficient, but transfer bandwidth and host memory can dominate latency. Measure the output rate and responsiveness rather than describing the configuration as fast because it loads successfully.

Compare local and hosted inference using accepted-task cost at your real utilization. A machine that sits idle most of the month has different economics from one running steady batch work.

Before you ship

0/5

Use this as a review aid. Checks reset when you leave this page.

Questions that come up

Will the Lab estimator tell me that a model definitely fits?

No. It estimates raw weights and configurable headroom. Runtime-specific cache and memory measurements are still required.

Put it into practiceOpen the memory estimator

Sources & verification

Last checked 23 Sept 2026. Publication dates and access conditions belong to the linked originals.

  1. 01Qwen3.8-Flash-Next official model cardQwen↗
  2. 02Mistral Medium 3.5 model cardMistral AI↗
Editorial policy and correction log →
#Open Weights#Local AI#Privacy