DeepSeek-V4-Flash on Two GB10s: 304B Params, 1M Context, 83 Watts

Our RTX 5090 vs GB10 comparison ended with a throwaway line: you can pair two GB10s for 235B-class models. This is what happened when we actually did it — with a 304-billion-parameter model, a 1-million-token context window, and a plug meter's worth of curiosity about the power draw.

Short version: DeepSeek-V4-Flash-0731 runs on two desk-side GB10 workstations connected by a single cable, at 88 tok/s peak decode on 83 watts of combined GPU power — under a fifth of what one RTX 5090 draws for a model that card cannot load at any power budget. Every number below is measured on our own hardware, not projected from a spec sheet.

The results at a glance

For scale: a single RTX 5090 draws 575 W under load and holds 32 GB. Two GB10s run a 304B model at 89 tok/s on 83 W of GPU power combined — and the 5090 cannot load this model at all. Those two facts, plus the last row, are what change a procurement conversation.

Why this is newly possible

Eighteen months ago, "run a 300B model locally" meant a rack, three-phase power, and a six-figure budget. Four things changed, and they compound.

1. Sparse mixture-of-experts. V4-Flash has 304B total parameters but activates only ~13B per token — 6 of 256 routed experts plus one shared expert, across 43 layers. You pay 304B in memory and 13B in compute. Memory is cheap to add; compute is not. This is the single most important architectural fact, and it is exactly the property we described in Mixture of Experts, Explained.

2. Models now ship pre-quantized. DeepSeek publishes 0731 with MoE experts already in FP4 and everything else in FP8. On disk that is 155 GiB, not the ~600 GB a BF16 checkpoint this size would need. Crucially this is the vendor's own quantization, validated against their own benchmarks — you are not gambling on a community quant of unknown provenance (why the format matters).

3. Sparse attention makes long context tractable. A "Lightning Indexer" selects the top-512 relevant keys instead of attending densely. A million-token context with dense attention would be arithmetically hopeless on this hardware; with sparse attention it is routine — and the KV cache stays a manageable size.

4. Speculative decoding is built into the checkpoint. The model ships with its own draft module (DeepSeek calls it DSpark). It proposes 5 tokens per step and the main model verifies them in parallel. We measured 4.86 of 6 tokens accepted per step — roughly 4–5× the useful work per forward pass, with no second model to source or version-match.

Stack those and a 304B model becomes a two-box problem instead of a two-rack problem.

The hardware, and the one part people underestimate

Two GB10 systems — Grace-Blackwell superchips in a workstation chassis. We used one HP ZGX Nano G1n and one ASUS GX10; they are OEM variants of the same platform and interoperate without ceremony.

The interconnect is the part people underestimate. We connected the two nodes directly — one cable, port to port, no switch. Configured as RoCE, ethtool reports 200,000 Mb/s and a 1.4 ms round trip.

This matters because tensor parallelism splits every weight matrix across both GPUs, so the halves must exchange activations twice per layer, for every layer, for every token. That traffic is latency-sensitive in a way bulk transfer is not. The same configuration technically works over ordinary gigabit Ethernet — but decode becomes network-bound and both GPUs sit mostly idle waiting on each other. One 200 Gb cable is the difference between a working system and an expensive disappointment.

How the model is split

    Node A (head)                      Node B (worker)
  ┌────────────────────┐            ┌────────────────────┐
  │ disk: 155 GiB      │            │ disk: 155 GiB      │
  │ GPU:  ~78 GiB      │◄──────────►│ GPU:  ~78 GiB      │
  │ (half the weights) │  200 Gb    │ (half the weights) │
  └────────────────────┘  RoCE      └────────────────────┘
        OpenAI-compatible API :8888

Both nodes keep the full checkpoint on disk — tensor parallelism slices individual matrices, so each rank reads from nearly every shard — but each holds only half in GPU memory. That is what puts a 304B model on hardware where neither box could hold it alone.

The model

DeepSeek-V4-Flash-0731, MIT licensed. "Flash" is DeepSeek's efficiency-focused line, and the 0731 release is a substantial step over the earlier preview, particularly for agentic work. From DeepSeek's own published evaluations:

Two things stand out. DeepSWE moves from 7.3 to 54.4 — the preview was barely functional as a coding agent, the release competes with frontier proprietary models. And 0731 beats DeepSeek's own larger V4-Pro on every row with far fewer activated parameters, which is the entire argument for the Flash line.

Caveats worth stating plainly: these are vendor-reported figures from DeepSeek's own harness — treat them as directional and validate on your own workload, which is exactly what local hosting lets you do. The proprietary column is also already dated: DeepSeek benchmarked against Claude Opus 4.8, and Opus 5 shipped since at the same price point, so the real frontier gap on coding is wider than this table shows.

The model is built for agentic and coding work. Terminal Bench, NL2Repo, DeepSWE, and Toolathlon all evaluate a model doing things — reading repositories, calling tools, running commands, fixing failures. It is not primarily a conversational model, and the performance profile reflects that.

What we measured

All figures from our own two-node deployment, temperature 0, warmed engine.

Throughput depends heavily on content

A 2.7× spread, and the cause is speculative decoding: the draft module predicts repetitive, structured output almost perfectly and prose much less well. Across this mix we measured 77.2% draft acceptance, still holding 0.65 at the fifth draft position.

The practical implication: code and structured output — what this model is built for — run at the fast end. Any vendor quoting a single throughput number without the content mix behind it is telling you very little.

Long context is real, not nominal

Prefill throughput rises with prompt length — bigger prompts amortize fixed overhead better. A 100,000-token codebase is ingested in about 30 seconds. The KV cache pool holds 2.87 million tokens, so long conversations and large documents coexist with concurrent requests. (For what a context that size costs you elsewhere, see What a 256K (or 1M) context window actually costs — on owned hardware the answer is different, and much of that piece's arithmetic stops applying.)

Concurrency

Aggregate throughput reaches ~148 tok/s across 6 simultaneous streams. This is a depth machine, not a breadth machine — excellent for a handful of agents doing serious work, wrong for serving hundreds of chat users.

Does it actually write correct code?

Benchmarks are one thing; we wanted executed tests. We gave it ten coding tasks — each demanding an exact function signature, code only — extracted the generated code and ran it against hidden assertions, including edge cases the prompt never mentioned.

10 / 10, mean 68.0 tok/s, every task first attempt, no retries, no human correction.

The tests were not softballs. merge_intervals had to handle touching intervals ([1,4],[4,5] → [1,5]), empty input, and fully-contained intervals. LRUCache was checked for correct eviction ordering across interleaved reads and writes. is_balanced had to reject ([)] — correct nesting, not just matched counts. spiral_order got empty, single-row, and single-column matrices.

The debugging task is the most telling. We handed it a peak-finding binary search with two deliberate bugs — an off-by-one in the initial bound and an unguarded arr[mid + 1] that indexes past the end — and asked for a fix. It found and corrected both, and the result passed on ascending, descending, single-element, and multi-peak inputs. Reading unfamiliar broken code and repairing it is what an agent spends most of its time doing, and it is much harder than writing fresh code from a clean spec.

Note the throughput column: 60–76 tok/s, right where the "code generation" content class predicted. Real coding work lands at the fast end of the range, not the slow prose end.

Where it breaks

Ten out of ten is a suspiciously clean result, so we pushed into harder territory: CUDA. We asked for a program that sorts 10 numbers on the GPU, compiled the answer with nvcc 13.0, and ran it on the GB10 itself.

The model volunteered three kernels — bitonic, bubble, and insertion sort — and recommended bitonic as "the most appropriate GPU approach."

The bitonic kernel declared its working array as int local[16]. In CUDA that is per-thread private memory — every thread gets its own copy. The bitonic algorithm works by exchanging values between threads, so the communication it depends on silently never happens. Output was not merely unsorted but corrupted: 2 0 1 1 3 3 4 4 0 0, values duplicated and half the input lost.

We confirmed the diagnosis by changing exactly one word — int local[16] → __shared__ int local[16] — after which the model's own algorithm sorted perfectly. Every other line of its bitonic network was correct: the XOR partner indexing, the loop structure, the direction logic, the padding to a power of two. It had the algorithm right and the memory model wrong.

The bubble kernel had all ten threads run the same complete sort over the same global array — a data race — plus a __syncthreads() that only part of the block reaches when the block is larger than the data. It printed the correct answer anyway, because ten threads sit in one warp and execute in lockstep. Correct by accident, on this hardware, today.

This is the honest shape of the limitation. The code compiles without warnings. It is well-organised, commented, idiomatic, and would plausibly pass code review. The defects live in the parallel memory model — thread-private versus shared versus global, barrier semantics, race freedom — and none of that is visible in how the text reads.

Two things generalise. First, the failure is domain-shaped, not effort-shaped: the same model in the same session fixed two planted bugs in a subtle binary search without difficulty. Sequential logic it handles; concurrent memory semantics it does not, reliably. Second — and this is the part that goes beyond CUDA — the model included a self-check in its own program, and that check printed Sorted correctly: No. It shipped code that failed its own test, then recommended that approach in the summary.

So: use it, and execute what it writes. That is the whole lesson, and it applies to every model at this capability level. The value of hosting locally is precisely that running the output against real tests is cheap, private, and unmetered — you can afford to verify everything.

It speaks your language

DeepSeek publishes no language list for this model, so we tested the endpoint directly. It produced fluent, technically correct explanations in 16 languages: Serbian (Latin and Cyrillic), Croatian, Slovenian, Polish, Hungarian, Russian, German, French, Spanish, Italian, Turkish, Chinese, Japanese, Korean, Arabic, and English.

Two details matter for anyone outside the anglosphere. It matched the script to the prompt — Cyrillic in, Cyrillic out; Latin in, Latin out — without being asked. And it correctly distinguished Serbian from Croatian, writing rešila / izbegla for one and riješila / spriječilo for the other, where many models flatten both into a generic Serbo-Croatian. For non-English teams, a local model that handles your language properly removes one more reason to send data abroad.

Power: the number that surprised us

We sampled GPU power on both nodes at 2 Hz — idle, then through a sustained 5,001-token generation, then under six concurrent streams:

Note how little the jump from one stream to six costs: 83.1 W → 88.8 W, under 7% more power for roughly 1.7× the aggregate throughput. The hardware is memory-bandwidth-bound, not compute-bound, so extra concurrent work rides along nearly free. This is why efficiency improves so sharply with batching, here and in every datacenter — and it is the same bandwidth story we told in The GPU Memory Wall, seen from the power side.

1.08 tokens per joule single-stream, 1.67 tok/J at six concurrent. Put differently: roughly 3.87 million tokens per kWh. At typical European electricity prices, a few cents' worth of power per million generated tokens.

The idle figure deserves its own moment. 22.4 W keeps a 304-billion-parameter model loaded and instantly available — not paged to disk, not cold-starting, resident in GPU memory and ready to answer. Leave it running permanently and it costs about as much as a couple of LED bulbs.

What this measurement is and isn't. These are nvidia-smi GPU-rail readings. They cover the GPU domain of the GB10 superchip and exclude the Grace CPU cores, system DRAM, NVMe, chassis fans, and PSU conversion losses. Total wall power is higher — these boxes ship with roughly 240 W supplies — and we did not have a plug meter, so we are not publishing a system-power figure we did not measure. The GPU-rail number is still the meaningful comparison, because it is the same quantity you would compare against a discrete accelerator's board power.

Why so low? The same sparsity that makes the model fit. Only ~13B of 304B parameters activate per token, and speculative decoding extracts ~4.86 tokens from each forward pass. The hardware spends most of its time moving weights, not burning watts on arithmetic — and unified LPDDR memory is dramatically more power-efficient per gigabyte than the HBM stacks on datacenter accelerators.

How the economics actually compare

Agent workloads consume enormous token volumes — an autonomous coding agent can burn millions of tokens in an afternoon, much of it re-reading context. Per-token pricing scales linearly with that. Owned hardware does not: past a certain sustained load, the marginal cost of a token approaches the cost of electricity.

And we now know what that floor is. At 1.08 tok/J, a million generated tokens costs roughly 0.26 kWh of GPU power. Even doubling it to account for total system draw, the electricity cost of running this model flat out is negligible against any metered tariff.

Two honest readings of that table.

First: we do not beat the datacenter, and shouldn't. A B200 deployment at scale is roughly 2–4× more energy-efficient per token, and that is entirely expected — large batches amortize weight movement across many concurrent requests, the single biggest lever on inference efficiency. What is notable is the magnitude of the gap: a pair of desk-side workstations lands within a small multiple of purpose-built datacenter silicon, and actually ahead of the H100 industry average for frontier-class models. Two adjustments push further in our favour, both real: datacenter figures are typically multiplied by a PUE of 1.10–1.20 for cooling and facility overhead, which an office does not pay; and our numbers are single-node-pair and lightly batched, which is the worst case for efficiency.

Second, and more important: the bottom three rows are the actual story. The consumer-GPU comparison is not about watts, because there is no comparison to make. A 5090 has 32 GB of VRAM. This model needs 155 GiB. You could buy five 5090s, draw nearly 3 kW, and still not run it — there is no memory-coherent way to shard a 304B model across consumer cards at that tier. The GB10's 128 GB of unified memory per node is the enabling feature, and it is why two of them at 83 W accomplish what a wall of gaming GPUs cannot at any power budget.

Datacenter reference figures are from published 2026 inference-energy analyses (Spheron's AI inference power and GPU electricity cost report; FifthRow's "Beyond the Joule"; TokenPowerBench, AAAI). Cross-setup energy comparisons are inherently rough — different models, batch sizes, sequence lengths, and measurement boundaries. Ours are GPU-rail and measured; theirs are published estimates that may include facility overhead. Treat the table as order-of-magnitude, not a leaderboard. Our own throughput and power were sampled in separate runs at matched load, not simultaneously.

Beyond cost, the two arguments that usually decide it: every token stays on your network — no data-processing agreement to negotiate because there is no third party — and nothing changes underneath you. A locally hosted model you validated in March behaves identically in November. No deprecation, no silent update, no rate limit, no queueing behind other tenants' traffic.

Honest limitations

• Two nodes minimum. 155 GiB does not fit in 128 GB. There is no single-box path at this quality level.
• Not a chat server. ~33 tok/s on prose and limited concurrency. A smaller dense model serves conversational traffic far better.
• Long context has a concurrency cost. Context and simultaneous requests share one memory pool.
• Setup requires competence. This is a two-node distributed inference deployment. It is well-documented and scriptable — we automated ours down to two commands — but it is not an app install.
• Verify concurrent code by running it. Sequential logic is reliable; parallel memory semantics are not.
• 1M context is extrapolated from a natively 64K-trained window. It is the vendor's calibrated ceiling, but quality at 900K is not quality at 9K.

Where it fits

Strong fits

• Self-hosted coding agents — the headline use case. Point it at a repository and let it edit, run tests, and iterate.
• Whole-repository comprehension — a million tokens is a mid-sized codebase in one prompt. No chunking, no retrieval pipeline, no vector database, no relevance tuning: the entire RAG problem simply does not arise.
• Tool-calling and workflow automation — OpenAI-compatible function calling, tested at 70.3 on Toolathlon.
• Regulated and confidential work — source code, client data, health or financial records.
• Sustained agent fleets — a team of agents working continuously, where metered pricing hurts most.
• Deep document and log analysis — long structured input plays to both the context window and the high acceptance rate on structured content.

Poor fits

• High-volume conversational serving
• Latency-critical single-token interactions
• Anything that must run on one machine
• Casual experimentation — this is infrastructure, not a toy

Trying it yourself

The honest summary of effort: the hard parts are already solved and published. The community recipe we used handles the distributed launch, the quantized KV cache, and the speculative decoding configuration. Our own wrapper reduces day-to-day operation to four scripts:

./gb10-start.sh      # boots both nodes, waits, prints how to use it
./gb10-status.sh     # health, GPU state, live acceptance rate
./gb10-chat.sh       # local chat UI with live throughput stats
./gb10-stop.sh       # frees both GPUs

Once running, it is an OpenAI-compatible endpoint, so existing tooling works unmodified:

export OPENAI_BASE_URL=http://<head-node>:8888/v1
export OPENAI_API_KEY=local

Point Crush, Aider, Cline, Continue, or any OpenAI SDK client at it and carry on. That compatibility is what makes local hosting practical rather than a science project — you change one environment variable, not your toolchain.

Wiring it into a terminal coding agent

That claim is easy to make, so here is the whole of it with Crush, a terminal-first coding agent. One block added to ~/.config/crush/crush.json:

"gb10-deepseek": {
  "name": "GB10 dual (DeepSeek-V4-Flash 0731, 1M)",
  "base_url": "http://192.168.1.10:8888/v1/",
  "type": "openai",
  "api_key": "local",
  "models": [{
    "id": "deepseek-v4-flash-dspark",
    "name": "DeepSeek-V4-Flash-0731 (304B MoE, 1M ctx)",
    "context_window": 1048576,
    "default_max_tokens": 16384
  }]
}

That is the entire integration. api_key is required by the schema but ignored by vLLM — any non-empty string works. Restart the agent and the model appears in its picker alongside every hosted provider.

End to end, in a scratch project containing a deliberately broken sample.py:

$ crush run "Read sample.py and tell me in one sentence what the bug is."
The bug is that `add` subtracts instead of adds, so `add(a, b)`
returns `a - b` instead of `a + b`.

The agent issued a tool call, the server executed it through vLLM's deepseek_v4 tool parser, the file came back, and the model answered. Tool calling is the part that actually matters for agent work and the part most likely to be missing — we verified it directly before trusting it, and got a clean finish_reason: tool_calls with well-formed JSON arguments.

Two practical notes from doing this for real. Declare context_window as the true 1,048,576 but keep default_max_tokens modest (16K is plenty) — the first is the conversation budget, the second caps a single reply, and conflating them is how you get answers truncated mid-function. And if your tooling autodiscovers models from Ollama, note that vLLM needs the model listed explicitly, as above.

What we'd tell someone starting today

• Budget for the interconnect. Two 200 Gb ports and a direct cable. Do not economise here; it is the single component that determines whether the system performs.
• Plan storage for the full checkpoint on every node — 155 GiB each.
• Download the weights once and copy them node-to-node over the fast link. We moved 155 GiB in 3 minutes 18 seconds that way, versus a second 40-minute internet download.
• Warm the engine before you judge performance. A freshly booted server measures roughly 30% slow until a few long generations have run. Benchmark cold and you will draw the wrong conclusion.
• Validate against your workload. Published benchmarks are directional; the point of local hosting is that you can measure the real thing.

Bottom line

• 304B parameters, 1M context, 88 tok/s on two desk-side boxes and one cable.
• 83 W of GPU power sustained, 22.4 W idle with the whole model resident.
• 10/10 on executed Python tests, 1/3 on CUDA — sequential logic is reliable, parallel memory semantics are not. Run what it writes.
• ~0.26 kWh per million tokens, within a small multiple of datacenter silicon and ahead of the H100 industry average.
• No consumer GPU can run this model at all, which is the comparison that actually matters.

The interesting thing is not that a 304B model can be made to run on two workstations. It is that it runs well — at the fast end of its range on exactly the structured, code-shaped output it was designed to produce, with benchmark scores within a few points of the strongest proprietary models. The gap between what you can rent and what you can own narrowed fastest for precisely the workload where renting is worst: long-running agents chewing through enormous context windows.

For a lot of teams, that calculation has quietly flipped. Two boxes, one cable, and a weekend of setup now buys a frontier-class coding model that never sends your source code anywhere, never rate-limits you, never changes underneath you, and costs the same on a heavy day as a light one. That is worth measuring for yourself.

Related reading

• AI Workstation Comparison: RTX 5090 vs GB10 (HP ZGX)
• Mixture of Experts (MoE), Explained
• 4-Bit Quantization Decoded: INT4 QAT, MXFP4, and NVFP4
• The KV Cache: the trick that makes LLMs fast — and slow
• What a 256K (or 1M) Context Window Actually Costs You