Running DeepSeek-V4-Flash and Kimi-K3 on Consumer Hardware with SSD Expert Pack

SGLang brings the core idea of SSD-LLaMA to MoE inference: keep routed experts that do not fit in VRAM and host RAM on an NVMe SSD, load only the experts selected by the router, and use Expert Pack layout, direct I/O, pinned staging, asynchronous H2D transfers, and a GPU cache to turn SSD capacity into a practical backing tier.

1. Introduction: turning a VRAM problem into a storage problem

The total parameter capacity of DeepSeek-V4-Flash and Kimi-K3 is far beyond the VRAM of a single consumer GPU. A conventional deployment therefore needs multiple GPUs or hundreds of gigabytes, and sometimes terabytes, of host memory. That capacity requirement creates a large barrier between frontier model capability and local hardware.

SGLang's SSD-backed Expert Pack path takes a different approach. Routed expert weights remain on an NVMe SSD. The router activates only a small subset of experts for each token, so the runtime moves only the selected experts that are not already cached to the GPU. Expert Pack reorganizes the weights of each layer/expert pair into a directly addressable contiguous expert block. The runtime reads that expert block into an aligned pinned host buffer using direct I/O, then transfers it to a GPU cache asynchronously.

This path changes how model weights are stored and delivered, not the model computation. It does not prune, replace, merge, or skip selected experts, and it does not reduce Expert Top-K. The result is a practical way to run DeepSeek-V4-Flash and the validated text-only Kimi-K3 path with an Intel Ultra5 230F CPU, 32 GB memory, a TiPro9000 2 TB disk, and an RTX 5090 with 32 GB VRAM.

MoE computation is sparse, but model capacity is not

Mixture-of-Experts models split the feed-forward network into many experts. After the router scores the experts for a token, only a small subset participates in that token's computation. The remaining experts are idle for that token.

The router's choice changes across tokens and prompts. The complete expert pool must therefore remain available even though only a small working set is active at any one time. Quantization reduces the artifact size, but it does not remove the need to store the expert pool. MoE inference consequently has two different properties:

• per-token computation is sparse;
• the total expert capacity that must be stored and delivered is very large.

This is why SSD is a useful backing tier. It provides much more capacity than consumer VRAM or RAM, and modern PCIe 5.0 NVMe SSDs provide enough sequential bandwidth to make a carefully designed delivery path viable. SSD capacity becomes executable model memory only when the layout, read path, and cache policy match expert-level access patterns.

The capacity-cost difference

A capacity-cost comparison makes the trade-off clear. The following figures are capacity-only lower bounds, not complete system prices:

The figure does not mean SSD and DRAM have the same latency, or that buying an SSD alone is sufficient to run the model. It shows that placing the complete expert pool in VRAM or DRAM quickly becomes impractical, while using SSD for capacity and a bounded GPU cache for the active working set can substantially lower the hardware barrier.

SGLang's SSD Expert Pack approach

The central SSD-LLaMA idea is to manage SSD, RAM, and VRAM as a runtime-controlled storage hierarchy. The complete expert pool stays in the high-capacity tier, while limited VRAM retains the experts with the highest observed reuse.

SGLang's Expert Pack is an implementation of the most important expert-centric parts of that idea inside the SGLang MoE runtime:

1. Expert Pack makes a layer/expert pair an independently addressable contiguous expert block.
2. O_DIRECT and aligned pinned buffers remove the extra page-cache staging copy.
3. A byte-budgeted LFU/LRU GPU cache retains complete experts that are reused.

The current SGLang path does not claim to reproduce every mechanism in the SSD-LLaMA paper. SGLang's implementation is GPU-centric: pinned host memory is a bounded transfer staging area, not a persistent host expert cache, and the current feature does not require the paper's CPU expert execution or lossless CUDA decompression. Keeping this distinction explicit makes the feature boundary precise.

2. Why the native GGUF loading path is not enough

With the original GGUF or multi-shard tensor layout, the gate, up, and down weights of one expert may be located in different file regions. One router hit can therefore trigger several small reads, tensor-name lookups, and staging operations.

An explicit on-demand read avoids speculative prefetch, but exposes the full SSD and H2D latency on the critical path of the current MoE layer. After the router produces its result, the GPU must wait for the selected experts. Prefetching can hide part of that latency, but it has two fundamental limitations:

• the correct expert may still arrive too late because the routing result is only known after the previous computation;
• an incorrect prediction consumes SSD bandwidth, staging space, and GPU cache capacity, after which the actually selected expert must still be read.

SGLang therefore first changes the physical expert layout, then reduces the cost of every unavoidable cache miss.

Why native GGUF cannot directly use expert-level O_DIRECT

Native GGUF is a tensor-oriented model container, not an expert-oriented direct-I/O store. Its metadata and tensor payloads are organized around individual tensors, and a single expert's gate, up, and down weights may be separated across file regions or across multiple shards. The original loading path commonly uses a parser, mmap, or buffered file reads, so the application sees pageable page-cache-backed mappings rather than a preallocated aligned DMA destination.

O_DIRECT requires all of the following to be controlled by the caller:

• a file offset aligned to the storage and filesystem contract;
• a read length aligned to that contract;
• a user-provided buffer whose address is also aligned and suitable for the read.

An arbitrary tensor slice in a native GGUF file does not provide that expert-level contract. Its offset may not be aligned, its length may not be a multiple of the required block size, and the three tensors needed for one expert are not guaranteed to form one contiguous range. A caller could issue separate aligned reads with padding and then reconstruct the expert in another buffer, but that gives up the main benefit: it reintroduces multiple reads and extra assembly work, while the original mmap/page-cache path still cannot use the page-cache pages themselves as an O_DIRECT destination.

Expert Pack is the offline transformation that makes direct I/O practical. It places all roles of one layer/expert pair into one padded, aligned expert block, records its exact offset and length in the manifest, and provides a pinned buffer whose address satisfies the same contract. The runtime can then read and transfer a complete expert without asking the native GGUF layout to behave like a direct-I/O layout.

3. Expert Pack: organizing weights by expert

Contiguous expert layout

SGLang Expert Pack v1 treats one (layer, expert) pair as one complete expert. A DeepSeek expert contains the gate, up, and down roles. The manifest records each role's tensor boundaries, format, integrity information, and pack offset.

The logical layout is:

The runtime does not scan the file for tensor names. It derives the expert offset from the pack metadata:

expert_offset = data_start
              + (layer * num_experts + expert) * expert_stride

Role offsets inside the expert block are validated as well. One manifest lookup can therefore resolve the complete expert read range. The runtime may split that range into a bounded number of parallel tasks according to read_splits.

The layout changes the physical organization of weights on SSD, not their tensor contents, quantization formats, routing decisions, or model mathematics. Kimi-K3 uses a separate GGML Expert Pack adapter. The currently validated input consists of 38 Q2_K GGUF shards; its routed experts use Q2_K for gate/up and Q3_K for down.

Alignment is required for direct I/O

Direct I/O cannot use arbitrary file offsets, lengths, and user buffers in the same way as ordinary read(). The SGLang runtime validates:

• Expert Pack offsets for each expert;
• the start and length of every read range;
• the address of every pinned staging buffer.

The current implementation checks 4096-byte alignment. If the pack or staging buffers do not satisfy the contract, initialization fails instead of silently falling back to an uncontrolled path during inference.

4. The key optimization: removing the page cache -> pinned memory copy

This is one of the most important differences between Expert Pack and a conventional file-reading path.

Traditional buffered I/O

Traditional file reads normally go through the operating system page cache:

The page cache is a kernel-managed file cache. It is not the same thing as the page-locked user memory that CUDA can use for asynchronous H2D. To issue an asynchronous H2D transfer, the application normally prepares a pinned buffer. The file data therefore has to be copied from the page cache into that pinned buffer before the GPU transfer can start. From the application's perspective, this page-cache-to-pinned handoff is a synchronous CPU memory copy: the host-side staging step must complete before the H2D operation has a valid pinned source buffer. It is not itself a cudaMemcpyAsync operation.

This is neither SSD reads nor H2D transfers. Rather, it is an extra synchronous memory copy operation performed by the CPU on the host side: it reads the payload from page-cache-backed memory and writes it into a pinned staging buffer. For a large expert, this translates to a read and write of the full expert size, consuming host memory bandwidth and introducing an additional kernel-to-userland staging handoff before the GPU transfer can proceed.

Expert Pack with direct I/O

When direct_io=True, SGLang opens the Expert Pack with O_DIRECT and makes the read target a preallocated, aligned pinned staging buffer:

The read target is already the pinned buffer required by CUDA, so the intermediate step below is removed:

page cache -> pinned memory

This is not a faster implementation of that copy. The copy is removed from the data path. A simplified cost model is:

Traditional path:
T = T(SSD -> page cache)
  + T(page cache -> pinned)
  + T(pinned -> GPU)
  + T(sync)

Expert Pack direct I/O:
T = T(SSD -> pinned)
  + T(pinned -> GPU)

O_DIRECT does not make the SSD's physical bandwidth increase. It removes one full host-memory traversal from the end-to-end path, which can provide these benefits:

• one less host-memory read and write, reducing CPU and memory-bandwidth pressure;
• one less synchronization handoff between kernel page cache and user-space staging;
• no large expert payload polluting the page cache and competing with unrelated data;
• a completed expert block can enter the H2D path without a page-cache staging copy;
• the same expert-level contract is reused for every layer/expert pair.

The page-cache-copy claim applies only to direct_io=True. The current SGLang Expert Pack loader and the DeepSeek/Kimi 5090 launchers enable this option by default. If direct I/O is explicitly disabled, the path may go through the page cache and an additional staging copy again.

5. GPU/VRAM cache: keeping the working set close to compute

The GPU cache is the mechanism that turns repeated expert access into a local VRAM hit. Expert Pack is not a cache of individual tensor fragments: one cache entry contains the complete gate/up/down data for one (layer, expert) pair. Keeping the complete expert together matters because a selected expert needs all of its roles for computation. Caching only one role would still force the other roles to be read and would not remove the cache-miss cost.

The cache is byte-budgeted rather than expert-count-budgeted. Since different models and adapters have different per-expert payload sizes, the runtime derives the number of available slots from the usable VRAM budget:

usable_vram = min(requested_cache, free_vram - reserve)
slot_count  = floor(usable_vram / expert_payload_bytes)

The reserve protects memory needed by the model, CUDA runtime, activations, and other non-cache allocations. Initialization fails if the resulting slot count cannot hold one complete top-k working set. This makes the cache contract explicit: a cache budget is not allowed to consume the memory required for the current MoE computation.

On a cache hit, the runtime reuses the resident expert and does not read the Expert Pack or issue an H2D for that expert. If a previous transfer is still pending, a CUDA event protects the consumer from observing a partially installed slot. On a cache miss, the runtime selects a victim slot, reads the complete expert into a reusable pinned staging buffer, copies the expert into the GPU slot, and publishes the slot only after the transfer event is ready.

Replacement combines frequency and recency. The runtime records how often each (layer, expert) is selected and when it was last used. A frequently selected expert is harder to evict than a cold expert; among similarly useful entries, an older entry is a better victim. Active experts for the current top-k request are protected from eviction, so the cache cannot evict the working set it is about to execute.

The source Expert Pack is immutable. Evicting a GPU entry therefore requires no write-back: the expert can always be reconstructed from its recorded SSD offset. This makes VRAM cache management simpler than a dirty data cache and keeps replacement focused on reuse value rather than persistence.

The runtime exposes counters that make cache behavior measurable:

• cache_hits and cache_misses;
• cache_evictions;
• pack_read_bytes;
• h2d_bytes;
• fallback_count and io_errors.

These counters distinguish a cache problem from an I/O problem. A low hit rate means the VRAM budget or workload locality is insufficient; high pack_read_bytes and h2d_bytes with a good hit rate may instead indicate that the active set is larger than the cache during a particular phase. io_errors reports observed I/O failures, while fallback_count is diagnostic telemetry whose meaning depends on an instrumented fallback path; neither is a cache-performance metric.

The current execution order places an expert-cache miss on the critical path for the MoE computation it feeds. acquire() copies routing IDs to CPU, waits for the SSD read futures, enqueues the expert-level H2D transfers, and makes the current CUDA stream wait for their transfer events. Only after those events are ready does apply() launch the MoE kernels. Reads and transfers for different missing experts may overlap during the delivery phase, but the current path does not overlap that delivery with the MoE computation that consumes the experts.

A simplified per-step model for the current path is therefore:

T_step ~= T(miss delivery) + T(GPU compute)

Here, T(miss delivery) includes route-ID preparation, SSD reads, staging, H2D submission, and the waits needed to make the selected experts available. On a GPU cache hit, the SSD-read and H2D portions can be skipped. A cross-step pipeline could change this model, but that is not part of the execution path described here. The actual result depends on SSD bandwidth, access distribution, cache hit rate, staging-slot count, and expert shapes.

6. Experimental setup

We evaluated DeepSeek-V4-Flash and Kimi-K3 with SGLang's SSD Expert Pack path on one consumer machine: an Intel Ultra5 230F CPU, 32 GB of memory, a TiPro9000 2 TB disk, and an RTX 5090 with 32 GB of VRAM. The workload represents endpoint inference rather than batched serving: the ten fixed requests are sent one at a time, and the next request starts only after the previous one finishes.

The test set contains five Alpaca requests and five MMLU requests. Both comparisons use the same prompt order, temperature 0, default EOS handling, and a 200-token generation target. The result section reports the mean prefill and decode rates, together with the SGLang cache hit rate and SSD traffic. Exact revisions, file preparation, and server commands are documented after the results.

Expert Pack is explicitly selected with --load-format expert_pack; ordinary auto, safetensors, and gguf loading paths are unchanged. The detailed DeepSeek-V4-Flash and Kimi-K3 reproduction procedures are in Section 9.

7. Correctness boundary

Expert Pack is a weight-layout and delivery optimization, not an approximate-inference algorithm. The correctness contract is:

• every expert selected by the router is executed;
• Expert Top-K is unchanged;
• a selected expert is not replaced by a different resident expert;
• selected experts are not pruned, skipped, or merged;
• the pack and manifest are structurally, dimensionally, and cryptographically validated as configured;
• fallback_count and io_errors are reported instead of silently hiding I/O failures.

Validation showed that DeepSeek-V4-Flash produced semantically equivalent answers to Ollama across multiple prompt categories. Kimi-K3 matched the 200-token SGLang reference output; all 92 routed layers executed Top-16 experts with io_errors=0. fallback_count is retained as diagnostic telemetry, but the current path does not expose an instrumented increment for every hypothetical fallback, so zero is not used as an independent correctness proof. Correctness is instead established by the route/output audit and the structural pack checks.

8. Performance results

All SGLang, Ollama, and llama.cpp measurements use the test environment and revisions documented in Section 9. The figures describe validation results under this hardware condition; token counts and runtime-specific software settings remain as stated in each comparison. The earlier summary tables are retired; the figures below are now the canonical presentation of the token-rate comparison.

The validated weight files and generated Expert Packs occupy:

These are file sizes for the validated weight payloads. Pack indexes, locks, manifests, and other metadata are separate.

DeepSeek-V4-Flash vs. Ollama

The comparison uses ten shared requests: five Alpaca and five MMLU. Both runtimes generated up to 200 tokens per request. The chart reports mean prefill and decode token rates for each dataset.

Relative to Ollama, SGLang improves prefill by 2.28x on Alpaca and 3.39x on MMLU. Decode improves by 6.92x and 6.55x, respectively.

The underlying per-dataset means are compactly reported below. Rates are in tokens per second and are arithmetic means over the five records in each dataset.

Kimi-K3 vs. llama.cpp

The comparison uses the same ten fixed requests in both runtimes: five Alpaca and five MMLU. Both clients use temperature 0 and default EOS handling; each request generated exactly 200 completion tokens, so the decode comparison is now matched for prompt set, stop behavior, and output length. The chart reports mean prefill and decode token rates for each dataset.

Relative to llama.cpp, SGLang improves prefill by 6.96x on Alpaca and 5.80x on MMLU. Decode improves by 3.30x and 3.52x, respectively. The ten-request aggregates use the ten retained request records described in Section 9.

Expert-cache hit rate and SSD traffic

Token rate should be read together with cache telemetry. The following Python-generated bar charts report the unique-key VRAM cache hit rate and mean SSD traffic per generated token. cache_hits and cache_misses count unique (layer, expert) keys after acquire() de-duplicates keys within each update; they are not per-token router-edge accesses. A cache hit avoids the SSD read and H2D transfer for that expert. The SSD traffic metric is the unweighted per-request mean of total pack_read_bytes across prefill and decode, normalized by generated completion tokens, and is therefore not decode-only traffic. Both figures show SGLang-only telemetry, so the single series is labeled by the surrounding text rather than a legend.

DeepSeek uses the complete ten-request run: five Alpaca and five MMLU requests, each with a 200-token completion. Its unique-key VRAM cache hit rate is 54.2% for Alpaca and 46.4% for MMLU. The mean SSD traffic, combining prefill and decode, is 1.66 and 2.04 decimal GB per generated token, respectively.

The Kimi figure uses all ten completed requests: five Alpaca and five MMLU, each with a 200-token completion. The unweighted per-request unique-key VRAM cache hit rates are 17.0% and 22.7%, respectively. The corresponding mean SSD traffic, combining prefill and decode, is 22.10 and 27.63 decimal GB per generated token. The difference from DeepSeek is expected: Kimi's configuration reserves 5 GiB for the GPU expert cache, while the DeepSeek run reserves about 21 GiB, and the two adapters have different expert payload sizes and routing behavior.

The improvement does not come from one isolated faster-copy primitive. It is the combined effect of:

1. Expert Pack turning scattered tensor accesses into addressable contiguous expert reads;
2. direct I/O removing the page cache -> pinned memory copy;
3. pinned staging provides a CUDA-compatible host source for expert-level H2D without page-cache staging;
4. the GPU cache skipping SSD reads and H2D transfers on a hit.

9. Detailed reproduction

This section gives the complete reproduction flow after the result tables. The two subsections use the same ten fixed requests. They are endpoint-style tests: the client sends ten HTTP requests in order, waits for each response before sending the next request, and uses concurrency 1. No batch of requests is used.

The requests are five Alpaca samples and five MMLU samples, in this order:

Each request uses temperature 0, default EOS handling, and a 200-token target. The clients record prompt tokens, completion tokens, TTFT/prefill timing, and decode timing for every request. The tables in Section 8 report arithmetic means over the five requests in each dataset.

In the SGLang commands below, model-meta is the lightweight directory holding the model configuration and tokenizer metadata used to initialize the server. It is not the raw GGUF weight file. The raw GGUF remains the source artifact, while the Expert Pack supplies the routed expert payload; the loader connects them through source_path, pack_path, and the manifest.

9.1 DeepSeek-V4-Flash: SGLang and Ollama

Versions and workload

The SGLang run uses commit 81c9f837f19ff8dfe1a9fcd1abfc6069dd28d2ec on branch support_deepseek-v4_and_kimi-k3_on_ssd. The baseline uses Ollama 0.33.1 with its managed llama.cpp runner at commit d222767c7. Both sides run the ten requests above serially, one request at a time, with the same sampling settings.

Starting the Ollama baseline server

Start the Ollama service before pulling the model and sending requests:

OLLAMA_HOST=127.0.0.1:11435 ollama serve >/tmp/deepseek-ollama.log 2>&1 &

Preparation

1. With the baseline service running, pull the validated DeepSeek-V4-Flash MXFP4 GGUF blob from the Ollama model page and use ollama show --modelfile to obtain its local file path. The corresponding model card is DeepSeek-V4-Flash-0731 on Hugging Face:

   ollama pull frob/deepseek-v4-flash-0731
   ollama show --modelfile frob/deepseek-v4-flash-0731
   

   The recorded blob SHA-256 is 947ac34c08c0e5c5752ac76398f934b3b6b4075cfe915ba43dd5ac754900a4cd and the Ollama manifest SHA-256 is 882b1398c0ca4e7ec8ca0a501fd8c4372f780f690536a3ec17ffc75306569ed3.

2. Install the SGLang checkout and build the DeepSeek Expert Pack manually. Use a matching DeepSeek model configuration JSON for --model-config:

   cd /path/to/sglang-latest-deepseek-v4-kimi-k3-ssd
   python3 -m pip install -e 'python'
   python3 tools/expert_pack/prepare_deepseek_pack.py \
     --gguf /path/to/deepseek-v4-flash-0731.gguf \
     --model-config /path/to/deepseek-v4-flash-config.json \
     --safety-margin-gib 16
   

   This creates or reuses DeepSeek-V4-Flash.expert-pack and its DeepSeek-V4-Flash.expert-pack.manifest.json beside the source GGUF.

3. Validate the generated Pack and create the metadata used by the server. This command does not start a service; because the Pack was built in the previous step, the validation phase reuses it instead of rebuilding it:

   python3 examples/runtime/deepseek_v4/benchmark_deepseek_5090.py \
     --gguf /path/to/deepseek-v4-flash-0731.gguf \
     --validate-only
   

   The generated metadata files are stored under ${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/deepseek-v4-flash/<fingerprint>/model-meta/: config.json, generation_config.json, tokenizer.json, tokenizer_config.json, and metadata.json.

   Here <fingerprint> is a short hash derived from the source GGUF state and preparation format. It is generated locally to isolate artifacts for different source files; it is not a fixed model name or a directory that must be downloaded.

Starting SGLang

The following is the direct Expert Pack server command. The hash values are read from the generated manifest before launch.

GGUF=/path/to/deepseek-v4-flash-0731.gguf
ARTIFACT_DIR=${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/deepseek-v4-flash/<fingerprint>
MODEL_META="$ARTIFACT_DIR/model-meta"
PACK_PATH="$(dirname "$GGUF")/DeepSeek-V4-Flash.expert-pack"
MANIFEST_PATH="$(dirname "$GGUF")/DeepSeek-V4-Flash.expert-pack.manifest.json"
STATS_PATH="$ARTIFACT_DIR/deepseek-v4-expert-pack.stats.json"
SOURCE_SHA256="$(jq -r '.source.sha256' "$MANIFEST_PATH")"
OLLAMA_MANIFEST_SHA256="$(jq -r '.model.model_identity_sha256 // .model.ollama_manifest_sha256' "$MANIFEST_PATH")"
CONFIG_SHA256="$(jq -r '.model.config_sha256' "$MANIFEST_PATH")"

python3 -m sglang.launch_server \
  --model-path "$MODEL_META" \
  --tokenizer-path "$MODEL_META" \
  --trust-remote-code \
  --load-format expert_pack \
  --model-loader-extra-config "{\"pack_path\":\"$PACK_PATH\",\"manifest_path\":\"$MANIFEST_PATH\",\"source_path\":\"$GGUF\",\"source_sha256\":\"$SOURCE_SHA256\",\"ollama_manifest_sha256\":\"$OLLAMA_MANIFEST_SHA256\",\"config_sha256\":\"$CONFIG_SHA256\",\"cache_vram_mib\":21504,\"cache_vram_reserve_mib\":2048,\"stage_slots\":12,\"read_splits\":4,\"direct_io\":true,\"stats_flush_interval\":43,\"stats_path\":\"$STATS_PATH\"}" \
  --attention-backend dsv4 \
  --tp-size 1 --ep-size 1 \
  --disable-cuda-graph --disable-flashinfer-autotune \
  --disable-shared-experts-fusion --skip-server-warmup \
  --max-running-requests 1 --mem-fraction-static 0.96 \
  --watchdog-timeout 1800 --host 127.0.0.1 --port 30001

Send the same ten rows to the running Ollama /api/generate endpoint. The retained client uses num_predict=200, temperature=0, the fixed seed, and one request at a time; it writes the per-request JSONL records and summary used by the result table.

9.2 Kimi-K3: SGLang and llama.cpp

Versions and workload

The SGLang run uses the same commit 81c9f837f19ff8dfe1a9fcd1abfc6069dd28d2ec. The baseline uses llama.cpp commit 5fff128451d7603857597ee1fc18ac1dfb90f148. The ten Alpaca/MMLU requests above are sent serially, one at a time, with temperature 0, default EOS handling, and a 200-token target on both runtimes.

Preparation

1. Download the 38 text-only Q2_K GGUF shards from Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED:

   hf download Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED \
     --include "KIMI-K3-MXP4-DERISKED-Q2_K-*.gguf" \
     --local-dir /path/to/kimi-k3
   

2. Download the tokenizer and configuration files from moonshotai/Kimi-K3 at the recorded revision:

   hf download moonshotai/Kimi-K3 \
     config.json tokenizer_config.json generation_config.json \
     tokenization_kimi.py encoding_k3.py tiktoken.model \
     --revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569 \
     --local-dir /path/to/kimi-k3-tokenizer
   

3. Build the Kimi Expert Pack manually. --gguf points to the first numbered shard; the script discovers all 38 shards in that directory. The Kimi model configuration is the downloaded config.json containing text_config:

   cd /path/to/sglang-latest-deepseek-v4-kimi-k3-ssd
   python3 tools/expert_pack/prepare_kimi_pack.py \
     --gguf /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \
     --model-config /path/to/kimi-k3-tokenizer/config.json \
     --safety-margin-gib 2
   

   This creates KIMI-K3-MXP4-DERISKED-Q2_K.expert-major.pack beside the GGUF shards. It is a separate GGML Expert Pack; the validated routed experts use Q2_K for gate/up and Q3_K for down.

4. Create the model metadata and manifest needed by SGLang. This preparation mode does not start the service:

   python3 examples/runtime/kimi_k3/benchmark_kimi_k3_5090.py \
     --gguf /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \
     --max-new-tokens 200 --direct-io --read-splits 1 --prepare-only
   

   It creates model-meta/ and kimi-k3-expert-pack.manifest.json under ${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/kimi-k3/<fingerprint>/. The metadata directory contains the rewritten text configuration and copied tokenizer files. The manifest records the shard inventory, tensor layout, Pack index, model configuration, and tokenizer hashes. A normal benchmark run additionally creates the stats JSON, report JSON, and server log.

Starting SGLang

GGUF_DIR=/path/to/kimi-k3
PACK_PATH="$GGUF_DIR/KIMI-K3-MXP4-DERISKED-Q2_K.expert-major.pack"
ARTIFACT_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/sglang-expert-pack/kimi-k3/<fingerprint>"
MODEL_META="$ARTIFACT_DIR/model-meta"
MANIFEST_PATH="$ARTIFACT_DIR/kimi-k3-expert-pack.manifest.json"
STATS_PATH="$ARTIFACT_DIR/kimi-k3-expert-pack.stats.json"

python3 -m sglang.launch_server \
  --model-path "$MODEL_META" \
  --tokenizer-path "$MODEL_META" \
  --trust-remote-code \
  --load-format expert_pack \
  --model-loader-extra-config "{\"pack_path\":\"$PACK_PATH\",\"manifest_path\":\"$MANIFEST_PATH\",\"cache_vram_mib\":5120,\"cache_vram_reserve_mib\":1536,\"stage_slots\":16,\"read_splits\":1,\"direct_io\":true,\"stats_flush_interval\":92,\"stats_path\":\"$STATS_PATH\",\"verify_pack_sha256\":false}" \
  --tp-size 1 --ep-size 1 \
  --disable-cuda-graph --disable-shared-experts-fusion \
  --disable-radix-cache --mamba-radix-cache-strategy no_buffer \
  --disable-overlap-schedule --skip-server-warmup \
  --chunked-prefill-size 64 --watchdog-timeout 1800 \
  --max-running-requests 1 --mem-fraction-static 0.98 \
  --host 127.0.0.1 --port 30001

Starting the llama.cpp baseline server

Start the pinned llama.cpp build with CPU expert execution:

/path/to/llama.cpp/build/bin/llama-server \
  -m /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \
  -ngl -1 --cpu-moe --host 127.0.0.1 --port 8081 \
  -t 16 -tb 16 --threads-http 16 -np 1 -c 4096 \
  --no-warmup --metrics \
  --log-file /path/to/kimi-k3-llama-cpp/server.log

The llama.cpp client sends the same ten prompts to /completion, one at a time, with cache_prompt=false, temperature=0, and n_predict=200.

10. Conditions and limitations

SSD capacity and preparation time

Expert Pack requires additional SSD capacity. The PR records an estimated 5-10 minutes to build the DeepSeek-V4-Flash pack and approximately 8-15 minutes for first-run readiness. Kimi-K3 pack construction took 29 minutes 42 seconds in the retained measurement, with approximately 35-45 minutes to first readiness. The 38 Kimi source shards and generated Expert Pack occupy about 1.814 TiB in total; the measurements use the TiPro9000 2 TB disk, while a 4 TB SSD is recommended for practical deployment headroom.

Direct I/O

O_DIRECT requires platform support and alignment of file offsets, read lengths, and user-buffer addresses. SGLang fails closed during runtime initialization. If the platform does not support direct I/O, or the pack does not satisfy the alignment contract, the result should not be described as direct-I/O performance.

Cache and workload

• A smaller GPU cache creates more misses, putting SSD reads and H2D transfers on the critical path more often.
• A change in prompt or workload distribution can change the hot experts, so one request's hot set is not guaranteed to fit every workload.
• If the complete expert pool already fits in GPU memory, Expert Pack adds an unnecessary data path and is not the right deployment mode.
• If the workload is dominated by GPU computation, the benefit of removing the host copy may be hidden by compute time.
• If SSD random-read behavior, queueing, or thermal stability is poor, increasing read_splits and staging slots may add queueing and memory pressure instead of throughput.

Current feature boundary

This feature focuses on routed-expert SSD delivery and GPU caching. It does not provide SSD KV-cache offload and does not change request scheduling. It is an explicit opt-in Expert Pack path, not a global replacement for every SGLang model-loading format.

11. Conclusion

SSD Expert Pack is not simply replacing GPU memory with a slower disk. It redesigns weight delivery around the sparse access pattern of MoE inference:

router selects a small set of experts
  -> Expert Pack resolves their offsets
  -> O_DIRECT reads into aligned pinned buffers
  -> expert-level asynchronous H2D fills the GPU cache
  -> the complete expert becomes available for computation

Removing the page cache -> pinned memory copy is an easy detail to miss, but it is a concrete end-to-end optimization. A traditional path reads into the OS page cache and then copies the payload into CUDA-usable pinned memory. With direct I/O, Expert Pack uses the preallocated pinned buffer as the read target and removes that full host-memory movement and synchronization handoff.

This is how SGLang applies the core SSD-LLaMA idea to real DeepSeek-V4-Flash and Kimi-K3 integrations: the complete expert pool remains on high-capacity SSD, a bounded GPU cache retains the current working set, and the runtime moves only the experts selected by the router. Very large MoE models no longer require stacking enough VRAM or DRAM to hold the complete model and can instead run on a consumer GPU paired with a high-speed NVMe SSD.