◉ Pulse
MiaAI-Lab avatar
#66 today ▲ 10

MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO

Shell AI

vLLM 0.25.1 serving stack for poolside/Laguna-S-2.1-NVFP4 with DFlash speculative decoding — DGX Spark & RTX 6000 PRO

Created Jul 21, 2026
Stars
40
Today
+7
Forks
4
Quality
83/100
Momentum
22

README

Fetched from GitHub on Nov 16, 2026, 3:29 PM UTC · README.md

View original on GitHub

Third-party content — Open Source Pulse is not the owner of this repository.

This README belongs to MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO and its contributors. We fetch it directly from the project’s public GitHub repository and display it for reference only. All text, code, images and trademarks remain the property of their respective authors. See the repository for its license terms. Copyright / takedown policy.

Laguna-S-2.1-NVFP4 · DFlash · vLLM Serving

Serves poolside/Laguna-S-2.1-NVFP4 — a 117.6B-parameter MoE model with NVFP4 quantization (~71 GB) — accelerated by DFlash speculative decoding via its paired draft model poolside/Laguna-S-2.1-DFlash-NVFP4.

Runs in Docker on vLLM 0.25.1. Designed for the DGX Spark (NVIDIA GB10, sm_121a).

Buy Me a Coffee at ko-fi.com Follow Mia on X

---

Hardware Requirements

Component Requirement
GPU NVIDIA GB10 (DGX Spark) — architecture sm_121a
Memory 128 GB unified (weight footprint ≈71 GB target + ≈2 GB draft)
Storage ~150 GB free for model cache (~/.cache/huggingface/hub)
Architecture aarch64 (ARM64)

Prerequisites

  • Docker with NVIDIA container toolkit (nvidia-ctk installed and configured)
  • curl
  • HuggingFace CLI (optional — for faster model downloads):
    pip install huggingface-cli
    or the newer hf tool:
    pip install huggingface-hub
  • HuggingFace token with access to poolside/Laguna-S-2.1-NVFP4 (set HF_TOKEN if the model is gated)

Quick Start

# 1. Download both models (one-time, ~150 GB total)
export HF_TOKEN="hf_..."   # only if the model is gated
./start.sh --download-only

# 2. Start the server
./start.sh

# 3. Stop the server
./stop.sh

On first start the server takes ≈15 minutes to become ready (weight load from NVMe, FlashInfer JIT kernel compilation, and CUDA graph capture). Subsequent starts are faster if the HuggingFace cache is preserved.


Scripts

start.sh

Launches or manages the vLLM serving container.

./start.sh                         # download if needed → start server
./start.sh --download-only         # download models only, then exit
./start.sh -h                      # show help

What it does:

  1. Checks ~/.cache/huggingface/hub for both models; downloads any that are missing (using hf → huggingface-cli → Docker fallback)
  2. Removes any stale container named laguna-s-2.1-nvfp4
  3. Pulls the Docker image (vllm/vllm-openai:v0.25.1)
  4. Starts the container with a bootstrap script that installs the matched FlashInfer nightly trio (when the stock image version differs) for FP4 kernel support, and mounts a persistent FlashInfer JIT cache
  5. Waits for the /v1/models endpoint to return HTTP 200
  6. Prints the OpenAI-compatible base URL

stop.sh

Stops and removes the container.

./stop.sh

API Endpoint

Once running, the server exposes an OpenAI-compatible API:

http://localhost:8888/v1

Example

curl http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "poolside/Laguna-S-2.1-NVFP4",
    "messages": [
      {"role": "user", "content": "Write a quick sort in Python."}
    ]
  }'

Configuration

Models

Variable Default Description
MODEL_ID poolside/Laguna-S-2.1-NVFP4 Target (main) model
DRAFT_MODEL_ID poolside/Laguna-S-2.1-DFlash-NVFP4 DFlash speculative draft model

Serving

Flag Value Notes
--host / --port 0.0.0.0:8888
--tensor-parallel-size 1 Single GPU (GB10)
--dtype bfloat16 Compute dtype for the NVFP4 path
--attention-backend FLASHINFER Locks FlashInfer attention (do not force flashinfer_b12x)
--gpu-memory-utilization 0.85 Leaves margin for JIT + graph capture
--max-model-len 262144 256K context window
--max-num-seqs 4 Required — DFlash crashes at the default of 256; 4 is a safe starting point for single-user use
--max-num-batched-tokens 8192 Higher values (e.g. 16384) can leave no KV blocks
--enable-auto-tool-choice Structured output / tool calling
--tool-call-parser poolside_v1
--reasoning-parser poolside_v1
--default-chat-template-kwargs {"enable_thinking":true} Reasoning on by default; clients can disable per request
--override-generation-config {"temperature":0.7,"top_p":0.95,"top_k":20} Recommended sampling defaults; top_k:20 matches Poolside's eval-certified truncation
--speculative-config {"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":7,"method":"dflash"} DFlash with 7 speculative tokens

Do not add --linear-backend flashinfer_b12x — it is broken on vLLM 0.25.1 and slower than auto-selected FlashInferCutlass.

Do not include min_p in --override-generation-config — vLLM rejects it under speculative decoding (400 error on every request).

KV cache dtype is FP8 (torch.float8_e4m3fn), auto-selected by FlashInfer — no need to pass --kv-cache-dtype.

Environment Variables

Variable Default Description
HF_TOKEN (empty) HuggingFace token for gated models
IMAGE vllm/vllm-openai:v0.25.1 Docker image override
MAX_JOBS 4 Cap for JIT compilation fan-out (prevents OOM on cold cache)
NVCC_THREADS 2 Cap nvcc parallelism during FlashInfer JIT
FLASHINFER_NVCC_THREADS 2 FlashInfer-specific nvcc thread cap
FLASHINFER_CACHE_DIR ~/.cache/flashinfer Host path mounted at /root/.cache/flashinfer (FlashInfer's default JIT cache path) so compiled kernels persist across restarts

First Start (≈15 Minutes)

The first launch includes:

  1. Weight loading — 66.98 GB of NVFP4 weights from disk
  2. JIT compilation — FlashInfer autotuner profiles ~21 CUDA kernels for sm_121a
  3. CUDA graph capture — Optimizes small-batch execution paths

You will see log output like:

[APIServer pid=1] WARNING ... registry.py ... LagunaForCausalLM  (harmless, resolved by --trust-remote-code)
[EngineCore pid=...] Available KV cache memory: 32.44 GiB
[EngineCore pid=...] flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) ...

The autotuner progress bar (7/21 [00:03<00:10, 1.40profile/s]) is normal.

KV Cache (with DFlash active)

Metric Value
Available KV cache ~32.44 GB
KV cache capacity 926,683 tokens
Max concurrency @ 262k ctx ~3.54× (3 full-length sequences with headroom)
KV cache dtype FP8 (torch.float8_e4m3fn) — auto-selected by FlashInfer
GPU memory utilization 85 % (effective ≈84.4 % with CUDA graph profiling)
Concurrent sequences 4 max

Performance

Decode throughput with DFlash speculative decoding (7 speculative tokens) on DGX Spark (GB10):

Decode throughput benchmark


Running on RTX 6000 PRO (Blackwell, 96 GB)

The recipe works on an RTX 6000 PRO with a few changes:

Setting DGX Spark (GB10) RTX 6000 PRO
Architecture aarch64 x86_64 (native Docker support)
GPU arch sm_121a sm_120 (Blackwell)
VRAM 128 GB unified 96 GB GDDR7
CUTE_DSL_ARCH sm_121a Needs Blackwell sm value (e.g. sm_120)
--gpu-memory-utilization 0.85 Start at 0.78 — model ~71 GB + draft + KV cache leaves less headroom on 96 GB
--max-num-seqs 4 Can likely be raised (start at 8, tune up)

The Docker image vllm/vllm-openai:v0.25.1 has x86_64 builds and works on Blackwell without modification. Set CUTE_DSL_ARCH to the correct Blackwell compute capability before starting.


Troubleshooting

Symptom Likely Cause Fix
Error in inspecting model architecture 'LagunaForCausalLM' Registry subprocess missing --trust-remote-code Set -e VLLM_TRUST_REMOTE_CODE=1 (already in the Docker run)
FlashInfer install fails / version mismatch Nightly trio unavailable or partial upgrade Bootstrap fails closed; check network access to flashinfer.ai nightlies
Container exits with OOM JIT fan-out too large Set MAX_JOBS=2 (and/or NVCC_THREADS=1)
docker: Conflict error Stale container Run ./stop.sh or docker rm -f laguna-s-2.1-nvfp4
Slow first start JIT + graph capture Normal; takes ≈15 minutes. Subsequent starts reuse ~/.cache/flashinfer

References

Rendered by GitHub from the repository’s README. Scroll to read the full file.

First trending

Jul 22, 2026

first spotted on the list

Best rank

#8

highest position reached

Appearances

3

times on the trending list

Star velocity

0.34

stars per day since creation

Activity — last 60 days

daily counts, collected every 6 hours
  • Stars
  • Forks
  • Merged PRs
  • Issues
  • Closed issues

Collecting activity data since launch — check back in a few days.

Star history

total stars per snapshot
15202530354045MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO · Jul 22, 2 AM UTC: 21 starsMiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO · Jul 22, 8 AM UTC: 33 starsMiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO · Jul 22, 2 PM UTC: 40 stars2 AM8 AM2 PM

Rank history

position on trending · lower is better
#1#10#20#30#40#50#60#70Jul 22, 2 AM UTC: rank #8Jul 22, 8 AM UTC: rank #18Jul 22, 2 PM UTC: rank #662 AM8 AM2 PM

3D card for your website

Live HTML card with a pointer tilt. Paste the iframe on any website.

<iframe src="https://www.notionpulse.cc/embed/MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO" width="420" height="240" style="border:0;max-width:100%;background:transparent" loading="lazy" title="MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO — trending on Open Source Pulse"></iframe>

Stars badge

Show current stars in your README or profile. Auto-updates every 6 hours.

MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO stars badge
Markdown
[![MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO](https://www.notionpulse.cc/api/badge/MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO.svg)](https://www.notionpulse.cc/repo/MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO)
HTML
<a href="https://www.notionpulse.cc/repo/MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO"><img src="https://www.notionpulse.cc/api/badge/MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO.svg" alt="MiaAI-Lab/Laguna-S-2.1-DGX-Spark-RTX-6000-PRO stars badge" /></a>

Repository of the Day

#8 Repository of the Day

Repository of the Week

#8 Repository of the Week

Repository of the Month

#8 Repository of the Month