This repository is the reproducible setup for serving
poolside/Laguna-S-2.1-NVFP4
with its
poolside/Laguna-S-2.1-DFlash-NVFP4
draft on a single 96 GB RTX PRO 6000 Blackwell GPU. It pins vLLM 0.25.1,
uses the native NVFP4 path, and includes the single-request FlashInfer patch
that removed the main host synchronization bubble in our profiles.
The default is deliberately narrow: one request at a time, a 262,144-token context, DFlash K=3, thinking enabled, and Poolside's recommended temperature 0.7 / top-p 0.95 sampling defaults.
See BENCHMARK.md for the no-speculation versus DFlash benchmark.
- Linux on an NVIDIA Blackwell GPU with enough memory for roughly 71 GB of target weights plus the draft, CUDA graphs, and KV cache. This setup was tested on an RTX PRO 6000 Blackwell Workstation Edition with 96 GB.
- A CUDA 13-capable NVIDIA driver and CUDA JIT toolchain.
- Python 3.12 headers,
uv,patch, andnvidia-smi. - Claude Code only if you want to use the optional
claude-lagunalauncher.
On Ubuntu, install the non-Python prerequisites first:
sudo apt-get update
sudo apt-get install -y python3.12-dev patch
curl -LsSf https://astral.sh/uv/install.sh | shFrom a fresh clone:
./scripts/setup.sh
./scripts/serve.shsetup.sh creates .venv, installs vLLM 0.25.1 against the CUDA 13.0
PyTorch wheels, installs the matching FlashInfer nightly trio, and applies
patches/vllm-0.25.1-single-request-gpu-metadata.patch
to the installed vLLM package. It is safe to rerun.
Validate an existing installation without changing it:
./scripts/setup.sh --checkThe first serve downloads the model weights and builds any missing JIT kernels.
Keep MAX_JOBS=4 on a cold FlashInfer cache. The serve script sets it by
default because an uncapped compilation fan-out can exhaust host memory.
Useful overrides are environment variables:
LAGUNA_GPU=0 \
LAGUNA_HOST=127.0.0.1 \
LAGUNA_PORT=8000 \
LAGUNA_API_KEY=laguna-local \
LAGUNA_SPECULATIVE_TOKENS=3 \
./scripts/serve.shThe official model card suggests larger speculative-token counts for its benchmark mix. K=3 is the retained default here because it won on our temperature-0.7 coding-agent prompts on this specific concurrency-one GPU. Override it if your acceptance distribution is different.
Run the included launcher instead of modifying ~/.claude/settings.json:
./scripts/claude-laguna --dangerously-skip-permissionsThe endpoint and model environment variables exist only for that process.
Ordinary claude sessions keep their normal provider and model configuration.
To make it available as a command:
mkdir -p ~/.local/bin
ln -s "$PWD/scripts/claude-laguna" ~/.local/bin/claude-laguna
claude-laguna --dangerously-skip-permissionsFor a GPU server reached over SSH, keep vLLM bound to localhost and tunnel it:
ssh -N -L 8000:127.0.0.1:8000 gpu-host
claude-laguna --dangerously-skip-permissionsOr point the launcher at an authenticated LAN endpoint:
LAGUNA_BASE_URL=http://gpu-host:8000 \
LAGUNA_API_KEY='replace-with-a-real-key' \
claude-laguna --dangerously-skip-permissionsWith speculative rejection, the exact next sequence length is produced on the GPU. The stock single-request FlashInfer path read that scalar back to the host before planning the next attention launch. CUDA graphs and fused model kernels could not hide that dependency.
For the safe single-request case, the patch uses the CPU optimistic upper bound for FlashInfer's host launch plan, then writes the exact device sequence length into FlashInfer's runtime metadata with a small Triton kernel before attention consumes it. Target and draft both take this path. It does not replace the model kernels; it removes the gap between them.
In the controlled temperature-0.7 A/B on the RTX PRO 6000, the patch reduced GPU-empty time per engine step from 2.624 ms to 0.839 ms and improved output throughput from 145.122 to 155.167 tokens/s. The model-kernel GPU-active time was effectively unchanged. Treat those as measurements of this hardware and workload, not a general throughput promise.
The optimization is guarded to num_reqs == 1, no decode-context parallelism,
and no cascade attention. This repository keeps --max-num-seqs 1 so the serve
configuration matches the profiled case.
curl http://127.0.0.1:8000/v1/models \
-H 'Authorization: Bearer laguna-local'Stop the foreground server with Ctrl-C. The model is text-only; requests that contain images are not supported.