◉ Pulse
Infatoshi avatar
#37 today ▲ 7

Infatoshi/laguna-s-vllm

Shell AI

Laguna-S 2.1 NVFP4 with DFlash on a single RTX PRO 6000 Blackwell

Created Jul 22, 2026
Stars
10
Today
+2
Forks
0
Quality
80/100
Momentum
20

README

Fetched from GitHub on Nov 18, 2026, 12:02 PM UTC · README.md

View original on GitHub

Third-party content — Open Source Pulse is not the owner of this repository.

This README belongs to Infatoshi/laguna-s-vllm and its contributors. We fetch it directly from the project’s public GitHub repository and display it for reference only. All text, code, images and trademarks remain the property of their respective authors. See the repository for its license terms. Copyright / takedown policy.

Laguna-S 2.1 NVFP4 on one RTX PRO 6000

This repository is the reproducible setup for serving poolside/Laguna-S-2.1-NVFP4 with its poolside/Laguna-S-2.1-DFlash-NVFP4 draft on a single 96 GB RTX PRO 6000 Blackwell GPU. It pins vLLM 0.25.1, uses the native NVFP4 path, and includes the single-request FlashInfer patch that removed the main host synchronization bubble in our profiles.

The default is deliberately narrow: one request at a time, a 262,144-token context, DFlash K=3, thinking enabled, and Poolside's recommended temperature 0.7 / top-p 0.95 sampling defaults.

See BENCHMARK.md for the no-speculation versus DFlash benchmark.

Requirements

  • Linux on an NVIDIA Blackwell GPU with enough memory for roughly 71 GB of target weights plus the draft, CUDA graphs, and KV cache. This setup was tested on an RTX PRO 6000 Blackwell Workstation Edition with 96 GB.
  • A CUDA 13-capable NVIDIA driver and CUDA JIT toolchain.
  • Python 3.12 headers, uv, patch, and nvidia-smi.
  • Claude Code only if you want to use the optional claude-laguna launcher.

On Ubuntu, install the non-Python prerequisites first:

sudo apt-get update
sudo apt-get install -y python3.12-dev patch
curl -LsSf https://astral.sh/uv/install.sh | sh

Install and serve

From a fresh clone:

./scripts/setup.sh
./scripts/serve.sh

setup.sh creates .venv, installs vLLM 0.25.1 against the CUDA 13.0 PyTorch wheels, installs the matching FlashInfer nightly trio, and applies patches/vllm-0.25.1-single-request-gpu-metadata.patch to the installed vLLM package. It is safe to rerun.

Validate an existing installation without changing it:

./scripts/setup.sh --check

The first serve downloads the model weights and builds any missing JIT kernels. Keep MAX_JOBS=4 on a cold FlashInfer cache. The serve script sets it by default because an uncapped compilation fan-out can exhaust host memory.

Useful overrides are environment variables:

LAGUNA_GPU=0 \
LAGUNA_HOST=127.0.0.1 \
LAGUNA_PORT=8000 \
LAGUNA_API_KEY=laguna-local \
LAGUNA_SPECULATIVE_TOKENS=3 \
./scripts/serve.sh

The official model card suggests larger speculative-token counts for its benchmark mix. K=3 is the retained default here because it won on our temperature-0.7 coding-agent prompts on this specific concurrency-one GPU. Override it if your acceptance distribution is different.

Use from Claude Code

Run the included launcher instead of modifying ~/.claude/settings.json:

./scripts/claude-laguna --dangerously-skip-permissions

The endpoint and model environment variables exist only for that process. Ordinary claude sessions keep their normal provider and model configuration.

To make it available as a command:

mkdir -p ~/.local/bin
ln -s "$PWD/scripts/claude-laguna" ~/.local/bin/claude-laguna
claude-laguna --dangerously-skip-permissions

For a GPU server reached over SSH, keep vLLM bound to localhost and tunnel it:

ssh -N -L 8000:127.0.0.1:8000 gpu-host
claude-laguna --dangerously-skip-permissions

Or point the launcher at an authenticated LAN endpoint:

LAGUNA_BASE_URL=http://gpu-host:8000 \
LAGUNA_API_KEY='replace-with-a-real-key' \
claude-laguna --dangerously-skip-permissions

What the patch changes

With speculative rejection, the exact next sequence length is produced on the GPU. The stock single-request FlashInfer path read that scalar back to the host before planning the next attention launch. CUDA graphs and fused model kernels could not hide that dependency.

For the safe single-request case, the patch uses the CPU optimistic upper bound for FlashInfer's host launch plan, then writes the exact device sequence length into FlashInfer's runtime metadata with a small Triton kernel before attention consumes it. Target and draft both take this path. It does not replace the model kernels; it removes the gap between them.

In the controlled temperature-0.7 A/B on the RTX PRO 6000, the patch reduced GPU-empty time per engine step from 2.624 ms to 0.839 ms and improved output throughput from 145.122 to 155.167 tokens/s. The model-kernel GPU-active time was effectively unchanged. Treat those as measurements of this hardware and workload, not a general throughput promise.

The optimization is guarded to num_reqs == 1, no decode-context parallelism, and no cascade attention. This repository keeps --max-num-seqs 1 so the serve configuration matches the profiled case.

Direct API smoke

curl http://127.0.0.1:8000/v1/models \
  -H 'Authorization: Bearer laguna-local'

Stop the foreground server with Ctrl-C. The model is text-only; requests that contain images are not supported.

Rendered by GitHub from the repository’s README. Scroll to read the full file.

First trending

Jul 23, 2026

first spotted on the list

Best rank

#37

highest position reached

Appearances

3

times on the trending list

Star velocity

0.08

stars per day since creation

Activity — last 60 days

daily counts, collected every 6 hours
  • Stars
  • Forks
  • Merged PRs
  • Issues
  • Closed issues

Collecting activity data since launch — check back in a few days.

Star history

total stars per snapshot
567891011Infatoshi/laguna-s-vllm · Jul 23, 8 AM UTC: 6 starsInfatoshi/laguna-s-vllm · Jul 23, 2 PM UTC: 8 starsInfatoshi/laguna-s-vllm · Jul 23, 7 PM UTC: 10 stars8 AM1 PM7 PM

Rank history

position on trending · lower is better
#1#35#40#45#50#55#60#65Jul 23, 8 AM UTC: rank #55Jul 23, 2 PM UTC: rank #64Jul 23, 7 PM UTC: rank #378 AM1 PM7 PM

3D card for your website

Live HTML card with a pointer tilt. Paste the iframe on any website.

<iframe src="https://www.notionpulse.cc/embed/Infatoshi/laguna-s-vllm" width="420" height="240" style="border:0;max-width:100%;background:transparent" loading="lazy" title="Infatoshi/laguna-s-vllm — trending on Open Source Pulse"></iframe>

Stars badge

Show current stars in your README or profile. Auto-updates every 6 hours.

Infatoshi/laguna-s-vllm stars badge
Markdown
[![Infatoshi/laguna-s-vllm](https://www.notionpulse.cc/api/badge/Infatoshi/laguna-s-vllm.svg)](https://www.notionpulse.cc/repo/Infatoshi/laguna-s-vllm)
HTML
<a href="https://www.notionpulse.cc/repo/Infatoshi/laguna-s-vllm"><img src="https://www.notionpulse.cc/api/badge/Infatoshi/laguna-s-vllm.svg" alt="Infatoshi/laguna-s-vllm stars badge" /></a>

Repository of the Day

#37 Repository of the Day

Repository of the Week

#37 Repository of the Week

Repository of the Month

#37 Repository of the Month