A small Slurm-oriented dashboard for monitoring multiple Linux servers through SSH.

中文说明见 README_zh.md.

It shows:

• Slurm CPU allocation and running/pending jobs by user
• Real CPU and memory usage from /proc
• Disk I/O latency from iostat
• D-state process counts by user
• Recent 30-day core-hour usage from Slurm accounting, cached in the server process

On the machine running the dashboard:

• Python 3.8+
• SSH access to the monitored servers

On monitored servers:

• Slurm commands: squeue, sinfo
• Optional Slurm accounting: sacct
• Optional job memory command: sstat
• Optional disk latency command: iostat

Copy the example config:

cp config.example.json config.json

Edit config.json:

{
  "default_hosts": ["local", "server1"],
  "local_hosts": ["local", "localhost"],
  "hosts": {
    "server1": {
      "display_name": "server1",
      "hostname": "server1.example.edu",
      "port": 22,
      "user": "your_ssh_user",
      "reserved_idle_cores": 0,
      "cpu_divisor": 1,
      "disk_allowlist": [],
      "disk_status_util_only": false,
      "web_url": ""
    }
  }
}

Field meanings:

• default_hosts: hosts shown in the dashboard input by default.
• local_hosts: host names that should be checked locally instead of through SSH.
• hostname, port, user: SSH connection settings.
• display_name: optional display name in the dashboard.
• reserved_idle_cores: mark the server full only when this many cores remain idle.
• cpu_divisor: use 2 when Slurm reports hyperthreads but you want physical-core units.
• disk_allowlist: only show these disks for that host, for example ["sda"].
• disk_status_util_only: ignore latency thresholds and judge disk status only by %util.
• web_url: optional URL shown below the host address.

You can also point to another config file:

MULTI_SERVER_MONITOR_CONFIG=/path/to/config.json python scripts/serve_squeue_usage_test.py

python scripts/serve_squeue_usage_test.py --host 0.0.0.0 --port 7878

Then open:

http://SERVER_IP:7878/

python scripts/check_squeue_usage_test.py --hosts local server1
python scripts/check_squeue_usage_test.py --json

config.json, cache files, logs, backup files, and Python bytecode are ignored by Git.