LLM Systems Manager — Deployment Guide

This guide walks you through installing, configuring, and maintaining LLM Systems Manager on your servers.

Prerequisites

Manager Server

• Operating system: Ubuntu 22.04 or later (other modern Linux distributions should work). The script installer also supports RHEL-family 10 hosts (Rocky Linux, AlmaLinux, RHEL 10) and installs packages with dnf; on version 9 of these systems, use the .rpm package.
• Python: 3.11 or later (the installer refuses to proceed below this)
• RAM: 2 GB minimum; 4 GB recommended
• Disk: At least 10 GB free for logs, metrics history, and model benchmark data

Remote Agent Hosts

Agents can run on:

• Linux — Ubuntu 22.04+ or equivalent
• macOS — macOS 13 (Ventura) or later

The script installer needs Python 3.10+ on the agent host. The native packages and the binary tarball bundle a self-contained agent and need no Python at all.

Each agent host needs network access to ports 8081 (alarm engine) and 5000 or 5443 (manager).

Supported Platforms

The manager, alarm engine, and InfluxDB are tested on Debian and Ubuntu derivatives and on Rocky Linux 10.

• RHEL-family 10 (Rocky Linux, AlmaLinux, RHEL 10): every mode works, and the installer installs the packages it needs with dnf. Hosts running version 9 of these systems use the .rpm package instead.
• Other Linux distros (Fedora, Arch, openSUSE, Alpine): the agent (mode 5) auto-detects dnf / yum / brew and works out of the box. The manager / alarm engine / InfluxDB modes (1–4, 6) stop at the prerequisites step with a hint for your package manager; install the listed packages by hand, then re-run.
• macOS (Apple Silicon, tested on M2 Pro): agent with the script installer; the dashboard through Homebrew.
• Windows is not supported.

The installer checks for: python3 (≥ 3.11), python3-venv, git, jq, curl, and rsync.

Required Ports

The following (configurable) ports must be reachable between the components listed. Browsers that use the Tools tab also need 5447 or 5448.

Installing the Full Stack

For a quick installation install on one host, choose the full install option:

Step 1:

bash <(curl -fsSL https://get.llmsysmanager.com/install.sh)

Step 2: Choose an Installation Mode

You will see a menu like this:

  Select the deployment option:

    1)  Full system          manager + alarm engine + agent + InfluxDB
    2)  Manager + alarm      manager + alarm engine (existing InfluxDB)
    3)  Manager only         Flask manager + dashboard
    4)  Alarm engine only    standalone FastAPI alarm engine
    5)  Agent only           Linux + macOS host agent
    6)  InfluxDB only        InfluxDB v2 + scoped tokens (DB host)
    7)  Update installed     detect, diff, backup, sync-only-changed, restart
    8)  Uninstall            remove all services and files, with confirmation prompts
    9)  Quit                 exit with no changes

Mode [1-9]:

Choose option 1 for a first-time setup on a single server. This installs everything you need in one step.

Step 3: Answer Configuration Prompts

The installer will ask a few questions:

• Manager address — the IP address or hostname other machines will use to reach this server
• Admin password — the password for the initial admin account. It is stored as a secure hash, never in plain text.
• SMTP settings (optional) — email address and credentials for alarm notifications
• InfluxDB details — the installer can provision InfluxDB automatically in mode 1

On a manager-only (mode 3) install, the installer also asks for the alarm engine's management_token. This prompt cannot be skipped: paste the value mode 4 printed on the alarm-engine host, type new to mint one here and copy it over, or press Enter to keep a value carried over from a previous config. Non-interactive installs can supply it via LLMSYS_CFG_AE_MGMT_TOKEN.

If you are unsure about any optional setting, you can accept the default and change it later in the configuration file.

Step 4: Start the Services

The installer enables the services but does not start them — you decide when to bring them up. Start all three services:

sudo systemctl start llm-systems-manager
sudo systemctl start llm-systems-alarm-engine
sudo systemctl start llm-systems-agent

Verify each one is running:

sudo systemctl status llm-systems-manager
sudo systemctl status llm-systems-alarm-engine
sudo systemctl status llm-systems-agent

Each should show active (running). If a service failed to start, check the logs (see Viewing Logs below).

Step 5: Open the Dashboard

Open a browser and go to:

http://<your-server-address>:5000

Sign in with the username and password you chose during the install. If you pressed ENTER at the password prompt, the installer created a temporary password and printed it under Next steps when it finished:

Sign in as:           llmadmin
Temporary password:   <shown once>

The first sign-in with a temporary password opens a change-password form; set your own password there (8 characters minimum) to reach the dashboard. Afterwards, change it any time from the settings cog → Change my password.

The temporary password is shown only once. If you lose it, or the login page says the admin password is not set, create a new one on the manager host:

sudo /opt/llm-systems-manager/llm-systems-manager/venv/bin/python \
  /opt/llm-systems-manager/llm-systems-manager/backend/admin_password.py reset

It asks before changing anything, then prints a new temporary password. Add --user <name> to reset a different account. After several failed sign-in attempts the account is locked for a while; wait for the lockout to pass or restart the manager before using the new password.

What the script does on your machine, step by step: What the install script does.

Choosing the release

The installer deploys the latest GitHub Release, a source tarball whose SHA-256 checksum is verified before anything is installed; a mismatch aborts. To pin a specific version, or to track the development tip from a git clone of main instead (code that hasn't been cut into a release yet):

bash <(curl -fsSL https://get.llmsysmanager.com/install.sh) --ref v1.0.0     # pin a specific release
bash <(curl -fsSL https://get.llmsysmanager.com/install.sh) --source git     # track unreleased main (advanced)

The same --ref / --source flags apply to every install mode and to --update. The short address serves the same script as https://raw.githubusercontent.com/llmsyscore/llm-systems-manager/main/tools/installer/install.sh, which still works.

Offline / air-gapped install

Hosts with no access to GitHub can install from a release tarball staged out-of-band. On a connected machine, download llm-systems-manager-<tag>.tar.gz from the Releases page (verify it against the published .sha256 yourself — the offline path trusts the tree you hand it). Copy it to the target host, then:

tar -xzf llm-systems-manager-v1.0.0.tar.gz
sudo bash llm-systems-manager-v1.0.0/tools/installer/install.sh --source local

--source local installs the extracted tree the script lives in: no release download, no git clone, no installer self-update (git itself is not required on the target host). It works with every install mode and with --update (offline update of an existing install). Only GitHub access is eliminated: installing system packages and the Python virtualenvs still uses apt and pip, so a fully air-gapped host needs local mirrors for those (or pre-provisioned dependencies).

Installing from Native Packages (.deb / .rpm)

Each GitHub Release also ships .deb and .rpm packages: llm-systems-manager installs the manager + alarm engine (the mode-2 layout: InfluxDB stays external, agents install separately), and per-arch llm-systems-agent packages install the self-contained agent binary. The script installer remains the preferred, fully automated path — packages exist for hosts managed through apt/dnf tooling. Download the package for your distro from the Releases page, then:

Debian / Ubuntu:

sudo apt install ./llm-systems-manager_<version>_all.deb

The install prompts (via debconf) for the dashboard admin login and SMTP settings; press ENTER to accept defaults. Non-interactive installs (DEBIAN_FRONTEND=noninteractive) take the defaults silently. When no admin password is given, the package prints the command that creates a temporary one (the admin_password.py reset command in Step 5).

RHEL / Rocky / Alma / Fedora:

sudo dnf install ./llm-systems-manager-<version>-1.noarch.rpm

RPM installs are non-interactive: config is generated with detected defaults at /opt/llm-systems-manager/config/llm-systems.toml — edit it and sudo systemctl restart llm-systems-manager afterwards. The package prints the command that creates a temporary admin password (the admin_password.py reset command in Step 5). EL9's default python3 is 3.9; install python3.11 (sudo dnf install python3.11 python3.11-pip) first — the package picks the newest Python ≥ 3.11 automatically.

Both manager packages create the llmsys runtime user, install and start the two systemd units, and build the Python venvs at configure time (network access to PyPI is required during install). On upgrades the live config is preserved (new keys are merged in). apt purge llm-systems-manager removes everything — config, data, logs, and the runtime user — when the package created the tree; state it didn't create is kept (see Mixing install methods). dnf remove always keeps config/data behind with a notice.

InfluxDB: the package declares influxdb2 only as a Recommends — it lives in InfluxData's third-party repo (not distro repos) and may legitimately run on another host, so a hard dependency would break both cases. If InfluxDB isn't reachable after install, the postinst prints a notice pointing at tools/installer/install-influxdb.sh (local install) or the [influxdb] config section (external server). Metric history and alarms need it; the dashboard runs without it in the meantime. While [influxdb.tokens] still holds its REPLACE_ME placeholders, the alarm engine is enabled but not started — the postinst prints the steps (running install-influxdb.sh prints the tokens to paste); after setting the tokens, systemctl start llm-systems-alarm-engine.

Agent package:

sudo apt install ./llm-systems-agent_<version>_amd64.deb        # or _arm64
sudo dnf install ./llm-systems-agent-<version>-1.x86_64.rpm     # or .aarch64

The deb prompts (debconf) for the manager URL; the rpm takes defaults — set MANAGER_URL in /opt/llm-systems-agent/agent_config.yaml and restart if left blank. The binary is installed owned by llmsys so manager-driven self-update (Admin → Agents → Update) keeps working; after a self-update the on-disk binary is newer than the package until the next apt/dnf upgrade re-syncs it. Provider toggles (llama.cpp/LM Studio/vLLM control, sudo wrappers) are what the script installer automates — enable them in agent_config.yaml per its inline docs.

Packages are built by tools/packaging/build-packages.sh and tools/packaging/build-agent-package.sh (fpm) — see those scripts for the build-from-source path.

Anonymous reports

The installer asks once, after the mode picker, whether to send the anonymous install report and shows it first. Saying yes also allows crash reports and the daily check-in. On a new install the install report, crash reports and daily check-in default to yes, the daily diagnostic report to no. An upgrade keeps the answers already in the config. Reports go to a private service run by llmsyscore; its code lives in a private llmsyscore GitHub repository.

Every report also carries a random install ID created at install time and the install date. Reports do not include names, addresses, paths, tokens or model names. The stored reports do not contain the sending address. The hosting provider keeps ordinary request logs for a short time. The last report that left the machine is stored in data/telemetry-last.json, and Settings › Telemetry & analytics › View shows it. Deleting data/install-id gives the machine a new random ID.

Turning reports off or on:

• Unattended installs: set LLMSYS_TELEMETRY=0|1, which covers the install report, crash reports and the daily check-in, and LLMSYS_HEARTBEAT=0|1. Homebrew installs read HOMEBREW_LLMSYS_TELEMETRY=0 instead, which turns off the install and crash reports and the daily check-in in the new config. You can also switch them off in Settings.
• Debian packages ask the package question. RHEL packages read LLMSYS_TELEMETRY, LLMSYS_HEARTBEAT and DO_NOT_TRACK, and otherwise print one line saying the report is on.
• Docker reads LSM_TELEMETRY and LSM_HEARTBEAT from .env. Docker and Homebrew installs are reported by the manager on its first start.
• Nothing is sent when a CI variable is set.
• DO_NOT_TRACK=1 (any value other than empty, 0 or false) turns every report off in the manager, alarm engine, agent and installer. The services do not see a variable set in a terminal. For the services, run sudo systemctl edit llm-systems-manager (and the same for the alarm engine unit and llm-systems-agent), add [Service] and Environment=DO_NOT_TRACK=1, then restart. For Docker, put DO_NOT_TRACK=1 in .env. For the installers, run sudo DO_NOT_TRACK=1 bash install.sh, because sudo drops the variable otherwise. Homebrew does not pass the variable through, so use HOMEBREW_LLMSYS_TELEMETRY=0 at install time and the Settings switches afterwards.
• Updating from a release without these settings: the updater asks once from a terminal; unattended updates and package or Homebrew upgrades keep reports on and print one line naming the off switch.
• Later changes: Settings › Telemetry & analytics on the dashboard, or the phone companion's Settings screen. It has switches for install and update reports, crash reports, the daily check-in and the daily diagnostic report. The Sending line names the reports that are on and links to the Field list. It also shows when DO_NOT_TRACK is set. The Send now button next to the last report sends a diagnostic report immediately.

Mixing install methods

Install methods do not mix on one host — the script installer, the native packages, Docker, and the agent binary tarball each own the install tree, the systemd units, and the llmsys user differently, and mixing them shadows units or desyncs the package database. Both sides now guard against it:

• The packages refuse a fresh install over script-installer state (units in /etc/systemd/system, a config the package didn't create, a venv in the agent tree) and over busy service ports (a Docker control plane or script install still running). Override: LLMSYS_PACKAGE_FORCE=1 in the environment — the tree is then marked adopted and a later apt purge keeps config/data instead of deleting them.
• The script installer, updater, and uninstaller refuse a package-managed host and point at apt/dnf instead (the updater skips just the agent when only the agent is packaged). Override: LLMSYS_IGNORE_NATIVE_PACKAGE=1.
• apt purge only deletes config/data/logs when the package created the tree and no script-installer state appeared since; otherwise it keeps them and says so.

Supported migrations:

• Script install → package: tools/installer/uninstall.sh first, then install the package (fresh config), or LLMSYS_PACKAGE_FORCE=1 to adopt in place (config preserved).
• Binary tarball → agent package: supported directly — a bare binary + config with no hand-made unit is adopted automatically (config/token preserved, binary replaced by the packaged one).
• Package → script install: apt purge / dnf remove first, then run the script installer.
• A self-updated agent binary (Admin → Agents → Update) is newer than what the package database recorded; a later package upgrade replaces it and warns if that was a downgrade — re-update from the dashboard or install a newer package.

Installing with Docker (control plane)

Multi-arch images for the manager and alarm engine are published to ghcr.io on every release. No repo checkout is needed: download docker-compose.yml and .env.example, fill in the InfluxDB password and token (the alarm engine tokens are generated for you when left blank), and docker compose up -d brings up the manager + alarm engine + InfluxDB together. See docker/README.md for the full walkthrough. Agents still install natively on each monitored host (they need sensor/GPU/systemd access).

Installing with Homebrew (control plane)

The tap also carries control-plane formulas for macOS (Apple Silicon) and Linux:

brew tap llmsyscore/tap && brew trust llmsyscore/tap
brew install llm-systems-manager llm-systems-alarm-engine influxdb@2 influxdb-cli

• Config is shared at $(brew --prefix)/etc/llm-systems-manager/llm-systems.toml, seeded on first install with generated alarm-engine ingest/management tokens (equivalent to a co-located script install). It is kept across upgrades: your values stay, settings a new release introduces are added with their defaults and comments, and the previous file is kept next to it as llm-systems.toml.bak.<timestamp> whenever an upgrade changed it.
• Anonymous reports: install with HOMEBREW_LLMSYS_TELEMETRY=0 brew install … to start with the install and crash reports turned off.
• State (internal CA, agent registry, TLS certs, SQLite DBs) lives under $(brew --prefix)/var/llm-systems-manager/ and also survives upgrades — the kegs hold only code + venvs.
• InfluxDB: run llm-systems-influx-setup (installed by the manager formula) — it starts the influxdb@2 service, onboards it on first boot, creates the buckets + scoped tokens, and writes them into [influxdb.tokens]. It needs both influxdb@2 (the v2 server — Homebrew's plain influxdb formula is InfluxDB 3.x, whose API this stack does not speak) and influxdb-cli (the influx command ships in the separate influxdb-cli formula). Manual alternative: brew services start influxdb@2, influx setup, then create the buckets/tokens per Configuration and fill [influxdb] + [influxdb.tokens] yourself. The manager and alarm engine run without it, but history and alert evaluation stay degraded until then.
• First sign-in: run llm-systems-admin-password reset (installed by the manager formula). It prints a temporary admin password; sign in with it and the dashboard asks you to replace it. The login page shows the same step when opened from the manager's own machine.
• Start order: brew services start llm-systems-manager first (first boot creates the internal CA and issues ae-tls.{crt,key} for the alarm engine), then brew services start llm-systems-alarm-engine.
• Don't mix with a script/package install on the same host — both would fight over ports 5000/8081.

Installing Agents on Remote Computers

If you already have a manager running and want to start monitoring an additional server, install only the agent on that remote machine. The script installer below is the preferred path; two alternatives exist for hosts where it doesn't fit:

• Native package (Linux, no Python needed): apt/dnf install of the per-arch llm-systems-agent package — see Installing from Native Packages.
• Binary tarball (Linux or macOS, no Python needed): each release ships llm-systems-agent-<platform>.tar.gz bundling the self-contained binary, agent_config.yaml.example, and the service unit template — extract to /opt/llm-systems-agent, set MANAGER_URL in a copied agent_config.yaml, install the unit, and systemctl enable --now llm-systems-agent. Full steps in Agent binary below.
• Homebrew (macOS/Apple Silicon, or Linux x86_64/arm64 with Homebrew): brew tap llmsyscore/tap && brew trust llmsyscore/tap && brew install llm-systems-agent, then set MANAGER_URL in $(brew --prefix)/etc/llm-systems-agent/agent_config.yaml and brew services start llm-systems-agent (launchd on macOS, systemd user unit on Linux). brew upgrade tracks new releases automatically. Full steps in Agent with Homebrew below.

Step 1: Get the Installer on the Remote Host

Copy just tools/installer/install.sh from an existing manager installation using scp or another file-transfer method, then run bash install.sh from the directory you copied them into.

The agent installer works on both Linux and macOS. It will ask for the manager server address so the agent knows where to register.

The manager's Admin › Agents page (Manage ▾ → Enrollment) shows two values for new agents: the enrollment secret, which a new machine must send to register, and the manager's CA fingerprint. Pass both to the installer, or copy the ready-made command from that menu:

sudo bash install.sh --mode 5 -- --manager-url https://<manager-host>:5443 --ca-fingerprint <fingerprint> --enrollment-secret <secret>

With the fingerprint the agent checks the manager's certificate before it sends anything and talks to the manager over TLS from the start. Without it the agent registers over plain HTTP and receives the certificate after approval. An agent that arrives without the secret while one is set is turned away; the agent log says the secret is missing and shows the install command to run. The same values can be given as environment variables (LLMSYS_ENROLLMENT_SECRET, LLMSYS_CA_FINGERPRINT); the Docker Compose file takes LSM_ENROLLMENT_SECRET, and the deb/rpm agent package asks for the secret during install (or takes LLMSYS_CFG_ENROLLMENT_SECRET).

Or you can optionally download and run the installer from github

bash <(curl -fsSL https://get.llmsysmanager.com/install.sh)

If an agent is waiting for a secure connection. The agent log shows a line that starts with control channel: and mentions holding the token, and the agent does not register or report. This happens when the agent already has the manager's certificate but its settings still point at the manager's plain HTTP address, and the manager's secure address does not answer or its certificate has changed. To fix it, either make sure the manager's https port (5443 by default) can be reached from the agent, or run the installer again with the fingerprint shown in Admin › Agents (Manage ▾ → Enrollment):

sudo bash install.sh --mode 5 -- --update --ca-fingerprint <fingerprint>

Step 2: Start the Agent (if not started during the installation)

On Linux:

sudo systemctl start llm-systems-agent

NOTE: On macOS, the installer registers a launchd service. Start it with:

launchctl start com.llm-systems-agent

Step 3: Approve the Agent

New machines can register only while enrollment is open. Enrollment is open for 15 minutes after each manager start; after that, open it from Admin › Agents with Open enrollment before starting the agent (each click adds 15 minutes, and Close enrollment sits in the Manage ▾ menu). The header chip shows whether enrollment is open and how long remains. An agent that registers while enrollment is closed keeps trying on its own and appears as soon as enrollment opens.

New agents must then be approved before the manager will accept their data. The agent should appear in the dashboard within about 30 seconds of starting. Approved agents do not keep the secret: the agent deletes its copy once it is approved, so a new secret made on Admin › Agents only matters for machines installed later. An agent whose row you delete from Admin › Agents must be installed again with the current secret before it can enrol.

1. Open the dashboard in your browser
2. Go to the Admin tab
3. Click Agents
4. Find the new agent in the list and click Approve

Once approved, the agent begins sending metrics and the manager can communicate with it. Fleet-wide actions — Approve all pending, Update all, Push CA, and the Agent security switch — live under the Manage ▾ menu on the same tab. Turning Agent security off is for troubleshooting from a browser or script on your network: agents then answer their read-only calls (status, metrics, config and log reads) without a token, while everything else — restart, update, config changes, pausing collection, terminals and server control — always needs the agent's token. The manager's own checks never change.

Approve a second agent that runs the same provider (e.g. a second llama.cpp box) and a host picker appears on the matching dashboard sub-tabs; every approved agent is independently viewable and controllable. One agent is the default (what the dashboard shows when you haven't picked); set it from Admin.

Agent with Homebrew (macOS / Linux)

On macOS (Apple Silicon) or a Linux host with Homebrew (x86_64 or arm64), install the agent from the project's Homebrew tap:

brew tap llmsyscore/tap
brew trust llmsyscore/tap        # newer Homebrew requires trusting third-party taps
brew install llm-systems-agent

The formula picks the right prebuilt binary for the platform. Set MANAGER_URL in $(brew --prefix)/etc/llm-systems-agent/agent_config.yaml (the fully documented agent_config.yaml.example is installed alongside it for reference), then run the agent as a service (launchd on macOS, a systemd user unit on Linux):

brew services start llm-systems-agent

brew upgrade llm-systems-agent picks up new releases automatically: a scheduled job in the tap tracks each GitHub Release and bumps the formula. The dashboard's Admin → Agents → Update self-update also works, but a later brew upgrade replaces the binary again, so prefer brew on Homebrew-managed hosts. Uninstall with brew services stop llm-systems-agent && brew uninstall llm-systems-agent.

Agent binary (no Python required)

Every release also ships the agent as a per-platform tarball (llm-systems-agent-linux-x86_64.tar.gz, -linux-arm64.tar.gz, -macos-arm64.tar.gz) with a .sha256 checksum; no Python or venv is needed on the host. Each tarball bundles the self-contained binary, a fully documented agent_config.yaml.example, and the platform's service-manager unit (llm-systems-agent-binary.service.tmpl on Linux, com.llm-systems-agent-binary.plist.tmpl on macOS). On Linux:

sudo mkdir -p /opt/llm-systems-agent && cd /opt/llm-systems-agent
sudo curl -fsSLO https://github.com/llmsyscore/llm-systems-manager/releases/latest/download/llm-systems-agent-linux-x86_64.tar.gz
sudo curl -fsSLO https://github.com/llmsyscore/llm-systems-manager/releases/latest/download/llm-systems-agent-linux-x86_64.tar.gz.sha256
sha256sum -c llm-systems-agent-linux-x86_64.tar.gz.sha256   # macOS: shasum -a 256 -c <file>.sha256
sudo tar -xzf llm-systems-agent-linux-x86_64.tar.gz         # -> binary + agent_config.yaml.example + .service.tmpl
sudo chmod +x llm-systems-agent
sudo cp agent_config.yaml.example agent_config.yaml         # then edit: at minimum set MANAGER_URL
sudo chown -R <run-as-user>: /opt/llm-systems-agent

Then install the systemd unit from the extracted llm-systems-agent-binary.service.tmpl (substitute ${AGENT_USER}, ${AGENT_GROUP}, ${AGENT_INSTALL_DIR}) into /etc/systemd/system/llm-systems-agent.service and systemctl enable --now llm-systems-agent.

Provider flags (LLAMA_ENABLED, LMS_ENABLED, sudo wrappers for service control, udev rules for liquidctl) are what the full installer automates; every option is documented inline in agent_config.yaml.example, so set them in your copied agent_config.yaml as needed.

On macOS, download the -macos-arm64.tar.gz tarball instead; it bundles the same binary + agent_config.yaml.example plus the com.llm-systems-agent-binary.plist.tmpl launchd unit. Clear the quarantine attribute first (xattr -d com.apple.quarantine llm-systems-agent), then use the extracted plist template (substitute ${AGENT_USER}, ${AGENT_USER_HOME}, ${AGENT_INSTALL_DIR}) as the launchd unit. Linux binaries need glibc 2.35+ (Ubuntu 22.04 / Debian 12 or newer).

Updating agents from the dashboard

Per-agent Update on Admin → Agents: the agent downloads the latest release tarball for its platform, verifies the .sha256, extracts and smoke-tests the staged binary, swaps it atomically (previous binary kept beside it as .self-update.bak.<ts>), and restarts. Update all in the Manage ▾ menu upgrades every agent, one at a time: each has to report the new version before the next starts, and the sequence stops at the first failure with the remainder left untouched.

Multiple Hosts

Typical lab topology:

                ┌─────────────────────┐
                │  Manager + Alarm    │
                │  Engine + InfluxDB  │
                │  + local agent      │
                └──────────┬──────────┘
                           │
        ┌──────────────────┼──────────────────┐
        │                  │                  │
   ┌────▼─────┐      ┌─────▼────┐       ┌─────▼────┐
   │  GPU     │      │  Mac     │       │  Other   │
   │  host    │      │  Studio  │       │  hosts…  │
   │  agent   │      │  agent   │       │  agent   │
   │  +llama  │      │  + LMS   │       │          │
   └──────────┘      └──────────┘       └──────────┘

When you want InfluxDB on its own host, run mode 6 (InfluxDB only) there first, then mode 2 (Manager + alarm) on the manager/alarm-engine host. The installer prompts for the InfluxDB URL and the InfluxDB tokens that were printed during the InfluxDB installation.

When you want the manager and alarm engine on separate hosts, run mode 3 (manager only) on the manager host and mode 4 (alarm engine) on the alarm engine host. The installer prompts for the cross-host URLs and then prints the exact commands to copy the alarm engine's TLS certs from the manager host to the alarm engine host.

The alarm engine's management_token is required on a split install. It locks the engine's rules/alerts/notifications API and lets Admin → Settings edit alarm-engine settings. Mode 4 prints a generated value; the mode 3 prompt accepts that paste or new to generate one on the manager host, and cannot be skipped. Non-interactive mode 3 installs can pass it as LLMSYS_CFG_AE_MGMT_TOKEN; otherwise one is generated and printed at the end. The same value must then be live on both hosts. Until it is, the engine logs ALARM ENGINE AUTH at startup and Admin → System Health shows an auth open chip on the alarm-engine row.

Choosing the run-as user

By default the manager and alarm engine run as a dedicated llmsys system account (auto-created, password-locked). Pass --user <name> to the installer to use a different account, or enter the account name when the installer asks. If the account exists, its real primary group is preserved; if it doesn't, the installer creates it as a system user. The agent installer accepts the same --user flag.

Configuration

All runtime settings for the manager and alarm engine live in a single configuration file. The installer creates this file for you. Most settings have defaults and do not need to be changed.

Configuration File Location

The configuration file is located at:

config/llm-systems.toml

It is readable only by the service user (file permission 0600).

A reference file listing every setting and its default is located at:

config/llm-systems.toml.reference

The installer writes a short file with only the settings it needs; any setting not in your file uses its built-in default.

Refer to the reference when you need to understand what a setting does or when adding a new key.

Key Settings

Applying Changes

After editing config/llm-systems.toml, restart the affected service for the changes to take effect.

For changes that affect the manager:

sudo systemctl restart llm-systems-manager

For changes that affect the alarm engine:

sudo systemctl restart llm-systems-alarm-engine

If you changed a setting used by both (such as InfluxDB credentials), restart both.

Pointing the agent at your own services

The agent attempts to configure itself. If your inference servers run on different ports, hosts, or paths, override them in agent/agent_config.yaml on each agent host (the installer drops a template alongside the agent). Common keys:

The installer fills most of these in at deploy time via auto-detect and prompts. Any field can also be set via environment variable LSA_<NAME> (e.g. LSA_LLAMA_API_URL=http://...). The agent installer offers --enable-llama, --enable-lms, --enable-vllm, and --enable-perf flags, and auto-detects most of these from what's installed on the host. A host with neither llama-server nor LM Studio just reports system metrics. Per-agent settings can also be edited from the Agents tab.

Trim your config file

An older install may have a config file that lists every setting, most of them at their default values. The slim command prints a shorter copy that keeps only the settings you changed. Secrets and the settings the installer wrote are kept.

sudo -u llmsys bash -c 'umask 077; /opt/llm-systems-manager/llm-systems-manager/venv/bin/python3 \
  /opt/llm-systems-manager/tools/installer/toml_reconcile.py slim \
  /opt/llm-systems-manager/config/llm-systems.toml \
  /opt/llm-systems-manager/config/llm-systems.toml.reference \
  > /opt/llm-systems-manager/config/llm-systems.toml.slim'

The removed settings are listed on screen. Review the new file, keep a copy of the original, then copy the new file over config/llm-systems.toml and restart both services.

InfluxDB Memory Settings

When the installer sets up InfluxDB (modes 1 and 6), it also protects it on hosts where memory is tight. Existing installs get the same settings on their next update; see Refreshing an existing install below.

• Out-of-memory protection. A systemd drop-in, /etc/systemd/system/influxdb.service.d/llm-systems-manager.conf, sets OOMScoreAdjust=-500. If the host runs out of memory, the kernel then stops other processes before InfluxDB. Without it, InfluxDB is usually the largest process, so it is the one that gets stopped, and because it needs 2–3 GB while it starts back up, it can get stuck being stopped and restarted over and over. Check it with systemctl show -p OOMScoreAdjust influxdb (expect -500).
• Memory limit (GOMEMLIMIT). When InfluxDB shares the host with the manager, alarm engine or an agent, the installer writes GOMEMLIMIT into /etc/default/influxdb2, the file the service reads its environment from. This makes InfluxDB free unused memory sooner instead of holding gigabytes it no longer needs. The value is 35% of the host's RAM, at least 1024 MiB, and at least 1600 MiB on hosts with 4 GiB or more (6 GiB → 2150 MiB, 16 GiB → 5734 MiB). It is a target, not a hard cap: InfluxDB can go above it when it really needs the memory. The installer summary shows the value. An InfluxDB-only host (mode 6) gets no limit.
• Your own value wins. The installer only edits the lines between its # === llm-systems-manager memory (managed) === markers. To pick your own limit, add a GOMEMLIMIT=... line outside the markers; the installer then removes its block and leaves your line alone.
• Write delay. The managed tuning block in /etc/influxdb/config.toml sets storage-wal-fsync-delay to 0s, so each write is saved to disk right away and the Self-monitor write latency shows the real storage cost. Installs from before October 2026 carried 100ms, which added a tenth of a second to every write; install.sh --update changes that line to 0s and asks to restart InfluxDB. A value you set yourself other than 100ms is left alone.
• Write cache snapshots. The managed block sets storage-cache-snapshot-memory-size to 16 MiB, so the day's incoming points are written out to disk files every few hours. Installs from before October 2026 used 128 MiB; at a small fleet's write rate a whole day fits under that, so every read of recent data had to walk the whole day's points in memory and got slower hour by hour until midnight UTC. install.sh --update changes a 128 MiB line to 16 MiB, with or without the markers; any other value is yours and is left alone.
• Query concurrency. The managed block sets query-concurrency to the host's core count (at least 2) and query-queue-size to 16 times that. install.sh --update brings both lines up to date when the host's core count changed (for example after a VM was given more cores), also on older installs whose tuning lines carry no markers.
• Don't add MemoryMax. A hard systemd memory cap below what InfluxDB needs while starting up gets it killed on every start, which is the same restart loop the protection above prevents.
• Refreshing an existing install. A fresh install restarts InfluxDB itself. How the settings reach an install that already exists depends on how it was installed:
  • Script install, InfluxDB on the same host as the manager or alarm engine: install.sh --update refreshes them and adds InfluxDB to its restart prompt.
  • InfluxDB-only host (mode 6): run the same install.sh --update command on that host. There is no app code to update there, so it refreshes these settings and asks to restart InfluxDB.
  • .deb / .rpm package: every package upgrade refreshes them when InfluxDB runs on the same host, and restarts InfluxDB if anything changed.
  • By hand, any time: sudo bash tools/installer/tune-influxdb.sh applies them and asks before restarting InfluxDB (--restart restarts without asking, --dry-run only reports). After editing /etc/default/influxdb2 yourself, run sudo systemctl restart influxdb.
• Uninstalling InfluxDB through the uninstaller removes the drop-in and the managed block.
• Docker installs don't get these settings: the container runs InfluxDB on its defaults.
• Homebrew (macOS) installs don't get these settings: macOS has no Linux out-of-memory killer to adjust, and Homebrew generates the influxdb@2 service file itself.

Setting Up the Main Features

Everything below works once the manager and at least one approved agent are running. None of it is required — each feature is independent.

Inference Gateway

One OpenAI-compatible endpoint fronts every provider you run (llama.cpp, LM Studio, vLLM), so applications target the manager instead of an individual backend.

Base URL: http://<manager-host>:5000/api/gateway/v1

• GET /v1/models returns the merged catalog from every pool, each entry tagged with its provider and deduplicated by id.
• POST /v1/chat/completions and POST /v1/completions accept the usual OpenAI request bodies, streaming or not.
• Provider-scoped twins exist when you want to force one backend family: /api/gateway/llama/v1/*, /api/gateway/lms/v1/*, /api/gateway/vllm/v1/*.

The manager resolves which provider owns a model from the model id in the request, then picks a host by per-model pin, explicit ?agent=, pool round-robin, and finally the system default. If that host can't be reached before the first token, it fails over to another live host that serves the same model. The response carries an X-Proxied-To header naming the host that answered.

By default the gateway only accepts a logged-in dashboard session. To let external clients in, add bearer keys:

[manager.gateway]
enabled = true
api_keys = ["sk-your-secret-key"]
read_timeout_s = 600.0

Restart the manager afterwards. These keys are accepted only on /api/gateway/*. A key can optionally be labelled (name=secret); a labelled key names the client in the Gateway card's flow diagram, and an unlabelled one shows there as key-1, key-2, … by position:

api_keys = ["laptop=sk-abc123", "sk-plainkey"]   # empty = dashboard-session access only

Call it like any OpenAI endpoint:

from openai import OpenAI

client = OpenAI(
    base_url="http://<manager-host>:5000/api/gateway/v1",
    api_key="sk-your-secret-key",       # any configured key
)
resp = client.chat.completions.create(
    model="<model-id>",                 # from GET /v1/models; drives pin routing
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

or with curl:

curl http://<manager-host>:5000/api/gateway/v1/chat/completions \
  -H "Authorization: Bearer sk-your-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Hello!"}]}'

The gateway forwards over the existing agent TLS channel; admin and control endpoints are not part of it.

Model Autopilot

Admin → Gateway → Model Autopilot. You describe which models should be resident and where; Autopilot compares that against what is actually loaded and proposes the difference.

Each entry names a model, its provider (llama, vllm, or lms), and a placement — either a specific host id or auto. Optional per-entry settings:

A separate opt-in toggle, Protect other models (protect_unmanaged, off by default), sits beside Autopilot on the Gateway card: when on, a host whose single resident slot holds a model Autopilot doesn't manage is blocked from displacement, and freed-memory credit from displacing a model is never counted toward it. Route sync also skips busy hosts.

Autopilot is disabled by default. While disabled it still evaluates continuously and shows what it would do, so you can watch it for a while before handing it control. Enable it with the toggle on the same card once the proposals look right.

Placement is gated on memory: a host is only offered a model if it has the VRAM to hold it, or the RAM if the host has no GPU. That is why a size override matters for models the manager can't measure — without a size, an entry can be skipped rather than placed badly.

GPU Report Card

LLM Control → Tools → Report Card. Pick a provider and host, choose the standard preset, and run. The card reports time-to-first-token, prefill and generation throughput, tokens/joule, and $/Mtok, along with the GPU and VRAM it ran on.

Report Card shares the Tools launcher with two sibling modules, Benchmark and Autotune, run in-tab from the same app-style workspace. All three record to a cross-tool run ledger, so a run started in one browser is visible in every other session and survives a closed tab.

The same preset runs against every provider, so cards are comparable between machines and between backends. Results are stored, so the Trends view plots them over time. Set the electricity price used for the cost figure with:

[manager.reportcard]
price_kwh = 0.15

It can also be overridden per run from the sub-tab. Leaderboard submission is present but disabled.

Energy & Cost Intelligence

Dashboards → Energy. Shows measured power draw converted to a $/Mtok figure, a monthly-savings comparison against hosted-API list pricing, and idle-power accounting so hardware that is powered but unused is attributed rather than ignored. The window selector covers trailing spans, Today (local midnight to now), Year to date, and a Custom range of up to 366 days; the hourly chart itself is capped at 45 days and says so in its label when a wider window truncates it.

[manager.energy]
# price_kwh = 0.15                # unset inherits [manager.reportcard].price_kwh
cloud_price_in_per_mtok = 0.15
cloud_price_out_per_mtok = 0.60
cloud_price_label = "budget cloud API tier"

Cost is only computed across hosts that report both power and token telemetry. A host that reports one but not the other is left out rather than being allowed to skew the total. LM Studio reports no token counts of its own, so the gateway counts them as requests pass through — LM Studio traffic that bypasses the gateway is not counted.

Discord Bot

Slash commands for host queries, model load/unload, and alarm acknowledgement.

[manager.discord]
enabled = true
bot_token = "<your-bot-token>"
guild_id = "<your-server-id>"
allowed_user_ids = ["111111111111111111", "222222222222222222"]
allow_model_control = false

Three things to get right:

• The allowlist is fail-closed. An empty allowed_user_ids refuses everyone. Each ID must be its own quoted string.
• Model control is separate. Queries work with allow_model_control = false; loading and unloading models requires setting it to true.
• The bot invite needs both the bot and applications.commands scopes, otherwise the slash commands never register.

Restart the manager after editing this section — the bot reads its allowlist at startup.

Tower assistant

An in-dashboard drawer that answers questions about hosts, models, alerts, energy, and saved model profiles over the inference gateway. Enable it in Admin › Settings › Tower assistant: turn Enabled on and, optionally, pick a Primary model from the loaded ones — the default auto lets Tower use any loaded chat model; Tower always uses a model that is already loaded.

Choosing a model. Tower needs a chat model that can call tools. Any resident chat model works with Primary model set to auto; a pinned model must already be loaded. Recommendations:

• Size: 7B parameters or more. Under 7B (the drawer marks these small model) expect tool calls written as text now and then; Tower corrects one per question and falls back to a canned line after the second, so answers get slower and shorter.
• Family: an instruction-tuned model with tool-call training — Qwen3 8B/14B/27B, Gemma 3 12B/27B, Llama 3.1 8B+ or their fine-tunes. Base, embedding, reranker, speech and image models are skipped automatically.
• Thinking models (Qwen3 in think mode, reasoning fine-tunes) spend part of every reply thinking. The Thinking setting under Temperature decides how much: Off asks the model to skip it, Low, Medium (the default) and High allow 1k, 2k or 6k reasoning tokens on top of Answer length, and the budget is a hard stop everywhere: llama.cpp stops the model itself, and on LM Studio and vLLM Tower stops the thinking at the budget and asks once more with thinking off, handing the model its notes so far. A model that loops in thought is cut off and made to answer. Expect the first token to take longer with thinking on; Tower retries a reply that finished thinking without writing an answer once, then reports the last step it completed.
• Server: built-in function calling needs llama.cpp with jinja templates on (the default in current builds), LM Studio, or vLLM; otherwise Tower uses its text format, which works everywhere but depends more on the model. Leave Tool calls on Auto: the Tool response check row in Settings › Tower assistant probes the primary and fallback models once and shows Tools OK (filled = built-in, outlined = text format), No tool support or small model; Verify re-runs it after a server or model change.
• Fallback: with a second chat model loaded, switch Fallback model on so a timeout, a failed check or a model that keeps writing tool calls as text hands the question to the other model.
• Measure it: the Model evaluation row in Settings › Tower assistant runs a handful of canned questions through Tower against the primary model (plain reads, a host named by alias, a two-step read, a timer, an action proposal, and a question card when two hostnames share a prefix) and scores each: the right tool with the right arguments, the right kind of reply, a real answer. More calls per question, corrections and retries mean a weaker model. Nothing changes on any host (every action card is denied, every question card takes its first choice); results are kept per model with its quant and server, listed in the Tools tab ledger, and Export downloads the full JSON.
• Download a Tower model: the row below it lists recommended models by the VRAM they need (Qwen3.5 4B / 9B, Nemotron 3 Nano 4B, Gemma 4 E4B / 12B / 26B-A4B, Qwen3 8B), capped at 14B dense plus one mixture-of-experts: bigger models cost first-token time and VRAM on every question without a better score. Q4_K_M is the floor (lower quants start dropping tool calls); the 12 GB and 16 GB tiers offer the 9B and 12B at Q6 instead. Every entry has a thinking mode the Thinking setting can switch, and its grade is the median of three Tower checks (8 of 8 = high, 6 or 7 = good; anything lower is not listed), shown as expected until the model has a check of its own. Download fetches one to any llama.cpp or LM Studio host (primaries listed first), loads it, runs the tool check and the eval, and then offers Pin as primary; the fallback stays as configured. On a llama.cpp host the model gets a config profile named tower (32k context, 99 GPU layers, flash attention, jinja, ubatch 1024, temperature 0.2, reasoning on with a 2048-token budget) that LLM Control shows as its active profile, and on either host the alias Tower Model unless it already has one. A model kept for Tower takes VRAM away from what the host serves, and on a llama.cpp host the server restarts after the download, so what it was serving is unloaded until it is loaded again; the Download button says so before it starts. Stop on a llama.cpp download also removes the half-finished files from the host's cache; on LM Studio it ends the job here only, and the download has to be cancelled in LM Studio as well (the row says so, and a download cancelled in LM Studio ends the job here). LM Studio loads the model with a 32k context and a 2048 batch; temperature is per request, and Tower sends its own. The Primary model list shows each model with its server and hosts, and marks a pinned model the gateway no longer lists as not available.

It starts scoped down: "Capabilities" defaults to read (answer only), and off-topic questions are refused. Both can be relaxed later in the same settings group, as can the tool list (Available tools).

Set "Capabilities" to Answer and act to let it load or unload a model, wake llama-server, or acknowledge / close an alert; incl. admin actions adds restarting a provider server (admins only). Every action pauses on a card in the drawer — what it will do, where, what it will not do, and who may approve — and runs only after Approve. Approvals expire after 10 minutes, the caller's role and the tier are re-checked at that moment, and each decision lands in Admin › Audit as tower via <user>. Any single tool can still be switched off under Available tools.

The phone companion carries the same assistant as a Tower tab: ask and read the answer, approve or deny action cards, answer question cards, switch between conversations, and read the Insights list with Dismiss, an Alert link into the Alerts screen, and Apply where the role and tier allow it. Apply and Approve confirm in the bottom sheet first. The tab is hidden while Tower is off.

Tower does not write code: a language-tagged code fence in a reply is withheld and replaced with a single "Code withheld" line, though configuration Tower reads for diagnosis — TOML, JSON, YAML, plain text and the like — still shows. "Report rule-bypass attempts" (Settings › Tower assistant, on by default) covers messages that try to make Tower ignore its rules, reveal its instructions, skip approvals or take on another persona: these are refused outright, and with reporting on the refusal is logged to Admin › Audit as a critical tower.violation and raises a critical alert; with reporting off, Tower still refuses but nothing is logged or alerted. Questions that probe the manager's security internals (vulnerabilities, exploits, security holes, attack surface, breaking into it, getting around the login) get the same treatment whatever the off-topic setting: Tower answers with a fixed block that points at the developer's support contacts and what to include in a report, the model is never asked, and with reporting on the message is logged as a warning tower.violation and raises a warning alert named "Tower security probe".

Request timeout (s) bounds how long Tower waits for the model to start answering. With Fallback model on, a question that misses it is asked of the next loaded chat model instead (another host first), and the reply says so; the next question goes back to the primary model.

Debug logging (Settings › Tower assistant, and the same switch under Inference Gateway) writes a per-step trace to the manager log with names, sizes and timings only.

"Diagnose new alarms" (Settings › Tower assistant, off by default) makes Tower look at every alert that opens at or above "Lowest severity". Each look is a read-only pass over the same tools, at most five reads, stopped after "Diagnosis time limit" (180 s by default; raise it for a slow model), and its result lands on the drawer's Insights tab as a compact card: a one-line summary, Details (what it checked), and a button for the playbook when one of Tower's playbooks fits. While a conversation is open, a one-line notice above it links to a new insight. The playbooks wake llama-server, reload an LM Studio model pinned to that host, acknowledge an alert once it has recovered (a threshold alert whose value is back past its threshold, or a non-engine alert that reports recovery), and restart llama-server (admin-only). Before anything runs, Tower re-reads the alert and refuses when it has closed or changed. The header badge counts insights not yet seen, including ones a safe playbook already applied, a toast announces one while the drawer is closed, opening Tower with unseen insights lands on the Insights tab, and insights show even when no chat model is loaded. "Apply safe playbooks" lets the three safe ones run without asking when "Capabilities" is at least Answer and act; each run is written to Admin › Audit as tower via alarm <id>. Alerts already open when the switch turns on are not diagnosed.

Timers let Tower poll something and report later: ask for "the RAM on box every minute for 10 minutes" or "check the alarms again in 5 minutes" and Tower schedules a timer instead of waiting. Each tick reads one live host metric (CPU, RAM, GPU, GPU temperature or watts) or any read tool, at most once every 30 s, for up to 60 minutes or 120 ticks, with three timers per user and eight per manager; the strip under the drawer's tabs shows each live timer with its next tick, the time left and a Cancel. When a timer finishes, its samples come back as a new turn in the same conversation with a small graph, and Tower reports the values, the range and the trend. Ticks count against the same per-minute limit as questions, a manager restart ends every live timer, and every schedule, cancel and report is written to Admin › Audit. The schedule tool can be switched off under Available tools.

llama.cpp serves native tool calls only when its server args include --jinja; otherwise Tower falls back to a fenced JSON tool format automatically. Nothing Tower reads or does leaves the lab.

Forecast

A scheduled check that looks for trends in up to 30 days of stored history — alarm history, per-host hardware and load, energy and cost, benchmark and report-card results, model and gateway activity, the audit log — and reports what is heading toward trouble before it becomes an alarm. It runs on its own schedule and works with or without Tower.

Turn it on in Admin › Settings › Forecast: switch Run forecasts on, pick how often it runs and at what local time (hourly through weekly, or Custom… for any number of hours or days and a time to the minute) and how far back it looks, and optionally let findings also raise alerts through your normal alert channels once they reach a chosen severity. Any check can be switched off individually from the same page. Run now starts an out-of-schedule pass at any time.

Each finding — the trend, the projected date, the confidence, and a suggested next step — comes from the measured numbers. When Tower is enabled, it can additionally add a short plain-language gloss on top of the code's own findings: a Tower effort setting controls how much of that Tower does, and Auto (the default) picks a level for you based on how well the current Tower model scores, its size and its speed, so a small or slow model does less and a strong one does more:

• Off — no Tower involvement; every finding shows only the code's own summary and next step.
• Light — one short digest tying the run's findings together at the top of the page.
• Standard — the digest, plus a one-line "Likely cause" under each finding.
• Full — the digest and likely causes, plus Tower looking into each flagged check with its own read tools for a closer explanation.

Tower adds summaries and explanations: a finding's chart, facts and suggested step stay as measured. A weekly digest summarising the run also appears on its own day and time, independent of how often forecasts run.

The Forecast page lists open findings with search, severity/host/check filters, sortable columns and pages; a 30-day line shows every dated prediction. Cleared and dismissed findings have their own tabs. Each finding offers Ask Tower (opens Tower with the finding as the question) and, where there is somewhere useful to go, a button that opens that part of the dashboard. The Overall page shows only critical and warning findings.

Dismissing a finding is not permanent: dismissed and cleared findings are swept after 90 days, so a trend that is still running when its row is swept comes back as a new finding. Tower conversations are kept by default (tower_history) — at the Full effort level each check's investigation runs in its own private conversation, and an admin can open it from the finding ("Open conversation"). It is read-only, and is swept with the rest of Tower's history. Turn the setting off to delete each conversation as soon as its check finishes.

Backups

Admin → Backups. A scheduled run writes an encrypted manager archive (config, agent registry, CA, users, model profiles, and the three SQLite files manager.db, audit.db, energy.db; archives made before the split, carrying a single metrics.db, still restore) and, when [alarm_engine].management_token is set, the alarm engine's own export in the same run — without a management token the run is recorded as manager only. Retention counts runs, not archives, so keep_last = 7 can retain up to 14 files.

[manager.backup]
enabled = true
interval_hours = 24.0
keep_last = 7
mirror_dir = "/mnt/backup-share"   # optional second copy of every archive

mirror_dir is validated on save — it must be an absolute path that exists or can be created, and writable by the service user — and applies without a restart. Any retained archive can be downloaded from the Backups card (backup.download is recorded in the audit log); a run where the alarm engine is configured but unreachable is marked partial rather than silently dropping that half.

OpenClaw Proxy Origin

If you run the OpenClaw tool through Tools → OpenClaw, the OpenClaw host's own gateway checks the browser's Origin header and rejects pages it doesn't recognize. The Tools tab pages are served on the tool ports, so add those origins to the allowlist on the OpenClaw host:

gateway.controlUi.allowedOrigins = ["https://<manager-host>:5448", "http://<manager-host>:5447"]

Without this, the embedded page fails with "Browser origin not allowed" even though the proxy itself is reachable.

OpenClaw Telemetry (OTLP)

OpenClaw's diagnostics-otel plugin can push metrics, traces, and logs to the alarm engine's OTLP receiver (/v1/metrics, /v1/traces, /v1/logs on the ingest port, bearer = [alarm_engine].ingest_token). The receiver is served over TLS with a certificate signed by the manager's internal CA, and the OpenClaw gateway (a Node process) verifies it. Node does not read that CA from the OS trust store even with NODE_USE_SYSTEM_CA=1, so give it the CA file directly through NODE_EXTRA_CA_CERTS. The agent on the OpenClaw host already keeps a current copy at <agent install dir>/data/tls-ca.pem, and Node adds it on top of its built-in public roots.

The gateway service reads its environment from a file OpenClaw generates at ~/.openclaw/service-env/ai.openclaw.gateway.env (macOS) or ~/.openclaw/service-env/openclaw-gateway.env (Linux). Either regenerate it with the variable set:

NODE_EXTRA_CA_CERTS=<agent install dir>/data/tls-ca.pem openclaw gateway install --force

or change the existing NODE_EXTRA_CA_CERTS= line in that file to the CA path and run openclaw gateway restart. Both only restart the gateway process; OpenClaw's config, sessions and plugins are untouched (a later openclaw gateway install regenerates the file, so set the variable again then).

Then point the plugin at the engine in openclaw.json:

diagnostics.otel.endpoint = "https://<alarm-engine-host>:8081"
diagnostics.otel.headers.Authorization = "Bearer <ingest_token>"

Without the CA the exporter fails silently: the OpenClaw log shows unable to verify the first certificate and the engine's heartbeat otlp journal line stays at metrics+0 traces+0 logs+0; the Admin System Health card notes it once the engine has run 15 minutes with no OTLP data (the Alarm Engine node's detail shows OTLP received). The ingest token is not involved in that failure. Note this feed is separate from the Dashboards → OpenClaw tab, which the agent fills from OpenClaw's own session store.

Which attributes become tags. The receiver keeps telemetry attributes as InfluxDB tags only when they are bounded dimensions. Keys that name identifiers, network endpoints or free text (*_id, *Id, *hash, token, session, trace/span/frame ids, pid, port, addr, endpoint, peer, url, path, userAgent, message(s), content, value, command, prompt, description, …) are dropped; numeric attributes and keys ending in a unit (_ms, _bytes, _chars, _count, _tokens, _ratio, …) are stored as fields on the point instead of tags; the standard gen_ai.*, http.*, rpc.*, error.type and service.version dimensions and OpenClaw's bounded security.policy_id / security.control_id are always kept. Every other string attribute is a tag until it has shown [alarm_engine.otlp].tag_value_cap distinct values (default 100), after which new values are stored as other; at most max_tags attribute tags (default 32) are kept per point. Add keys to tag_allow / tag_deny under [alarm_engine.otlp] to override the pattern rules (dotted OpenTelemetry names or their underscored tag form both work). The heartbeat otlp journal line reports tags dropped+N capped+N fields+N, and the same counters are written as source=otlp-receiver metrics (otlp.tags_dropped, otlp.tags_capped, otlp.attr_fields, otlp.tag_keys_tracked) so a rule can watch them.

Existing installs. Series minted by the earlier receiver (one per event id, timing value or connection) stop growing as soon as the engine restarts on this version and age out with the raw bucket's retention; the rollup bucket keeps its copies for its own, longer retention. No manual cleanup is needed. To reclaim the index sooner, delete the OpenClaw source's history from both buckets with the InfluxDB CLI — note this removes all history for that source, and predicate deletes leave tombstones that slow queries until the next compaction:

influx delete --org <org> --bucket alarm_engine_metrics        --start 1970-01-01T00:00:00Z --stop $(date -u +%Y-%m-%dT%H:%M:%SZ) --predicate 'source="openclaw-gateway"'
influx delete --org <org> --bucket alarm_engine_metrics_rollup --start 1970-01-01T00:00:00Z --stop $(date -u +%Y-%m-%dT%H:%M:%SZ) --predicate 'source="openclaw-gateway"'

Internal Certificate Roles

The manager signs a certificate for itself, for the alarm engine and for each agent. Each certificate names what it belongs to: the manager, the alarm engine, or one agent. Each side checks that name when it connects. An agent checks the manager and the alarm engine; the manager checks each agent and the alarm engine.

After updating to a version with certificate roles:

1. Restart the manager. It issues new certificates for itself and for the alarm engine.
2. Restart the alarm engine. On a split install, first copy ae-tls.crt and ae-tls.key from the manager's data/ directory to the alarm engine's data/ directory.
3. Update the agents with Update all under Admin → Agents → Manage ▾. Each agent receives its new certificate on its next heartbeat and starts using it when it restarts.

Admin → System Health shows the progress. The Agents node lists certificate role checked with a count, and the Alarm engine node lists certificate role.

Admin → Agents shows each agent. Its TLS chip reads tls ✓ once its role is checked, and the Connection details list certificate role with one of:

• checked and the date.
• restart the agent to finish — the agent has its new certificate and uses it after a restart.
• new certificate not received yet — the manager sends it again on a later heartbeat.
• not yet — nothing to report so far, or the agent runs a version from before certificate roles.

The first time one side sees a certificate with a role, it requires the role on every later connection to that peer.

An agent that has checked the manager keeps using the manager's HTTPS address, including after a restart, even when its configuration names an http:// address. The same applies to the alarm engine's address. The manager does not connect to a checked agent over plain HTTP and gives the browser no direct link to it. System Health lists a warning for a checked agent that reports a plain-HTTP address.

• If the manager moves to a new address, set the new https:// address in the agent's configuration and restart the agent. A new alarm-engine HTTPS address reaches the agent from the manager.
• To clear what an agent remembers about the manager and the alarm engine, including their HTTPS addresses, delete data/tls-roles.json on that agent's host and restart the agent. Do this on each agent after turning HTTPS off for the manager or the alarm engine.
• To clear what the manager remembers about the alarm engine, delete data/tls-roles.json on the manager host and restart the manager.
• To clear what the manager remembers about one agent, delete that agent under Admin → Agents. It registers again as a new agent and needs approval.
• System Health lists a warning for an agent that has not stored its new certificate after 15 minutes, and for one that serves it but is still not checked. Push CA under Manage ▾ sends every agent its certificate again.

Operator-Provided TLS Certificate

By default the HTTPS port ([manager].tls_port, 5443) serves a certificate from the manager's internal CA, which browsers on other devices do not trust. To serve a certificate they do trust — a Let's Encrypt cert for your domain, or one from a corporate CA:

[manager]
tls_cert_file = "/etc/ssl/private/lab.example.com.fullchain.pem"
tls_key_file  = "/etc/ssl/private/lab.example.com.key.pem"

How it behaves:

• Selection is by SNI. The operator cert is served only for hostnames its DNS SANs cover (a wildcard matches one label). Requests by IP or by names outside the SANs — including every agent, which pins the internal CA — still get the internal-CA cert, so configuring this never breaks the agent control channel.
• Both keys must be set and readable by the service user. A half-set pair or unreadable files log a warning and fall back to the internal CA. Relative paths resolve against the install root, and a group/world-readable key file is warned about.
• Renewals are your job. The internal-CA cert auto-rotates; an operator cert does not — the Admin system-health card tracks the expiry of whichever cert is actually served and says so. Point the config at the live paths your ACME client maintains and restart the manager after renewal.
• Set [manager].ws_proxy_tls_port (default 5446) so the Events/alerts WebSocket has a wss endpoint — an HTTPS page cannot open a plain ws:// connection.
• Baseline security headers are always on. Every manager response carries X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN, Referrer-Policy: strict-origin-when-cross-origin and Content-Security-Policy: frame-ancestors 'self'. The dashboard embeds the alarm-engine UI and the tool proxies in same-origin iframes, which these allow; embedding the manager from another origin is refused. A route that sets its own header (the proxies' [manager.security].proxy_html_csp) keeps it. That policy lets the embedded tools talk only to the manager (plus inline data: resources and the manager's own WebSocket bridge, written as ws://{host}:* where {host} becomes the hostname in the browser's address bar), lets them embed frames only from the manager, and runs them sandboxed so a page cannot navigate the dashboard tab away. The image-generation port an agent advertises is used only when it is a whole number from 1 to 65535; otherwise the default port is used.
• HSTS is opt-in. [manager].hsts_max_age_s (default 0) adds a Strict-Transport-Security header to HTTPS responses. Leave it at 0 unless the plain-HTTP port is closed — HSTS preserves the port, so one HTTPS visit makes the browser rewrite http://host:5000 to https://host:5000 and fail.

This is also the prerequisite for installing the phone companion below.

Protecting the Manager from Request Floods

Most of the manager needs a login or an agent token, so a stranger can only reach a few routes: agent registration, the status check a waiting agent repeats until it is approved, the certificate fetch an approved agent makes once, and the login page. The manager protects those routes in two layers:

• Per address. One address may send 6 registrations a minute and 5 failed logins in 15 minutes; past that it is refused for a while. Each refusal is logged with the address: registration REFUSED (rate limit) from <address> (once a minute per address) and manager login LOCKED (user=<name>) from <address>.
• For everyone together. [manager.security].anon_request_budget_per_min (default 600) caps the total number of requests per minute (a fixed 60-second window, not aligned to the clock's minutes) on those routes, whatever addresses they come from. Past the budget the manager answers 429 Too many requests with a Retry-After: 60 header before it reads the request or builds a page. Requests that carry a login session or an approved agent's token are not counted, so during a flood your dashboard keeps working and approved agents keep fetching their certificates; a waiting agent's status check and a fresh login share the budget with the flood, so they may have to wait a minute. The log gets one line when the budget runs out and one line per minute while the flood lasts (refused N anonymous requests in the last minute). The default leaves room for the largest normal load (100 waiting agents each checking every 15 seconds) and only bites at a real flood; raise it for a very large fleet, or set 0 to turn it off when a proxy in front already limits requests.
• Watching it. When requests were refused in the last minute, Admin › System Health lists it under Warnings, colours the Manager box, and opens the Manager details with the open routes row (requests on those routes and how many were refused, for the last full 60-second window); close the details and they stay closed until the flood is over. The Manager dashboard's Stream & Connection Health card shows the same two numbers and turns its badge to "flooded". The agent on the manager host stores the same two numbers once a minute as the manager_anon_requests_per_min and manager_anon_refused_per_min series (source manager_self_monitor), so the Manager Perf card on the dashboard shows them, the alarm engine's history keeps them, and an alarm rule on manager_anon_refused_per_min above 0 tells you a flood happened while you were away.

On a network you do not fully trust, put a reverse proxy in front of the manager and let it absorb floods before they reach the service. Note that the manager then sees every client as the proxy's own address: the per-address limits above apply to the proxy as a whole, and the log lines name the proxy, so address-based banning has to happen at the proxy. With nginx, add a limit for the open routes (agents behind one NAT share one address, so raise the rate for a large site):

limit_req_zone $binary_remote_addr zone=lsm_anon:10m rate=30r/m;

server {
    listen 443 ssl;
    server_name manager.example.net;
    # ssl_certificate / ssl_certificate_key: your real certificate

    location ~ ^/(login|api/agents/register|api/agents/[^/]+/(status|cert-bundle))$ {
        limit_req zone=lsm_anon burst=20 nodelay;
        proxy_pass http://127.0.0.1:5000;
    }
    location / {
        proxy_pass http://127.0.0.1:5000;
    }
}

The Tools tab loads the tool pages straight from the tool ports (5447, or 5448 for HTTPS), not through the dashboard address. With a reverse proxy, either open ports 5447 and 5448 on the manager host so browsers can reach them, or give the proxy a second server that forwards <proxy-host>:5448 to the manager's port 5448 and passes the Host header unchanged. When the manager uses its internal certificate, each browser must accept it for that port once: open https://<host>:5448/tools-ping in a tab and accept the certificate.

When clients connect to the manager directly, fail2ban can ban an address at the firewall after repeated refusals by watching the manager's own log lines (behind a proxy, use fail2ban's stock nginx-limit-req filter on the proxy's error log instead). The manager logs to the journal, so the jail reads it with the systemd backend:

# /etc/fail2ban/filter.d/llm-systems-manager.conf
[Definition]
failregex = registration REFUSED \(rate limit\) from <HOST>:
            manager login LOCKED \(user=.*\) from <HOST>$
            manager login FAILED \(user=.*\) from <HOST>$
journalmatch = _SYSTEMD_UNIT=llm-systems-manager.service

# /etc/fail2ban/jail.d/llm-systems-manager.conf
[llm-systems-manager]
enabled  = true
backend  = systemd
filter   = llm-systems-manager
maxretry = 3
findtime = 600
bantime  = 3600

The registration line is written once a minute per address, so a machine that keeps flooding registrations is banned after three refused minutes; failed logins count per attempt. Check it with fail2ban-client status llm-systems-manager. Keep your own machines out of the jail with ignoreip so a mistyped password on the admin workstation does not lock you out.

Phone Companion (PWA)

The companion at /companion installs to a phone's home screen and receives alarm push notifications. It needs a trusted HTTPS origin (previous section) — service workers and web push do not work from an untrusted certificate.

[manager.companion]
push_contact = "mailto:you@example.com"   # VAPID contact browser push services see
release_check = false                     # opt-in GitHub release check (Settings toggle)
push_notify_token = ""                    # blank = fall back to the alarm-engine tokens

[manager.telemetry]
install_report = true    # anonymous install report
crash_reports = true     # anonymous crash reports
checkin = false         # daily check-in
heartbeat = false        # daily diagnostic report

Setup:

1. On the phone, open https://<your-domain>:5443/companion, sign in, and use the browser's Add to Home Screen / Install prompt.
2. Enable notifications from the companion's Settings screen, then use Send test notification to confirm delivery end-to-end.
3. Alerts are pushed by the alarm engine through the manager's /api/companion/push/notify bridge. A co-located install needs no token configuration; a split install presents [alarm_engine].management_token (or push_notify_token if you set one).

Home, Alerts, Tower, Energy, and Models screens work for every role (the Tower tab shows only while the Tower assistant is enabled); control actions (model swap/pin, autopilot approval, restarts) and the Admin screen require an admin session, and every action confirms in a sheet before it runs. pywebpush must be installed in the manager venv for push delivery — the installer includes it.

Updating

Updating All Components

To update the manager, alarm engine, and any locally installed agent to the latest version, run the installer in update mode:

cd /opt/llm-systems-manager
bash tools/installer/install.sh --update

The update process:

• Detects which components are currently installed
• Compares installed files against the latest version
• Backs up any files that will change
• Syncs only the changed files
• Reloads systemd units and restarts affected services
• Refreshes the InfluxDB memory settings on hosts that run InfluxDB (on an InfluxDB-only host this is the whole update)
• Runs the smoke test to confirm the update succeeded
• Ends with a Next steps list for this host: restarts you deferred, a temporary admin password when one was set, and, on a multi-host install, which hosts still need the update, whether the alarm engine certificate has to be copied to the alarm engine host, where to refresh the InfluxDB tuning, and the Update all step for the agents

Every update also re-stamps the install root's RELEASE marker, so an install that could not previously name its release — and therefore never reported an available update — self-heals on its next update.

You do not need to stop services first — the updater handles restarts.

When the update spans more than one host, update the manager host first, then the alarm engine host, then the agents (Admin → Agents → Manage ▾ → Update all). Certificates issued since v3.1.0 name what they belong to, and each side starts requiring that name once it has seen it, so this order keeps every connection working during the update. The updater also corrects the alarm engine's allowed-origins list (see [alarm_engine].cors_origins in the configuration table) and, when run from a terminal on a host that has never answered the question, asks once about the anonymous reports.

Releases up to v3.0.0 created the first admin account with a fixed password. If that account still has it, the updater tells you, asks to continue, and replaces it with a temporary password that it shows once. Sign in with it and set your own. If you answer no, or update without the installer (a git pull, a package upgrade, a new Docker image), that account cannot sign in until you run the admin_password.py reset command from Step 5.

Updating a Remote Agent

To update an agent running on a remote machine without logging into that machine:

1. Open the dashboard
2. Go to the Admin tab
3. Click Agents
4. Find the agent's row and open its overflow (⋯) menu
5. Click Update

The agent downloads and applies the latest version of itself, then restarts. To update every agent at once, use Update all under the Manage ▾ menu, which also carries a pending-update badge.

Monitoring Service Health

Checking Service Status

Check whether each service is running:

sudo systemctl status llm-systems-manager
sudo systemctl status llm-systems-alarm-engine
sudo systemctl status llm-systems-agent

A healthy service shows active (running). A failed service shows failed and usually includes the last few log lines explaining why.

Viewing Logs

Manager — log file updated continuously:

tail -f /var/log/llm-systems-manager/llm-systems-manager.log

Or via journald:

journalctl -u llm-systems-manager -f

Alarm Engine:

journalctl -u llm-systems-alarm-engine -f

Agent:

journalctl -u llm-systems-agent -f

Add --since "1 hour ago" to any journalctl command to limit output to recent entries.

Dashboard Health Page

The Admin tab in the dashboard includes a System Health card. It shows:

• Slim service rows for each component, with version and uptime
• A live data-flow diagram whose edges carry current rates, with a click on any node for the underlying numbers
• An overall Healthy / Attention / Down pill
• A restart-pending chip per service, and an alarm-engine auth chip (auth open, no token, or token mismatch)
• Release and agent-update status rows
• A restart control that always offers the alarm engine alongside the manager

The Admin tab button in the navigation bar turns red when any component reports a problem — you do not need to check manually.

Troubleshooting

Uninstalling

To remove LLM Systems Manager from a server:

bash tools/installer/install.sh --uninstall

The uninstaller:

• Stops and disables the systemd services
• Removes the installed files
• Prompts before deleting the runtime user account and InfluxDB data, so you can preserve your data if needed

Next Steps

After your deployment is up and running, refer to these documents for deeper reference:

• ARCHITECTURE.md — How the components fit together, data flow from agent to dashboard, and the multi-agent model
• COMPONENTS.md — Detailed description of each component: manager, alarm engine, and agent
• API_REFERENCE.md — Full reference for the manager and alarm engine HTTP APIs, including request/response formats