Fleet
Alerts & logs
Nodes are designed to hold no history: the panel is the only place logs are read from, pulling rolling tails on demand, while node events ride the heartbeat and land in the alerts feed and audit log.
Renders the node event feed (user.blocked, service.down, counter.reset, disk.pressure, user.quota.reset, etc.) that rides every heartbeat's events array and is stored server-side (src/lib/node-alerts-store.ts).
Two independent forwarding paths
These events are optionally forwarded to Telegram/Discord via the node's own
webhook.py dispatcher — the node never talks to Telegram through the panel; it fires its own outbound webhook independent of the panel's forwarding. Don't assume disabling one silences the other.| Event | Fired when |
|---|---|
| user.blocked / user.unblocked | Enforcement changes an account's servable state (expiry, quota, panel force-block). |
| user.quota.reset | A panel-side usage/daily reset watermark is honoured locally. |
| user.removed / user.ledger.pruned | An account disappears from a users.apply push; its ledger/counter baselines are wiped. |
| counter.reset | A traffic counter is detected going backwards (counter_delta.delta_since()). |
| disk.pressure | Free disk drops below 10% (logclean.disk_pressure()). |
| limits.shape_unenforceable-style alert | A speed cap cannot be enforced for a specific (usually CDN-fronted) client. |
Central log/audit viewer with filters; shows "no audit activity yet" / "no logs match filters" empty states. Surfaces both panel-side audit events and node-pushed log bundles.
- Node log bundles refresh every ~10 min (
LOGS_EVERY=40beats) viajournalctl -u <unit> -n <lines> --output short-isoforLOG_UNITS = (xray, zivpn, ssh, nginx, autoscript-node), capped to 8000 chars per unit, plus the certificate issuance log tail. - On-demand tails are also available via
POST /rpc/logs, and a full wipe viaPOST /rpc/logs.purge {deep}— both surfaced from this page or node detail actions. - The audit log itself is append-only and records every fleet action (documented and linked from the Node security & audit page).
Nodes are designed to hold no history — the panel is the only place logs are read, pulling rolling tails on demand via /rpc/logs.
- configure() sets journald to volatile (RAM) storage with RuntimeMaxUse=24M, SystemMaxUse=24M, MaxRetentionSec=30min, MaxFileSec=10min, no forward-to-syslog/wall. Also writes a logrotate config for Xray/nginx logs (size 8M, copytruncate, no compression).
- janitor() is called every heartbeat, a cheap no-op unless 10 minutes (every_sec=600) have elapsed, then calls purge(deep=False).
- purge(deep) truncates a fixed list of files (Xray error log, nginx access/error, syslog, auth.log, messages, wtmp/btmp/lastlog, cert.log), removes rotated .gz/numbered leftovers, vacuums the journal (--vacuum-time=5m --vacuum-size=8M), and truncates the Xray access log only once it exceeds ACCESS_CAP_BYTES=8MiB (or always when deep=True) — the access log can't simply be deleted because services/limits.py tails it for IP-limit enforcement. deep=True also wipes /var/log/journal/* outright.
- disk_pressure() is true when free space < 10% of the filesystem; the heartbeat loop calls a deep purge and emits a disk.pressure event when this trips.
- Also installed as a standalone systemd timer autoscript-node-logclean.timer (every 10 min, OnBootSec=2min) running autoscript-node-logclean.service, which vacuums the autoscript-node journal and deletes anything under /var/log/autoscript older than 10 minutes (except install.log/cert.log) — belt-and-braces alongside the in-process janitor.
- Manual trigger: RPC POST /rpc/logs.purge {deep: true|false} — surfaced by the panel's Logs page / node detail actions.
| Symptom | What to check |
|---|---|
| Node stuck on "Installing" | Missing /var/lib/autoscript-node/install-complete.json or a failed install phase. Run journalctl -u autoscript-node -f -o cat and re-run the (idempotent) install one-liner. |
| "fingerprint mismatch" on every RPC | The node's certificate changed but the panel's pin is stale. Use "Force re-pin" or wait for auto-recovery on the next heartbeat. |
| "node not paired — waiting for handshake" (HTTP 409) | The panel has no pin yet. Confirm the node can reach $MASTER/api/public/node-handshake, then check journalctl -u autoscript-node for handshake POST errors. |
| RPC calls time out / "unreachable (cooling down)" | Circuit breaker tripped after 2 consecutive failures. Confirm the agent port is reachable from the panel's egress IP (check the GRVPN_AGENT iptables chain); re-run the firewall phase or use the Node security page's "Save allowlist". |
| A user still connects after being blocked/expired | Check the admission gate at /run/autoscript-node/gate/{managed,allow}/<login> — if managed/<login> doesn't exist the account isn't recognised as panel-managed. If managed exists but allow still exists too, publish_gate() hasn't run since the block. |
| Usage numbers look doubled/halved/negative | Check statsSource in the heartbeat: "sockets" means Xray's stats API is down. Restore it (xray.repair_startup, systemctl restart xray) for authoritative per-user byte counts. |
| Disk filling up on a node | logclean.disk_pressure() should auto-purge below 10% free. If not, run RPC logs.purge {deep:true} or on-box journalctl --vacuum-time=5m; journalctl --vacuum-size=8M. |
| Agent update doesn't seem to apply | Check /var/log/autoscript/agent-update.log. The updater runs in its own transient unit (systemd-run --unit=autoscript-node-update); if systemd-run is unavailable it falls back to a detached Popen, which can be killed if the parent's cgroup is torn down. |
| nginx fails to start after cert issuance / port publishing | The installer retries without HTTP/2 automatically and falls back to a minimal ACME-only vhost on port 80. Check /var/log/autoscript/cert.log for the exact nginx -t error. |
Manual on-box commands
bash
systemctl status autoscript-node
journalctl -u autoscript-node -f -o cat
systemctl restart autoscript-node
systemctl status xray zivpn ssh nginx stunnel4 autoscript-node-ssh-ws
cat /etc/default/autoscript-node # 0600 - contains secrets, handle carefully
cat /var/lib/autoscript-node/install-complete.json
/usr/local/bin/autoscript-node-update # manually trigger the same path agent.update RPC uses