Fleet
Monitoring
Monitoring in Vortexa spans per-node analytics, live NIC/traffic accounting, on-demand benchmarking, WARP egress verification, and the HTB/tc-based per-user limit enforcement running on every node.
Per-node/per-region traffic and revenue analytics: VPN traffic breakdown (Xray/SSH/ZIVPN payload), historical charts built from the hourly buckets the panel accrues server-side (__nodeHourly in node-heartbeat.ts).
Every heartbeat carries two distinct sets of counters so the panel can separate "server bandwidth" from "customer VPN traffic":
| Field | Meaning |
|---|---|
| nicIn / nicOut | Live VPN throughput (bits/s) — NOT the raw VPS NIC. |
| rxBytes / txBytes | Lifetime customer VPN bytes (idle background VPS traffic never counted). |
| xrayRxBytes/xrayTxBytes, sshRxBytes/sshTxBytes, zivpnRxBytes/zivpnTxBytes | Per-protocol cumulative split of the same VPN bytes. |
| vpsRxBytes/vpsTxBytes, vpsNicIn/vpsNicOut | Raw physical-interface counters, kept separate. |
| nics | Per-interface inventory: name, up/down, speedMbps, mtu, duplex, ip, byte counters, rxBps/txBps, error/drop counts. |
Accounting correctness caveats
- SSH bytes are billed exactly once via the separate POST /api/public/node-ssh-usage channel; the heartbeat zeroes SSH's protocol delta to avoid double-billing.
- Reverse-proxy double counting: raw NIC deltas see every proxied byte twice. When Xray's stats API is unreachable, the agent prefers iptables client-leg counters and only falls back to halving raw NIC deltas as a last resort.
- statsSource: "xray" means authoritative per-user counts; "sockets" means the node is estimating from ss socket counts because Xray's stats API is down.
- MAX_REASONABLE_BPS = 100 Gbps — anything above this is treated as a counter glitch and reported as 0.
- Counter resets are detected centrally (counter_delta.delta_since()) and pushed as a counter.reset alert event rather than silently corrupting totals.
Two panel-triggered RPCs, both signed and time-budgeted so a slow node degrades to a partial result instead of an opaque upstream timeout (node/agent/services/netbench.py):
| RPC | What it does |
|---|---|
net.speedtest | Downloads from Cloudflare's keyless speed endpoint (speed.cloudflare.com/__down), falling back through a mirror list (Hetzner, OVH, tele2); uploads to speed.cloudflare.com/__up. Wall clock budget TOTAL_BUDGET_SEC=55s (NETBENCH_BUDGET_SEC override), ~45% download / remainder upload. Reads the NIC's advertised link speed from /sys/class/net/<iface>/speed as an upper bound. |
net.probe | Parallel ICMP ping (fallback to a raw TCP-connect RTT when ICMP is filtered/unprivileged) against up to 8 targets (default 1.1.1.1, 8.8.8.8, google.com), reporting loss%, min/avg/max RTT and jitter per target plus fleet averages. |
net.speedtest derives serving capacity: usableMbps = min(down, up, linkMbps), monthly bytes at a DUTY_CYCLE=0.25 (25% sustained utilization), andconcurrentUsers = usableMbps / PER_USER_MBPS(4.0). It runs automatically once right after install, and on demand from the panel (bulk-run UI insrc/components/nodes/benchmark-all-nodes.tsx). Both benchmarks run viaasyncio.to_thread on the node so blocking sockets/subprocesses don't stall the agent's own HTTP server or heartbeat.
Assigning an account to a WARP outbound only ever affected Xray natively — SSH and ZIVPN traffic never traverse Xray. node/agent/services/warp.py gives them real kernel-level WARP egress too:
profile_path(policy)locates a wgcf profile: staged from the panel-pushed policy to/etc/wireguard/grvpn-warp.conf(withTable = offso wg-quick doesn't hijack the default route), or one of the static fallback paths.ensure_up()brings up interfacegrvpn-warpviawg-quick up(idempotent), then waits up toHANDSHAKE_TIMEOUT=15sfor a verified WireGuard handshake (wg show grvpn-warp latest-handshakes> 0) before trusting the tunnel.- Installs routing table 51821 with a default route through grvpn-warp for both IPv4 and IPv6.
apply(users, policy)resolves every account whose outboundId matches a WARP outbound to its OS UID (grvpn-<name>SSH login), clears old ip rule entries at priority 8341, then addsip rule add priority 8341 uidrange <uid>-<uid> lookup 51821for both address families.- Fail-closed to direct, never black-hole: if the tunnel doesn't come up, or installing the rule fails for either address family, no rule is installed at all and the account falls back to normal internet.
- clear() is the full uninstall path: flush rules, flush table, wg-quick down.
Verifying WARP is actually applied
Run
wg show grvpn-warp latest-handshakes on the node. If it never handshakes (blocked outbound UDP, bad wgcf profile), no routing rule is installed by design, and the account keeps normal internet rather than losing connectivity.Xray-core has no native "max devices" or "max Mbps" knob, so the node agent implements both inservices/limits.py.
IP limiting
- Tails the Xray access log and samples who/SSH sessions to build username -> {client IP} over a WINDOW_SEC=180s sliding window.
- Presence window for "online now" reporting is ONLINE_WINDOW_SEC=45s (longer-lived streaming tunnels don't emit new access-log lines).
- Surplus IPs beyond the panel-set ipLimit are dropped via iptables chain AUTOSCRIPT_IPLIMIT; SSH sessions over the cap are killed outright.
- Fleet-wide combined limit: the panel aggregates userIps/userConns from every node and pushes back set_fleet_bans() (per-account surplus IPs), authoritative for up to FLEET_TTL=120s before a node falls back to its own local cap.
- Counts live connections per IP, not just presence, so several devices behind one NAT (userConns) are not collapsed into "1 device".
Speed limiting (HTB + tc)
- Primary path: identity-based shaping via firewall marks (
services/marks.py) — two HTB classes per account per direction (MARK_CID_BASE=30100), using cls_fw/act_connmark/xt_connmark/xt_owner so CDN-fronted or IPv6 clients are still classified correctly. The account's Linux UID stamps SO_MARK on outbound Xray sockets, which is why the xray.service systemd override grants CAP_NET_ADMIN and runs Xray as root:root. - Fallback: per-IP classid pool (_MIN_CID=100..._MAX_CID=30000), IFB device ifb-autoscript mirrors ingress so uplink (customer upload) can be shaped too.
- Rate = admin cap exactly, not "cap + headroom" — HTB rate/ceil are set to the configured Kbps; only bucket/queue sizing is tuned for ramp smoothness (BURST_SEC=0.05s, _BURST_FLOOR=16KiB, fq_codel CODEL_TARGET=40ms/CODEL_INTERVAL=400ms, QUEUE_SEC=0.25s).
- Refuses to touch a foreign HTB root it doesn't recognise (_qdisc_is_ours, signature "default 0x9999 r2q 1000") — protects against clobbering the legacy single-server shaper (autoscript-ip-limit), which install.sh actively detects, stops, disables and cleans up.
- Unlimited accounts are actively unshaped (_unshape) and the whole HTB/IFB tree is torn down when no account needs shaping (_cleanup_shaping_if_idle).
- Emits on_event (wired to push_event in heartbeat) when a cap cannot actually be enforced (e.g. peer never appears on the wire — a CDN-fronted client), throttled to once per hour per user (WARN_EVERY=3600s).
Speed cap “does nothing” for a specific account
Usually a CDN-fronted client whose real address never appears in ss output on the node, so per-IP HTB can't match it. Mark-based shaping (uid/connmark) should catch this instead; verify xray.service's systemd override still grants CAP_NET_ADMIN (a botched update can silently strip it —
xray.repair_startup() on agent boot exists specifically to fix this).- Status badges on /nodes reflect online/offline/installing/degraded, driven by heartbeat recency and the RPC circuit breaker's cooldown state.
- A node stuck "installing" means install-complete.json hasn't been written yet or one of the 17 install phases failed — check journalctl -u autoscript-node -f -o cat on the box.
- Amber "Restarting…/Stopping…/Updating…" badges reflect an in-flight RPC action from the panel, not a heartbeat state.
- Cert expiry indicators on /infrastructure read cert.log / cert metadata pushed via heartbeat/logs — treat an approaching expiry as high priority since Xray/nginx will refuse the vhost once the cert lapses.
- Agent version mismatch on /infrastructure flags nodes that need a manual "Update" (remote nodes never self-update; only the panel's own local node does).