The AutoPilot Architecture & Incident Radar

Most cloud platforms treat "AI" as a passive chat widget that searches documentation or suggests CLI commands for an engineer to copy and paste at 3:00 AM during an outage.

RasnuOS AI AutoPilot is a continuous, event-driven Site Reliability Engineering (SRE) engine. It combines real-time 15-second host anomaly detection with smart alert debouncing, scheduled Maintenance Window suppression, configurable autonomy modes (Notify Only, Co-Pilot, and Full AutoPilot), multi-channel alert fanout, and live root-cause investigation inside the Incident Radar.

---

How an Incident Flows Through AutoPilot

Rendering diagram…

---

1. Real-Time Anomaly Detection & Debouncing (Every 15 Seconds)

Every 15 seconds, the RasnuOS Agent collects CPU usage, memory consumption, root disk utilization, swap pressure, disk IO wait, 5-minute load average, top processes, service states, and kernel Out-Of-Memory (OOM) events.

To avoid waking your team for brief 3-second CPU spikes during a build job—while still reacting immediately when a critical daemon crashes—the Agent divides anomalies into Debounced Resource Alerts (which must exceed the threshold for 3 consecutive 15-second intervals = 45 seconds) and Immediate P0 Crash/Kernel Alerts (zero delay):

Trigger Type	Default Threshold	Debounce Window	Priority & Context Captured
oom_kill	Linux Kernel OOM Killer invoked	Immediate (0s) — Highest Priority	Overwrites lower alerts; captures exact kernel OOM log line & top memory processes.
service_crash	Monitored service enters failed state	Immediate (0s) — P0 Priority	Overwrites resource alerts; captures crashed daemon name (nginx, postgresql, mysql, etc.).
cpu_spike	> 95.0% CPU utilization	3 consecutive checks (45s)	Captures live top_processes table (PID, Name, CPU%, RAM%).
ram_spike	> 95.0% RAM utilization	3 consecutive checks (45s)	Captures live top_processes table sorted by memory usage.
disk_full	> 90.0% Disk utilization	3 consecutive checks (45s)	Captures disk usage percentage and active processes.
swap_pressure	> 80.0% Swap utilization	3 consecutive checks (45s)	Detects severe memory thrashing before the kernel OOM killer triggers.
io_wait	> 50.0% Disk IO Wait	3 consecutive checks (45s)	Detects storage bottlenecks and runaway disk write processes.
high_load	5m Load > CPU Cores × 2.0	3 consecutive checks (45s)	Scales automatically with the server's logical CPU core count.

Custom Thresholds & 15-Minute Alert Cooldown

• Live Threshold Customization: You can customize the cpu, ram, disk, swap, iowait, and load_multiplier thresholds per Server Group or individual server. Updated thresholds are pushed to the Agent immediately without restarting any services.
• 15-Minute Host Cooldown: Once the Agent dispatches an alert and receives acknowledgment from the Control Plane, it starts a 15-minute local cooldown timer on that server so a sustained incident never floods your notification channels.

---

2. Smart Filtering: Maintenance Windows, Deduplication & Scope

When an alert arrives at the RasnuOS Control Plane, it passes through four operational filters before triggering an AI investigation:

1. Scheduled Maintenance Windows: If the server is currently inside a configured Maintenance Window (for example, Sunday 02:00–04:00 UTC during weekly database maintenance), RasnuOS automatically suppresses the alert so planned restarts never trigger false-positive incidents.
2. Critical Services Allowlist (critical_services): You can optionally configure AutoPilot to watch only a specific list of mission-critical services on a server (such as ["nginx", "postgresql", "redis-server"]), ignoring non-critical background workers.
3. 60-Minute Incident Deduplication: If an active incident for the same server and trigger type is already in Investigating, Remediating, or Escalated status within the past 60 minutes, RasnuOS attaches the event to the existing incident rather than opening duplicate tickets.
4. 24-Hour Anti-Loop Protection & Token Budget Guard: To prevent a chronic hardware issue from consuming your AI token budget in an endless loop, RasnuOS enforces a 24-hour per-trigger lock (server + trigger_type) on autonomous AutoPilot runs and verifies your organization's monthly token budget before launching an investigation.

---

3. Choosing Your Autonomy Mode (Action Mode)

Different environments demand different levels of automation. You may want Full AutoPilot on your staging fleet, Co-Pilot on your production API nodes, and Notify Only on a legacy database server. RasnuOS lets you configure three distinct Action Modes:

Action Mode	Read-Only AI Diagnostics	Low-Risk Remediation (e.g., Service Restart)	High-Risk / Destructive Operations
1. Notify Only (notify_only)	Disabled (Zero AI tokens used)	Disabled	Disabled — Opens an Escalated incident on the Incident Radar and dispatches alerts immediately.
2. Co-Pilot (copilot)	Automatic — Runs read-only commands (journalctl, ps, df) to find the root cause.	Requires Human Approval — Presents an Approval Card with the fix and rollback plan.	Requires Human Approval — Pauses at an Approval Card (and optional external CAB ticket).
3. Full AutoPilot (full_autopilot)	Automatic — Runs read-only commands to diagnose the root cause.	Automatic — Executes verified safe fixes (such as restarting a crashed service or clearing /tmp).	Requires Human Approval — The AI Safety Engine still intercepts and gates any destructive command!

IMPORTANT

Hard Safety Boundary in Full AutoPilot Mode: Even when a server is set to Full AutoPilot, the AI can never bypass the RasnuOS Safety Engine or the Agent's compiled-in host guardrails. If resolving an incident requires a high-risk or destructive command (such as modifying firewall rules, dropping database tables, or deleting files outside /tmp), AutoPilot automatically pauses and requests human approval.

---

4. Multi-Channel Alert Fanout

Whenever an incident is triggered or changes status, RasnuOS notifies your on-call team across your configured alert channels:

• Telegram Bot Integration: Sends instant formatted alerts displaying the server name, IP address, trigger type, metric value, and a direct link to the live Incident Radar view.
• Email Dispatch: Sends detailed HTML incident alerts to organization administrators.
• Cryptographically Signed Webhooks (X-RasnuOS-Signature): Sends real-time JSON payloads signed with HMAC-SHA256 to Slack, PagerDuty, Discord, Microsoft Teams, or your custom SIEM endpoint:

{
  "event": "incident.triggered",
  "incident_id": 409,
  "server": {
    "id": 14,
    "name": "prod-api-01",
    "ip": "10.0.1.20"
  },
  "trigger_type": "service_crash",
  "metric_value": 100.0,
  "status": "investigating",
  "action_mode": "copilot",
  "message": "CRITICAL: Service 'nginx' has unexpectedly crashed (Status: failed).",
  "triggered_at": "2026-10-13T21:20:15+00:00"
}

---

5. Tracking Incidents on the Incident Radar

The Incident Radar gives your engineering team a single real-time command center for every anomaly across your fleet. Each incident moves through four clear lifecycle states:

1. Investigating: AutoPilot has received the alert and is streaming live read-only diagnostics (journalctl -xeu nginx, dmesg -T, ss -tulnp, df -h) to isolate the root cause.
2. Remediating: AutoPilot has pinpointed the root cause (for example, "Nginx failed to reload due to an invalid upstream certificate path in /etc/nginx/sites-enabled/app.conf"), formulated the fix and rollback plan, and is either executing the fix or awaiting your click on the Approval Card.
3. Resolved: Post-remediation verification confirms that the service is running and metrics have returned below threshold. The Incident Radar records the total resolution time, root-cause summary, and complete audit trail.
4. Escalated: The incident requires human intervention (or occurred in Notify Only mode) and has been paged out to the on-call engineering team.

---

6. Per-Server & Server-Group AutoPilot Configuration

You can configure AutoPilot globally for an entire Server Group (for example, Production API Fleet) or override settings on individual servers:

{
  "autopilot_enabled": true,
  "action_mode": "copilot",
  "critical_services": [
    "nginx",
    "postgresql",
    "redis-server",
    "docker"
  ],
  "thresholds": {
    "cpu": 95.0,
    "ram": 95.0,
    "disk": 90.0,
    "swap": 80.0,
    "iowait": 50.0,
    "load_multiplier": 2.0
  },
  "maintenance_window": {
    "enabled": true,
    "day_of_week": "Sunday",
    "start_time_utc": "02:00",
    "end_time_utc": "04:00"
  }
}

---

7. Real-World AutoPilot Incident Walkthroughs

Scenario A: Disk Exhaustion (disk_full > 90% for 45 Seconds)

1. Detection: The Agent detects root disk usage at 92.4% across three consecutive 15-second checks and fires a disk_full alert.
2. Read-Only Diagnosis: AutoPilot runs df -h, du -xhd1 /var/log, and journalctl --disk-usage, discovering 18 GB of uncompressed rotated systemd journal logs and 6 GB of dangling Docker build cache.
3. Remediation & Rollback: AutoPilot proposes sudo journalctl --vacuum-time=7d && sudo docker builder prune -af --filter "until=168h", reclaiming 24 GB of disk space and resolving the incident automatically.

Scenario B: Unexpected Service Crash (service_crash on nginx)

1. Detection: Within 15 seconds of nginx entering a failed state, the Agent fires an immediate P0 service_crash alert (bypassing the 45-second debounce).
2. Read-Only Diagnosis: AutoPilot inspects sudo systemctl status nginx and sudo nginx -t, identifying a duplicate listen 443 directive introduced in a recent virtual host file.
3. Safe Recovery: In Co-Pilot mode, AutoPilot highlights the exact offending line and presents a 1-click Approval Card to restore the .bak_rasnuos virtual host backup and reload Nginx cleanly.