View source on GitHub →

ICN Operations Runbooks

Production runbooks for ICN daemon operations.

Runbook Index

Runbook When to Use
Emergency Restart Node unresponsive, need immediate restart
Data Recovery Data loss or corruption detected
Version Upgrade Deploying new ICN version
Security Incident Suspected security breach
Troubleshooting Common issues and fixes
Secrets Rotation Rotating keys, passphrases, certificates
Pilot Vertical Slice Smoke Verify decision→effect→ledger linkage in one command

Quick Reference

Check Node Status

# K8s deployment
kubectl -n icn get pods
kubectl -n icn logs -f deployment/icn-daemon

# Systemd deployment
systemctl status icnd
journalctl -u icnd -f

Emergency Stop

# K8s
kubectl -n icn scale deployment/icn-daemon --replicas=0

# Systemd
systemctl stop icnd

View Metrics

# Prometheus endpoint
curl http://localhost:9100/metrics | grep icn_

# Key metrics to check
curl -s http://localhost:9100/metrics | grep -E "icn_gossip_messages|icn_ledger_entries|icn_trust_edges"

Runbook Template

All runbooks follow this structure:

  1. Summary - What this runbook addresses
  2. Prerequisites - What you need before starting
  3. Procedure - Step-by-step instructions
  4. Verification - How to confirm success
  5. Rollback - How to undo if needed
  6. Related - Links to related runbooks

Environment Variables

Variable Description Default
ICN_KEYSTORE_PASSPHRASE Daemon keystore passphrase (preferred for icnd) (optional)
ICN_PASSPHRASE CLI passphrase (icnctl) and daemon legacy fallback (optional)
ICN_GATEWAY_JWT_SECRET Gateway JWT secret (when gateway enabled) (required if gateway enabled)
KUBECONFIG K8s config (if K8s) ~/.kube/config

Use CLI flags for paths:

  • icnd --data-dir /path --config /path/config.toml
  • icnctl --data-dir /path

Contact

  • On-call: Check PagerDuty/Slack
  • Escalation: See incident response runbook
← Back to Docs