ICN Deployment & Operations Guide

A practical guide for deploying and operating ICN nodes in production environments.

Table of Contents

1. Quick Start
2. System Requirements
3. Installation
4. Configuration
5. Running as a Service
6. Monitoring
7. Backup & Recovery
8. Troubleshooting
9. Security Best Practices

Quick Start

For testing on a local machine:

# Build release binaries
cd icn/
cargo build --release

# Initialize identity
./target/release/icnctl --data-dir /var/lib/icn id init
# Enter passphrase when prompted

# Start daemon
./target/release/icnd --data-dir /var/lib/icn

The daemon will:

1. Load identity from {data_dir}/identity.age (for this example: /var/lib/icn/identity.age)
2. Start QUIC listener on 0.0.0.0:7777 (default)
3. Enable mDNS discovery on LAN
4. Expose Prometheus metrics on :9100/metrics

System Requirements

Minimum (Development/Testing)

• CPU: 2 cores
• RAM: 512 MB
• Disk: 1 GB
• Network: 1 Mbps, low-latency LAN

Recommended (Production)

• CPU: 4+ cores (for signature verification)
• RAM: 2 GB (for gossip buffers, connection pools)
• Disk: 10 GB SSD (for ledger, fast I/O)
• Network: 10+ Mbps, public IPv4/IPv6

Operating System

• Linux: Ubuntu 22.04+, Debian 12+, Fedora 38+ (primary)
• macOS: 12+ (development)
• Windows: WSL2 (development only)

Installation

From Source

# Clone repository
git clone https://github.com/your-org/icn.git
cd icn/icn

# Build release binaries
cargo build --release

# Install to /usr/local/bin (optional)
sudo cp target/release/{icnd,icnctl} /usr/local/bin/

Using Docker

# Dockerfile
FROM rust:1.75 AS builder
WORKDIR /app
COPY icn/ ./
RUN cargo build --release

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/icnd /usr/local/bin/
COPY --from=builder /app/target/release/icnctl /usr/local/bin/

# Data directory
VOLUME ["/data"]
EXPOSE 7777/udp 9100/tcp 8080/tcp
ENTRYPOINT ["icnd", "--data-dir", "/data"]

Build and run:

docker build -t icnd:latest .
docker run -d \
  --name icnd \
  -v icn-data:/data \
  -p 7777:7777/udp \
  -p 9100:9100 \
  -p 8080:8080 \
  icnd:latest

Configuration

Environment Variables

Note: icnd does not currently read ICN_DATA_DIR, ICN_CONFIG, ICN_LISTEN_ADDR, or ICN_METRICS_PORT directly from environment. Use --data-dir, --config, and config file fields.

Passphrase for Automated Deployments:

For systemd, Docker, or Kubernetes deployments where interactive prompts aren't available:

# Option 1: Set in environment (preferred)
export ICN_KEYSTORE_PASSPHRASE="your-secure-passphrase"
icnd --config /etc/icn/config.toml

# Option 2: Use systemd EnvironmentFile
# /etc/icn/icn.env (chmod 600)
ICN_KEYSTORE_PASSPHRASE=your-secure-passphrase

# Option 3: Docker/K8s secrets
docker run -e ICN_KEYSTORE_PASSPHRASE="$(cat /run/secrets/icn_passphrase)" icn:latest

Security Warning: Store passphrases in secrets management systems (HashiCorp Vault, K8s Secrets, AWS Secrets Manager), not plain text files.

Configuration File

ICN supports TOML configuration files. See config/ for examples:

# /etc/icn/config.toml
data_dir = "/var/lib/icn"

[network]
listen_addr = "0.0.0.0:7777"
mdns_enabled = true
bootstrap_peers = []

[observability]
metrics_port = 9100
health_port = 8080
log_level = "info"

CLI Flags

icnd --help

# Common flags:
icnd --config /etc/icn/config.toml
icnd --data-dir /var/lib/icn
icnd --log-level debug

Running as a Service

systemd (Linux)

Create /etc/systemd/system/icnd.service:

[Unit]
Description=ICN Daemon
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=icn
Group=icn
ExecStart=/usr/local/bin/icnd --data-dir /var/lib/icn --config /etc/icn/config.toml
Restart=on-failure
RestartSec=5s

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/icn

# Environment
Environment="ICN_KEYSTORE_PASSPHRASE=REPLACE_ME"
Environment="RUST_LOG=icn=info"

# Limits
LimitNOFILE=65536
LimitNPROC=512

[Install]
WantedBy=multi-user.target

Setup and start:

# Create user
sudo useradd -r -s /bin/false icn
sudo mkdir -p /var/lib/icn
sudo chown icn:icn /var/lib/icn

# Initialize identity (as icn user)
sudo -u icn /usr/local/bin/icnctl --data-dir /var/lib/icn id init

# Enable and start service
sudo systemctl daemon-reload
sudo systemctl enable icnd
sudo systemctl start icnd

# Check status
sudo systemctl status icnd
sudo journalctl -u icnd -f

Docker Compose

See docker/ for complete Docker Compose examples.

version: '3.9'

services:
  icnd:
    image: icn:latest
    container_name: icn-alpha
    restart: unless-stopped
    ports:
      - "7777:7777/udp"
      - "9100:9100"
      - "8080:8080"
    volumes:
      - icn-data:/data
    command: ["icnd", "--data-dir", "/data", "--log-level", "info"]
    networks:
      - icn-net

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9091:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    networks:
      - icn-net

volumes:
  icn-data:
  prometheus-data:

networks:
  icn-net:

Monitoring

Prometheus Configuration

prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'icn'
    static_configs:
      - targets: ['icnd:9100']

Key Metrics

Network Health:

# Connection rate
rate(icn_network_connections_total[5m])

# Active connections
icn_network_connections_active

# Rate limiting events (potential attack)
rate(icn_network_messages_rate_limited_total[5m]) > 10

Gossip Protocol:

# Message throughput
rate(icn_gossip_announces_received_total[5m])
rate(icn_gossip_requests_received_total[5m])

# Total gossip entries
icn_gossip_entries_total

# Subscriptions
icn_gossip_subscriptions_total

Ledger:

# Transaction rate
rate(icn_ledger_transactions_total[5m])

# Account count
icn_ledger_accounts_total

# Currencies in use
icn_ledger_currencies_total

Grafana Dashboard (Example)

Import this JSON or create panels manually:

{
  "panels": [
    {
      "title": "Message Rate",
      "targets": [{
        "expr": "rate(icn_network_messages_received_total[5m])"
      }]
    },
    {
      "title": "Active Connections",
      "targets": [{
        "expr": "icn_network_connections_active"
      }]
    },
    {
      "title": "Rate Limited (Security Alert)",
      "targets": [{
        "expr": "rate(icn_network_messages_rate_limited_total[5m])"
      }],
      "alert": { "condition": "> 10" }
    }
  ]
}

Log Monitoring

# Follow logs (systemd)
journalctl -u icnd -f

# Search for errors
journalctl -u icnd --priority=err

# Security events
journalctl -u icnd | grep -E "Rate limited|Certificate verification|SECURITY"

# Export logs (structured JSON)
journalctl -u icnd -o json > icnd-logs.json

Backup & Recovery

Identity Backup

CRITICAL: The keystore contains your node's cryptographic identity. Loss = permanent identity loss.

# Backup keystore (encrypted with passphrase)
cp /var/lib/icn/identity.age ~/backup/identity.age

# Or use icnctl
icnctl id export ~/backup/identity-backup.age

# Store backup in multiple locations:
# - Encrypted USB drive
# - Password manager (as attachment)
# - Offline storage (paper wallet for recovery phrase - future)

Recovery:

# Restore from backup
icnctl id import ~/backup/identity-backup.age

# Or copy directly
cp ~/backup/identity.age /var/lib/icn/identity.age
chmod 600 /var/lib/icn/identity.age

Ledger Backup

The ledger is a Merkle-DAG synced via gossip, so data is distributed. However, local backups prevent data loss:

# Backup entire data directory
tar czf icn-backup-$(date +%F).tar.gz /var/lib/icn/

# Exclude keystore if backing up separately
tar czf icn-ledger-backup-$(date +%F).tar.gz \
  --exclude='identity.age' \
  /var/lib/icn/

Automated backups (systemd timer):

/etc/systemd/system/icn-backup.service:

[Unit]
Description=ICN Backup

[Service]
Type=oneshot
User=icn
ExecStart=/usr/local/bin/icn-backup.sh

/etc/systemd/system/icn-backup.timer:

[Unit]
Description=ICN Backup Timer

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable: sudo systemctl enable --now icn-backup.timer

Troubleshooting

Node won't start

Symptom: icnd exits immediately

Checklist:

1. Identity initialized? Run icnctl id init
2. Port already in use? Check with sudo lsof -i :7777
3. Permissions? Ensure data directory is readable/writable
4. Logs? Check journalctl -u icnd -n 50

Common errors:

Error: Failed to unlock keystore
→ Passphrase incorrect or keystore corrupted

Error: Address already in use
→ Another instance running or port conflict

Error: Permission denied
→ User lacks write access to data directory

No peers discovered

Symptom: icnctl status shows 0 peers

Checklist:

1. mDNS working? Check with avahi-browse -a (Linux) or dns-sd -B (macOS)
2. Firewall blocking? Open UDP 7777
3. Multiple networks? mDNS doesn't cross subnets
4. Docker network mode? Use host mode for mDNS

Manual peer dialing:

icnctl network dial did:icn:abc123... 192.168.1.100:7777

High memory usage

Symptom: icnd using >2GB RAM

Potential causes:

1. Too many gossip entries: Check icn_gossip_entries_total metric
2. Connection leak: Check icn_network_connections_active
3. Message flood: Check icn_network_messages_rate_limited_total

Actions:

1. Review Prometheus metrics for anomalies
2. Check for misbehaving peers in logs
3. Restart daemon (graceful shutdown preserves state)

Certificate verification errors

Symptom: Connections failing with TLS errors

Checklist:

1. System clock correct? TLS checks certificate validity
2. Certificate expired? Rotate with icnctl id rotate
3. DID format invalid? Check logs for format errors

Note: Trust graph integration pending - currently accepts all valid DIDs.

Security Best Practices

1. Secure Keystore

DO:

• Use strong passphrase (20+ characters, mix of types)
• Store backup in encrypted location (KeePass, 1Password)
• Use separate passphrase from system password
• Restrict permissions: chmod 600 /var/lib/icn/identity.age

DON'T:

• Share keystore file (contains identity)
• Store passphrase in plain text
• Use weak/common passphrases
• Commit keystore to version control

2. Network Exposure

DO:

• Use firewall (ufw, iptables) to limit exposure
• Enable rate limiting (on by default)
• Monitor metrics for anomalies
• Use VPN or WireGuard for WAN peers

DON'T:

• Expose metrics port (9100) publicly
• Disable rate limiting without good reason
• Run as root (use dedicated user)

3. Updates

DO:

• Subscribe to security announcements (GitHub watch)
• Test updates on staging before production
• Review changelogs for breaking changes
• Use versioned releases (not main branch)

DON'T:

• Auto-update in production without testing
• Skip patch releases (may contain security fixes)
• Run multiple versions in same network (protocol mismatch)

4. Monitoring

DO:

• Set up alerts for rate limiting spikes
• Monitor connection count and patterns
• Review logs regularly for suspicious activity
• Track trust graph changes

DON'T:

• Ignore rate limiting metrics
• Run without monitoring in production
• Expose Prometheus publicly without auth

Additional Resources

• Architecture Documentation
• Production Hardening Details
• Topic Subscriptions API
• GitHub Repository
• Community Forum (future)

Support

For issues:

1. Check this guide and architecture docs
2. Search GitHub issues
3. Open new issue with logs and system info
4. Security issues: security@intercooperative.network (future)

Include in bug reports:

• ICN version (icnd --version)
• OS and architecture
• Relevant logs (journalctl -u icnd -n 100)
• Metrics snapshots (if available)