tyto/docs/troubleshooting.md

# Troubleshooting Guide

Solutions for common issues with Tyto.

## Quick Diagnostics

```bash
# Check service status
systemctl status tyto

# View recent logs
journalctl -u tyto -n 50

# Test API
curl http://localhost:8080/health

# Check listening ports
ss -tlnp | grep tyto
```

## Installation Issues

### Install Script Fails

**Symptom**: `curl | bash` fails or binary not found.

**Solutions**:

1. Check internet connectivity:
   ```bash
   curl -I https://somegit.dev
   ```

2. Download manually:
   ```bash
   wget https://somegit.dev/vikingowl/tyto/releases/latest/download/tyto-linux-amd64.tar.gz
   tar -xzf tyto-linux-amd64.tar.gz
   sudo mv tyto /usr/local/bin/
   ```

3. Check architecture:
   ```bash
   uname -m
   # x86_64 → amd64, aarch64 → arm64
   ```

### Permission Denied

**Symptom**: Cannot access `/proc`, `/sys`, or other system paths.

**Solutions**:

1. For Docker, ensure volume mounts:
   ```yaml
   volumes:
     - /proc:/host/proc:ro
     - /sys:/host/sys:ro
   ```

2. For native install, check user permissions:
   ```bash
   sudo usermod -aG docker tyto  # For Docker access
   ```

3. Run with elevated privileges (not recommended for production):
   ```bash
   sudo tyto
   ```

## Startup Issues

### Service Won't Start

**Symptom**: `systemctl start tyto` fails.

**Check logs**:
```bash
journalctl -u tyto -e
```

**Common causes**:

1. **Configuration error**:
   ```bash
   tyto validate-config --config /etc/tyto/config.yaml
   ```

2. **Port already in use**:
   ```bash
   ss -tlnp | grep 8080
   # Kill conflicting process or change port
   ```

3. **Database connection failed**:
   ```bash
   # For PostgreSQL
   psql -h localhost -U tyto -d tyto -c "SELECT 1"
   ```

4. **Certificate file not found**:
   ```bash
   ls -la /etc/tyto/certs/
   ```

### Docker Container Exits

**Check logs**:
```bash
docker compose logs backend
```

**Common causes**:

1. **Missing environment variables**:
   ```bash
   docker compose config  # Validate compose file
   ```

2. **Volume mount issues**:
   ```bash
   docker inspect tyto-backend | grep Mounts
   ```

3. **Build needed**:
   ```bash
   docker compose build --no-cache
   docker compose up -d
   ```

## Metrics Not Displaying

### No Data in Dashboard

**Check backend is running**:
```bash
curl http://localhost:8080/health
```

**Check SSE connection**:
```bash
curl -N http://localhost:8080/api/v1/stream
# Should see JSON data every few seconds
```

**Browser console errors**:
- Open DevTools (F12)
- Check Console and Network tabs
- Look for CORS or connection errors

### Missing Specific Metrics

#### CPU Metrics Missing

```bash
# Check /proc/stat access
cat /proc/stat

# For Docker
docker exec tyto-backend cat /host/proc/stat
```

#### Memory Metrics Missing

```bash
# Check /proc/meminfo access
cat /proc/meminfo
```

#### Disk Metrics Missing

```bash
# Check mount table
cat /etc/mtab

# Check disk stats
cat /proc/diskstats
```

#### Network Metrics Missing

```bash
# Check network stats
cat /proc/net/dev
```

#### Temperature Missing

```bash
# Check hwmon sensors
ls /sys/class/hwmon/

# For each sensor
cat /sys/class/hwmon/hwmon0/temp1_input
```

### GPU Not Detected

#### NVIDIA GPU

```bash
# Check nvidia-smi
nvidia-smi

# For Docker, add GPU access
docker run --gpus all tyto
```

#### AMD GPU

```bash
# Check amdgpu driver
ls /sys/class/drm/card*/device/

# Check GPU metrics files
cat /sys/class/drm/card0/device/gpu_busy_percent
```

#### Intel GPU

```bash
# Check i915 driver
ls /sys/class/drm/card*/driver

# Should show i915 or xe
```

### Docker Containers Not Showing

```bash
# Check Docker socket mount
docker exec tyto-backend ls -la /var/run/docker.sock

# Check Docker API access
docker exec tyto-backend curl --unix-socket /var/run/docker.sock http://localhost/containers/json
```

### Systemd Services Not Showing

```bash
# Check D-Bus socket mount
docker exec tyto-backend ls -la /run/dbus/system_bus_socket

# Test D-Bus access
docker exec tyto-backend busctl list
```

## Agent Issues

### Agent Not Connecting

**Check network connectivity**:
```bash
# From agent host
telnet tyto-server.example.com 9849
nc -zv tyto-server.example.com 9849
```

**Check certificates**:
```bash
# Verify certificate chain
openssl verify -CAfile /etc/tyto/certs/ca.crt /etc/tyto/certs/agent.crt

# Check certificate expiration
openssl x509 -in /etc/tyto/certs/agent.crt -enddate -noout

# Check certificate details
openssl x509 -in /etc/tyto/certs/agent.crt -text -noout | grep -A1 Subject
```

**Check server is in server mode**:
```bash
# On server
curl http://localhost:8080/health
# Response should indicate server mode
```

### Agent Shows "Pending"

1. Log into dashboard
2. Navigate to **Agents** → **Pending**
3. Approve the agent

Or via CLI:
```bash
tyto agents approve agent-id
```

### Agent Certificate Revoked

Generate new certificate:
```bash
# On server
tyto pki gen-agent --ca-dir /etc/tyto/pki/ --agent-id agent-id --out /tmp/new-certs/

# Copy to agent
scp /tmp/new-certs/* user@agent-host:/etc/tyto/certs/

# Restart agent
ssh user@agent-host 'sudo systemctl restart tyto'
```

## Authentication Issues

### Cannot Login

**Check credentials**:
```bash
# Test login API
curl -X POST http://localhost:8080/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "password"}'
```

**Reset admin password**:
```bash
tyto user reset-password --username admin --password new-password
```

**Check LDAP connectivity** (if using LDAP):
```bash
ldapsearch -x -H ldap://ad.example.com:389 \
  -D "cn=svc-tyto,dc=example,dc=com" \
  -W -b "dc=example,dc=com" "(sAMAccountName=testuser)"
```

### Session Expired

- Clear browser cookies
- Log in again
- Check server clock sync

### Permission Denied

1. Check user roles: **Admin** → **Users** → Select user
2. Verify role has required permission: **Admin** → **Roles**
3. Check role assignment

## Database Issues

### SQLite Locked

**Symptom**: `database is locked` errors.

**Solutions**:
```bash
# Stop service
sudo systemctl stop tyto

# Check for stale locks
fuser /var/lib/tyto/tyto.db

# Remove lock file if exists
rm -f /var/lib/tyto/tyto.db-journal

# Restart
sudo systemctl start tyto
```

### PostgreSQL Connection Failed

**Check connectivity**:
```bash
psql -h localhost -U tyto -d tyto -c "SELECT 1"
```

**Check connection string**:
```bash
# Verify URL format
echo $TYTO_DB_URL
# Should be: postgres://user:pass@host:5432/dbname
```

**Check PostgreSQL logs**:
```bash
sudo tail -f /var/log/postgresql/postgresql-16-main.log
```

### Migration Failed

**View migration status**:
```bash
tyto db migrate-status
```

**Force migration** (use with caution):
```bash
tyto db migrate --force
```

## Performance Issues

### High CPU Usage

**Check collection interval**:
```yaml
# Increase interval to reduce load
refresh_rate: 10  # seconds
```

**Disable unused collectors**:
```yaml
collectors:
  docker: false
  systemd: false
```

### High Memory Usage

**Check for memory leaks**:
```bash
# Monitor memory over time
watch -n 5 'ps aux | grep tyto'
```

**Limit history size**:
```yaml
history:
  max_entries: 720  # 1 hour at 5s intervals
```

### Slow Dashboard

1. Increase refresh rate (R key)
2. Reduce visible cards
3. Check network latency to backend
4. Check browser console for errors

## Log Issues

### Logs Not Appearing

**Check log collector config**:
```yaml
agent:
  logs:
    enabled: true
    journal:
      enabled: true
```

**Check journalctl access**:
```bash
journalctl -n 10
```

**Check file permissions for file logs**:
```bash
ls -la /var/log/nginx/
```

### Log Storage Full

**Check database size**:
```bash
du -h /var/lib/tyto/tyto.db
```

**Adjust retention**:
```yaml
retention:
  logs: 72h  # Reduce from 168h
```

**Manual cleanup**:
```bash
tyto db cleanup --older-than 72h
```

## Network Issues

### CORS Errors

**Check backend CORS config**:
```yaml
http:
  cors:
    allowed_origins:
      - https://tyto.example.com
```

**For development**:
```yaml
http:
  cors:
    allowed_origins:
      - "*"
```

### SSE Connection Drops

**Nginx buffering** - disable for SSE:
```nginx
location /api/v1/stream {
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 86400s;
}
```

**Load balancer timeout** - increase timeout:
```nginx
proxy_read_timeout 3600s;
```

### TLS/Certificate Errors

**Verify certificate**:
```bash
openssl s_client -connect tyto.example.com:443 -servername tyto.example.com
```

**Check certificate chain**:
```bash
openssl verify -CAfile ca.crt server.crt
```

## Getting Help

### Collect Debug Information

```bash
# System info
uname -a
cat /etc/os-release

# Tyto version
tyto version

# Configuration (sanitize passwords!)
cat /etc/tyto/config.yaml

# Recent logs
journalctl -u tyto -n 100 > tyto-logs.txt

# Docker info
docker compose ps
docker compose logs > docker-logs.txt
```

### Report Issues

Open an issue at: https://somegit.dev/vikingowl/tyto/issues

Include:
- Tyto version
- Operating system
- Deployment method (Docker/native)
- Error messages
- Steps to reproduce