tyto/docs/troubleshooting.md

Troubleshooting Guide

Solutions for common issues with Tyto.

Quick Diagnostics

# Check service status
systemctl status tyto

# View recent logs
journalctl -u tyto -n 50

# Test API
curl http://localhost:8080/health

# Check listening ports
ss -tlnp | grep tyto

Installation Issues

Install Script Fails

Symptom: curl | bash fails or binary not found.

Solutions:

1. Check internet connectivity:

   curl -I https://somegit.dev
   

2. Download manually:

   wget https://somegit.dev/vikingowl/tyto/releases/latest/download/tyto-linux-amd64.tar.gz
   tar -xzf tyto-linux-amd64.tar.gz
   sudo mv tyto /usr/local/bin/
   

3. Check architecture:

   uname -m
   # x86_64 → amd64, aarch64 → arm64
   

Permission Denied

Symptom: Cannot access /proc, /sys, or other system paths.

Solutions:

1. For Docker, ensure volume mounts:

   volumes:
     - /proc:/host/proc:ro
     - /sys:/host/sys:ro
   

2. For native install, check user permissions:

   sudo usermod -aG docker tyto  # For Docker access
   

3. Run with elevated privileges (not recommended for production):

   sudo tyto
   

Startup Issues

Service Won't Start

Symptom: systemctl start tyto fails.

Check logs:

journalctl -u tyto -e

Common causes:

1. Configuration error:

   tyto validate-config --config /etc/tyto/config.yaml
   

2. Port already in use:

   ss -tlnp | grep 8080
   # Kill conflicting process or change port
   

3. Database connection failed:

   # For PostgreSQL
   psql -h localhost -U tyto -d tyto -c "SELECT 1"
   

4. Certificate file not found:

   ls -la /etc/tyto/certs/
   

Docker Container Exits

Check logs:

docker compose logs backend

Common causes:

1. Missing environment variables:

   docker compose config  # Validate compose file
   

2. Volume mount issues:

   docker inspect tyto-backend | grep Mounts
   

3. Build needed:

   docker compose build --no-cache
   docker compose up -d
   

Metrics Not Displaying

No Data in Dashboard

Check backend is running:

curl http://localhost:8080/health

Check SSE connection:

curl -N http://localhost:8080/api/v1/stream
# Should see JSON data every few seconds

Browser console errors:

• Open DevTools (F12)
• Check Console and Network tabs
• Look for CORS or connection errors

Missing Specific Metrics

CPU Metrics Missing

# Check /proc/stat access
cat /proc/stat

# For Docker
docker exec tyto-backend cat /host/proc/stat

Memory Metrics Missing

# Check /proc/meminfo access
cat /proc/meminfo

Disk Metrics Missing

# Check mount table
cat /etc/mtab

# Check disk stats
cat /proc/diskstats

Network Metrics Missing

# Check network stats
cat /proc/net/dev

Temperature Missing

# Check hwmon sensors
ls /sys/class/hwmon/

# For each sensor
cat /sys/class/hwmon/hwmon0/temp1_input

GPU Not Detected

NVIDIA GPU

# Check nvidia-smi
nvidia-smi

# For Docker, add GPU access
docker run --gpus all tyto

AMD GPU

# Check amdgpu driver
ls /sys/class/drm/card*/device/

# Check GPU metrics files
cat /sys/class/drm/card0/device/gpu_busy_percent

Intel GPU

# Check i915 driver
ls /sys/class/drm/card*/driver

# Should show i915 or xe

Docker Containers Not Showing

# Check Docker socket mount
docker exec tyto-backend ls -la /var/run/docker.sock

# Check Docker API access
docker exec tyto-backend curl --unix-socket /var/run/docker.sock http://localhost/containers/json

Systemd Services Not Showing

# Check D-Bus socket mount
docker exec tyto-backend ls -la /run/dbus/system_bus_socket

# Test D-Bus access
docker exec tyto-backend busctl list

Agent Issues

Agent Not Connecting

Check network connectivity:

# From agent host
telnet tyto-server.example.com 9849
nc -zv tyto-server.example.com 9849

Check certificates:

# Verify certificate chain
openssl verify -CAfile /etc/tyto/certs/ca.crt /etc/tyto/certs/agent.crt

# Check certificate expiration
openssl x509 -in /etc/tyto/certs/agent.crt -enddate -noout

# Check certificate details
openssl x509 -in /etc/tyto/certs/agent.crt -text -noout | grep -A1 Subject

Check server is in server mode:

# On server
curl http://localhost:8080/health
# Response should indicate server mode

Agent Shows "Pending"

1. Log into dashboard
2. Navigate to Agents → Pending
3. Approve the agent

Or via CLI:

tyto agents approve agent-id

Agent Certificate Revoked

Generate new certificate:

# On server
tyto pki gen-agent --ca-dir /etc/tyto/pki/ --agent-id agent-id --out /tmp/new-certs/

# Copy to agent
scp /tmp/new-certs/* user@agent-host:/etc/tyto/certs/

# Restart agent
ssh user@agent-host 'sudo systemctl restart tyto'

Authentication Issues

Cannot Login

Check credentials:

# Test login API
curl -X POST http://localhost:8080/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "password"}'

Reset admin password:

tyto user reset-password --username admin --password new-password

Check LDAP connectivity (if using LDAP):

ldapsearch -x -H ldap://ad.example.com:389 \
  -D "cn=svc-tyto,dc=example,dc=com" \
  -W -b "dc=example,dc=com" "(sAMAccountName=testuser)"

Session Expired

• Clear browser cookies
• Log in again
• Check server clock sync

Permission Denied

1. Check user roles: Admin → Users → Select user
2. Verify role has required permission: Admin → Roles
3. Check role assignment

Database Issues

SQLite Locked

Symptom: database is locked errors.

Solutions:

# Stop service
sudo systemctl stop tyto

# Check for stale locks
fuser /var/lib/tyto/tyto.db

# Remove lock file if exists
rm -f /var/lib/tyto/tyto.db-journal

# Restart
sudo systemctl start tyto

PostgreSQL Connection Failed

Check connectivity:

psql -h localhost -U tyto -d tyto -c "SELECT 1"

Check connection string:

# Verify URL format
echo $TYTO_DB_URL
# Should be: postgres://user:pass@host:5432/dbname

Check PostgreSQL logs:

sudo tail -f /var/log/postgresql/postgresql-16-main.log

Migration Failed

View migration status:

tyto db migrate-status

Force migration (use with caution):

tyto db migrate --force

Performance Issues

High CPU Usage

Check collection interval:

# Increase interval to reduce load
refresh_rate: 10  # seconds

Disable unused collectors:

collectors:
  docker: false
  systemd: false

High Memory Usage

Check for memory leaks:

# Monitor memory over time
watch -n 5 'ps aux | grep tyto'

Limit history size:

history:
  max_entries: 720  # 1 hour at 5s intervals

Slow Dashboard

1. Increase refresh rate (R key)
2. Reduce visible cards
3. Check network latency to backend
4. Check browser console for errors

Log Issues

Logs Not Appearing

Check log collector config:

agent:
  logs:
    enabled: true
    journal:
      enabled: true

Check journalctl access:

journalctl -n 10

Check file permissions for file logs:

ls -la /var/log/nginx/

Log Storage Full

Check database size:

du -h /var/lib/tyto/tyto.db

Adjust retention:

retention:
  logs: 72h  # Reduce from 168h

Manual cleanup:

tyto db cleanup --older-than 72h

Network Issues

CORS Errors

Check backend CORS config:

http:
  cors:
    allowed_origins:
      - https://tyto.example.com

For development:

http:
  cors:
    allowed_origins:
      - "*"

SSE Connection Drops

Nginx buffering - disable for SSE:

location /api/v1/stream {
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 86400s;
}

Load balancer timeout - increase timeout:

proxy_read_timeout 3600s;

TLS/Certificate Errors

Verify certificate:

openssl s_client -connect tyto.example.com:443 -servername tyto.example.com

Check certificate chain:

openssl verify -CAfile ca.crt server.crt

Getting Help

Collect Debug Information

# System info
uname -a
cat /etc/os-release

# Tyto version
tyto version

# Configuration (sanitize passwords!)
cat /etc/tyto/config.yaml

# Recent logs
journalctl -u tyto -n 100 > tyto-logs.txt

# Docker info
docker compose ps
docker compose logs > docker-logs.txt

Report Issues

Open an issue at: https://somegit.dev/vikingowl/tyto/issues

Include:

• Tyto version
• Operating system
• Deployment method (Docker/native)
• Error messages
• Steps to reproduce