# Troubleshooting

Common issues and their resolutions for FluxiQ PIX.

## Prerequisites

- Access to backend logs (`kubectl logs` or Docker logs)
- Familiarity with Elixir/OTP error messages
- Access to PostgreSQL, Redis, and NATS for diagnostics

## Quick Diagnostics

```bash
# Check all pods
kubectl get pods -n pix

# Backend logs (last 100 lines)
kubectl logs -n pix deployment/pix-backend --tail=100

# Health check
curl -s http://localhost:4003/health | jq .

# Database connectivity
kubectl exec -n pix deployment/pix-backend -- bin/monetarie_pix eval "Shared.Repo.query!('SELECT 1')"
```

## Common Issues

### Backend Fails to Start

**Symptom**: Pod in CrashLoopBackOff

**Causes and Solutions**:

| Cause | Log Pattern | Solution |
|-------|-------------|----------|
| Missing SECRET_KEY_BASE | `KeyError: SECRET_KEY_BASE` | Set secret in K8s Secret |
| Database unreachable | `Postgrex.Error: tcp connect` | Check DB_HOST and network |
| Invalid DATABASE_URL | `URI.parse error` | URL-encode special characters |
| Port conflict | `eaddrinuse` | Check no other process on 4001-4003 |

### Migrations Fail

**Symptom**: `Shared.Release.migrate()` errors

```bash
# Check migration status
kubectl exec -n pix deployment/pix-backend -- \
  bin/monetarie_pix eval "Ecto.Migrator.migrations(Shared.Repo, \"apps/shared/priv/repo/migrations\")"
```

**Common causes**:
- Extension not installed: Run `CREATE EXTENSION IF NOT EXISTS "uuid-ossp"` and `pg_trgm`
- Table already exists: Migrations handle DROP + CREATE for legacy tables
- Permission denied: Database user needs CREATE/ALTER/DROP privileges

### NATS Workers Not Starting

**Symptom**: No worker log messages

**Checklist**:
1. Verify `NATS_ENABLED=true` in environment
2. Check NATS connectivity: `nats server check connection`
3. Verify JetStream is enabled on NATS server
4. Check logs for `"NATS not enabled, skipping worker init"`

### CoreEventProcessor Timeout

**Symptom**: `Finch.Error: timeout` in logs

**Causes**:
- DICT service not responding (check `DICT_SERVICE_URL`)
- MONETARIE_CORE stream not created (Core team dependency)
- Network issue between pods

**Resolution**:
```bash
# Verify DICT service
curl http://localhost:4001/api/health

# Check if MONETARIE_CORE stream exists
nats stream info MONETARIE_CORE
```

### Login Failures

| Error | Cause | Solution |
|-------|-------|----------|
| 401 Unauthorized | Wrong credentials | Verify username/password |
| 429 Too Many Requests | Rate limited | Wait 15 minutes or check Redis |
| Account blocked | 10 failed attempts | Admin must unblock user |
| MFA required | MFA enabled | Complete MFA verification step |

### Circuit Breaker Open

**Symptom**: HTTP 503 responses from Settlement service

```bash
# Check circuit breaker state via logs
kubectl logs -n pix deployment/pix-backend | grep "circuit_breaker"
```

**Resolution**: Circuit breaker auto-recovers after 30 seconds (half-open state). If persistent, check the failing downstream service.

### GIN Trigram Index Failure

**Symptom**: Migration error on CHAR column with `gin_trgm_ops`

**Solution**: Cast CHAR to text in the index expression:
```sql
CREATE INDEX idx_e2e_trgm ON monetarie_spi.messages USING gin ((end_to_end_id::text) gin_trgm_ops);
```

### WebSocket Connection Failed

**Symptom**: Monitoring dashboard shows "Disconnected"

**Checklist**:
1. JWT token valid and not expired
2. nginx/Traefik configured for WebSocket upgrade
3. Backend running Settlement Service on port 4003
4. Check browser console for WebSocket errors

### Redis Connection Issues

**Symptom**: Rate limiting or idempotency not working

```bash
# Test Redis connectivity
redis-cli -h $REDIS_HOST -p $REDIS_PORT ping

# Check Redis memory
redis-cli -h $REDIS_HOST INFO memory
```

## Log Correlation

All requests carry an `x-trace-id` header for distributed tracing:

```bash
# Find all logs for a specific trace
kubectl logs -n pix deployment/pix-backend | grep "trace_id=abc123"
```

NATS messages also carry `trace_id` in message headers for end-to-end correlation.

## Useful Elixir Diagnostics

```bash
# Remote console (development only)
kubectl exec -it -n pix deployment/pix-backend -- bin/monetarie_pix remote

# Check running processes
Process.list() |> length()

# Check ETS tables
:ets.all() |> length()

# Check memory
:erlang.memory()
```

## Expected Outcome

After using this troubleshooting guide:

- Common issues identified and resolved quickly
- Log correlation enabling end-to-end request tracing
- Understanding of circuit breaker, rate limiter, and NATS worker behavior
- Diagnostic commands for all infrastructure components