# Monitoring

Real-time monitoring through WebSocket channels, Prometheus metrics, and health checks.

## Prerequisites

- FluxiQ PIX backend running with `NATS_ENABLED=true`
- Admin portal access for the WebSocket dashboard
- Prometheus configured for metrics scraping (optional)

## WebSocket Monitoring

The admin portal connects to 6 Phoenix Channel topics via WebSocket at `/socket`:

| Channel | Topic | Data |
|---------|-------|------|
| Transactions | `transactions:live` | Live transaction feed (status changes, new payments) |
| System Health | `system:health` | Service status, DB, Redis, NATS health |
| Settlement | `settlement:status` | Settlement cycle progress and results |
| DICT Operations | `dict:operations` | Key lookups, claims, CID sync events |
| BACEN Channels | `bacen:channels` | CPM/CSM channel health and failover events |
| Queue Depth | `queues:depth` | NATS consumer lag per stream |

### WebSocket Authentication

Socket connections require a valid JWT token:

```javascript
import { Socket } from "phoenix"

const socket = new Socket("/socket", {
  params: { token: jwtToken }
})
socket.connect()

const channel = socket.channel("transactions:live")
channel.join()
channel.on("new_transaction", (payload) => { ... })
```

## Health Checks

### Endpoints

| Endpoint | Port | Checks |
|----------|------|--------|
| `GET /health` | 4003 | Settlement service + DB + Redis + NATS |
| `GET /api/health` | 4001 | Dict service health |
| `GET /api/health` | 4002 | SPI service health |
| `GET /metrics` | 4003 | Prometheus metrics |

### HealthChecker GenServer

`SettlementService.Monitoring.HealthChecker` runs periodic checks:

- **Every 10 seconds**: Service health (Dict, SPI, Settlement), DB connectivity, Redis ping, NATS process alive
- **Every 30 seconds**: BACEN channel health (CPM/CSM)
- State transition logging: `healthy -> degraded -> down`

### QueueMonitor GenServer

`SettlementService.Monitoring.QueueMonitor` polls NATS consumer lag every 15 seconds.

## Prometheus Metrics

The `/metrics` endpoint exposes Telemetry-based metrics:

### Request Metrics

| Metric | Type | Labels |
|--------|------|--------|
| `phoenix_http_request_duration_ms` | Histogram | method, route, status |
| `phoenix_http_request_count` | Counter | method, route, status |

### Database Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `ecto_query_duration_ms` | Histogram | Query execution time |
| `ecto_pool_size` | Gauge | Connection pool size |
| `ecto_pool_available` | Gauge | Available connections |

### Worker Metrics

| Metric | Type | Labels |
|--------|------|--------|
| `pix_worker_message_processed_duration_ms` | Histogram | worker, subject, status |
| `pix_worker_batch_completed_count` | Counter | worker |
| `pix_worker_poll_interval_ms` | Gauge | worker, batch_size |

### Circuit Breaker Metrics

| Metric | Type | Labels |
|--------|------|--------|
| `pix_circuit_breaker_state_change` | Counter | name, from, to |
| `pix_circuit_breaker_request` | Counter | name, result |

## Monitoring Dashboard

The admin portal includes a 6-panel real-time dashboard:

1. **TPS Chart**: Line chart with sparkline showing transactions per second
2. **Status Distribution**: Bar chart of transaction statuses
3. **System Health**: Service status indicators (green/yellow/red)
4. **Settlement**: Current cycle progress
5. **Queue Depth**: NATS consumer lag per stream
6. **BACEN Channels**: CPM/CSM health status

## Alerting

### Recommended Alert Rules

| Alert | Condition | Severity |
|-------|-----------|----------|
| Service Down | Health check fails 3x | Critical |
| High Latency | p99 > 2000ms | High |
| Queue Backlog | Consumer lag > 10,000 | High |
| Circuit Open | Circuit breaker opens | High |
| DB Pool Exhausted | Available connections = 0 | Critical |
| BACEN Channel Down | CPM or CSM down | Critical |
| Rate Limit Spike | > 100 429s per minute | Medium |

## Expected Outcome

After configuring monitoring:

- WebSocket dashboard showing real-time data across 6 channels
- Prometheus scraping metrics at configured intervals
- Health checks running every 10-30 seconds
- Alerting rules triggering on critical conditions