# Observability

Comprehensive observability setup including metrics, logging, tracing, and alerting for Monetarie PIX.

## Prerequisites

- Prometheus and Grafana deployed (or cloud equivalents)
- Understanding of Telemetry and metric types
- Access to log aggregation system

## Three Pillars

```mermaid
graph TB
    APP[Monetarie PIX] --> METRICS[Metrics<br/>Prometheus /metrics]
    APP --> LOGS[Logs<br/>Structured JSON]
    APP --> TRACES[Traces<br/>x-trace-id headers]

    METRICS --> GRAFANA[Grafana Dashboards]
    LOGS --> LOKI[Loki / Elasticsearch]
    TRACES --> CORR[Log Correlation]
```

## Metrics (Prometheus)

### Endpoint

`GET /metrics` on port 4003 exposes all Telemetry-based metrics.

### Available Metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `phoenix_http_request_duration_ms` | Histogram | method, route, status | HTTP request latency |
| `phoenix_http_request_count` | Counter | method, route, status | Request count |
| `ecto_query_duration_ms` | Histogram | source | Database query time |
| `ecto_pool_size` | Gauge | -- | Connection pool size |
| `pix_worker_message_processed_duration_ms` | Histogram | worker, subject, status | Worker processing time |
| `pix_worker_batch_completed_count` | Counter | worker | Completed batches |
| `pix_worker_poll_interval_ms` | Gauge | worker | Current poll interval |
| `pix_circuit_breaker_state_change` | Counter | name, from, to | Circuit state transitions |
| `pix_circuit_breaker_request` | Counter | name, result | Request outcomes |

### Prometheus Configuration

```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'monetarie-pix'
    scrape_interval: 15s
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [pix]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: pix-backend
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        target_label: __address__
        replacement: ${1}:4003
```

### Grafana Dashboard Panels

Recommended dashboard layout:

1. **TPS**: `rate(phoenix_http_request_count[1m])`
2. **Latency p95/p99**: `histogram_quantile(0.99, phoenix_http_request_duration_ms_bucket)`
3. **Error Rate**: `rate(phoenix_http_request_count{status=~"5.."}[5m])`
4. **DB Pool Utilization**: `ecto_pool_size - ecto_pool_available`
5. **Worker Throughput**: `rate(pix_worker_batch_completed_count[5m])`
6. **Circuit Breaker State**: `pix_circuit_breaker_state_change`

## Logging

### Structured Logging

All backend components use Elixir Logger with structured metadata:

```elixir
Logger.info("Transaction processed",
  trace_id: trace_id,
  request_id: request_id,
  transaction_id: tx_id,
  status: "STLD",
  duration_ms: duration
)
```

### Log Levels

| Level | Usage |
|-------|-------|
| `error` | Failed operations, ACK/NAK failures, unrecoverable errors |
| `warning` | Degraded states, silent rescue blocks, audit events |
| `info` | Transaction lifecycle, worker start/stop, health transitions |
| `debug` | Detailed processing steps (disabled in production) |

### Log Aggregation (Kubernetes)

```yaml
# Fluentd/Fluent Bit DaemonSet configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [INPUT]
        Name              tail
        Path              /var/log/containers/pix-backend-*.log
        Parser            json
    [OUTPUT]
        Name              loki
        Match             *
        Host              loki.monitoring
        Port              3100
```

## Distributed Tracing

### W3C-Style Trace IDs

`Shared.Plugs.TraceContext` generates a 16-byte hex `x-trace-id` on every request:

1. HTTP request arrives -> TraceContext plug generates/extracts `x-trace-id`
2. Logger.metadata set with `trace_id`
3. Internal HTTP calls propagate `x-trace-id` header
4. NATS messages carry `trace_id` in message headers
5. All log lines include `trace_id` for correlation

### Tracing Flow

```mermaid
sequenceDiagram
    Client->>+Settlement: POST /payment (x-trace-id: abc123)
    Settlement->>+Dict: GET /entries/key (x-trace-id: abc123)
    Dict-->>-Settlement: Key data
    Settlement->>+NATS: Publish (header: trace_id=abc123)
    Settlement-->>-Client: 200 OK
    NATS->>+Worker: Consume (header: trace_id=abc123)
    Worker->>Worker: Logger.metadata(trace_id: abc123)
```

## Alerting

### Recommended Rules

```yaml
# Prometheus alerting rules
groups:
  - name: monetarie-pix
    rules:
      - alert: HighErrorRate
        expr: rate(phoenix_http_request_count{status=~"5.."}[5m]) > 0.05
        for: 2m
        labels:
          severity: critical

      - alert: CircuitBreakerOpen
        expr: pix_circuit_breaker_state_change{to="open"} > 0
        for: 0m
        labels:
          severity: high

      - alert: HighLatency
        expr: histogram_quantile(0.99, phoenix_http_request_duration_ms_bucket) > 2000
        for: 5m
        labels:
          severity: high

      - alert: DBPoolExhausted
        expr: ecto_pool_available == 0
        for: 1m
        labels:
          severity: critical
```

## NetworkPolicy for Monitoring

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring
  namespace: pix
spec:
  podSelector:
    matchLabels:
      app: pix-backend
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: monitoring
      ports:
        - port: 4003
```

## Expected Outcome

After configuring observability:

- Prometheus scraping metrics every 15 seconds
- Grafana dashboards showing TPS, latency, error rates, and worker throughput
- Structured logs with trace IDs for end-to-end correlation
- Alert rules firing on critical conditions
- NetworkPolicy allowing monitoring traffic