# Observabilidade

Stack de observabilidade da plataforma Monetarie PIX: logging estruturado, metricas Prometheus, tracing distribuido e recomendacoes de alertas.

## Pre-requisitos

- Prometheus configurado e coletando metricas do cluster
- Grafana instalado para visualizacao de dashboards
- Acesso aos logs dos pods via `kubectl logs`
- Entendimento basico de Telemetry (Elixir) e Prometheus

## Visao Geral da Stack

```mermaid
graph TB
    subgraph Aplicacao Monetarie PIX
        APP[Backend Elixir<br/>4 apps umbrella]
        TELEMETRY[Shared.Telemetry<br/>Metricas + Poller]
        TRACE[TraceContext Plug<br/>x-trace-id]
        LOGGER[Logger<br/>JSON Estruturado]
    end

    subgraph Coleta
        PROM[Prometheus<br/>Scrape /metrics]
        LOKI[Loki / Cloud Logging<br/>Agregacao de Logs]
    end

    subgraph Visualizacao
        GRAFANA[Grafana<br/>Dashboards]
        ALERT[Alertmanager<br/>Alertas]
    end

    subgraph Infraestrutura
        NATS_MON[NATS Monitoring<br/>:8222/varz]
        PG_EXP[PostgreSQL<br/>pg_stat_statements]
        REDIS_MON[Redis<br/>INFO command]
    end

    APP --> TELEMETRY
    APP --> TRACE
    APP --> LOGGER

    TELEMETRY -->|/metrics| PROM
    LOGGER -->|stdout/stderr| LOKI
    NATS_MON --> PROM
    PG_EXP --> PROM
    REDIS_MON --> PROM

    PROM --> GRAFANA
    LOKI --> GRAFANA
    PROM --> ALERT
```

## Logging Estruturado

### Formato dos Logs

Todos os logs sao emitidos em formato JSON estruturado para facilitar parsing e busca:

```json
{
  "time": "2026-02-13T14:30:00.123Z",
  "level": "info",
  "msg": "[InboundProcessor] Heartbeat: processed=4521 failed=3 retried=12 uptime=3600s",
  "module": "SpiService.Workers.InboundProcessor",
  "request_id": "F8xK2mN9pL",
  "trace_id": "a1b2c3d4e5f6a7b8"
}
```

### Correlacao via x-trace-id

O plug `Shared.Plugs.TraceContext` gera um trace ID unico (16 bytes hex) para cada requisicao HTTP e o propaga:

```mermaid
sequenceDiagram
    participant C as Cliente
    participant GW as Settlement (4003)
    participant DICT as Dict (4001)
    participant SPI as SPI (4002)
    participant NATS as NATS JetStream

    C->>GW: POST /api/v1/payments<br/>x-trace-id: (gerado se ausente)
    Note over GW: TraceContext plug<br/>Logger.metadata(trace_id: "a1b2...")
    GW->>DICT: GET /api/v2/entries/:key<br/>x-trace-id: a1b2...
    Note over DICT: TraceContext plug<br/>Logger.metadata(trace_id: "a1b2...")
    GW->>NATS: monetarie.spi.transaction.created<br/>header: x-trace-id: a1b2...
    NATS->>SPI: InboundProcessor<br/>set_trace_id(headers["x-trace-id"])
    Note over SPI: Logger.metadata(trace_id: "a1b2...")
```

O trace ID e incluido em:

| Componente | Mecanismo | Onde |
|-----------|-----------|------|
| HTTP requisicao | Header `x-trace-id` | Todas as 3 pipelines de servico |
| HTTP entre servicos | Header `x-trace-id` via InternalClient | Settlement → Dict, Settlement → SPI |
| NATS mensagens | Header `x-trace-id` | JetStream publish |
| Workers NATS | `Logger.metadata(trace_id: ...)` | BaseWorker.set_trace_id |
| Logs | Campo `trace_id` no JSON | Todos os log entries |

### Niveis de Log por Ambiente

| Nivel | Producao | Staging | Desenvolvimento |
|-------|----------|---------|-----------------|
| error | Sim | Sim | Sim |
| warning | Sim | Sim | Sim |
| info | Nao | Sim | Sim |
| debug | Nao | Nao | Sim |

Configuracao via `config/runtime.exs`:

```elixir
config :logger, level: String.to_atom(System.get_env("LOG_LEVEL", "info"))
```

### Buscar Logs por Trace ID

```bash
# Buscar todos os logs de uma transacao especifica
kubectl logs deployment/pix-backend -n pix | grep "a1b2c3d4e5f6a7b8"

# Buscar logs de erro da ultima hora
kubectl logs deployment/pix-backend -n pix --since=1h | grep '"level":"error"'

# Buscar logs de um worker especifico
kubectl logs deployment/pix-backend -n pix --tail=100 | grep "InboundProcessor"

# Buscar logs de circuit breaker
kubectl logs deployment/pix-backend -n pix | grep "CircuitBreaker"
```

## Metricas Prometheus

### Endpoint de Metricas

O endpoint `/metrics` esta disponivel no Settlement Service (porta 4003):

```bash
# Acessar metricas
curl -s https://pixapi-dev.fluxiq.com.br/metrics | head -30

# Saida esperada:
# # HELP phoenix_endpoint_stop_duration_milliseconds Duration of HTTP requests
# # TYPE phoenix_endpoint_stop_duration_milliseconds summary
# phoenix_endpoint_stop_duration_milliseconds_sum 45231.5
# phoenix_endpoint_stop_duration_milliseconds_count 12450
# ...
```

### Catalogo de Metricas

#### Metricas Phoenix (HTTP)

| Metrica | Tipo | Tags | Descricao |
|---------|------|------|-----------|
| `phoenix.endpoint.stop.duration` | Summary | - | Duracao total das requisicoes HTTP |
| `phoenix.router_dispatch.stop.duration` | Summary | route | Duracao por rota |

#### Metricas Ecto (Banco de Dados)

| Metrica | Tipo | Tags | Descricao |
|---------|------|------|-----------|
| `shared.repo.query.total_time` | Summary | - | Tempo total de execucao de queries |
| `shared.repo.query.query_time` | Summary | - | Tempo de execucao da query no DB |
| `shared.repo.query.queue_time` | Summary | - | Tempo de espera por conexao do pool |
| `shared.repo.query.decode_time` | Summary | - | Tempo de decodificacao do resultado |

#### Metricas de Workers (BaseWorker)

| Metrica | Tipo | Tags | Descricao |
|---------|------|------|-----------|
| `pix.worker.message_processed.count` | Counter | worker, status | Mensagens processadas (ok/error/retry) |
| `pix.worker.message_processed.duration` | Summary | worker | Tempo de processamento por mensagem |
| `pix.worker.batch_completed.duration` | Summary | worker | Tempo de processamento por batch |
| `pix.worker.batch_completed.size` | Summary | worker | Numero de mensagens por batch |
| `pix.worker.poll.duration` | Summary | worker | Duracao de cada ciclo de polling |
| `pix.worker.poll.queue_depth` | Summary | worker | Profundidade da fila por poll |

#### Metricas de Circuit Breaker

| Metrica | Tipo | Tags | Descricao |
|---------|------|------|-----------|
| `pix.circuit_breaker.state_change.count` | Counter | service, from_state, to_state | Transicoes de estado |
| `pix.circuit_breaker.request.count` | Counter | service, result | Requisicoes (ok/error/rejected) |

#### Metricas ANS (SLA)

| Metrica | Tipo | Tags | Descricao |
|---------|------|------|-----------|
| `pix.ans.settlement.duration_ms` | Summary | e2e_id | Tempo de liquidacao em ms |
| `pix.ans.breach.duration_ms` | Counter | e2e_id | Violacoes ANS (>1600ms) |

#### Metricas VM (BEAM)

| Metrica | Tipo | Tags | Descricao |
|---------|------|------|-----------|
| `vm.memory.total` | Summary | - | Memoria total da VM (bytes) |
| `vm.total_run_queue_lengths.total` | Summary | - | Run queues totais |
| `vm.total_run_queue_lengths.cpu` | Summary | - | Run queues de CPU |
| `vm.total_run_queue_lengths.io` | Summary | - | Run queues de I/O |

### Configuracao Prometheus

ServiceMonitor para Kubernetes:

```yaml
# deploy/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pix-backend-metrics
  namespace: pix
  labels:
    app: pix-backend
spec:
  selector:
    matchLabels:
      app: pix-backend
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
      scrapeTimeout: 10s
```

Aplicar:

```bash
kubectl apply -f deploy/servicemonitor.yaml

# Verificar se o target aparece no Prometheus
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.app=="pix-backend")'
```

### Configuracao Grafana

#### Dashboard de Workers

Painel de throughput dos workers:

```
# PromQL: Taxa de mensagens processadas por worker
rate(pix_worker_message_processed_count[5m])

# PromQL: Taxa de erros por worker
rate(pix_worker_message_processed_count{status="error"}[5m])

# PromQL: Latencia P95 de processamento
histogram_quantile(0.95, rate(pix_worker_message_processed_duration_bucket[5m]))

# PromQL: Tamanho medio de batch
avg(pix_worker_batch_completed_size)
```

#### Dashboard de Circuit Breaker

```
# PromQL: Transicoes de estado
rate(pix_circuit_breaker_state_change_count[5m])

# PromQL: Taxa de rejeicao (circuit open)
rate(pix_circuit_breaker_request_count{result="rejected"}[5m])

# PromQL: Proporcao de erros vs total
rate(pix_circuit_breaker_request_count{result="error"}[5m])
/
rate(pix_circuit_breaker_request_count[5m])
```

#### Dashboard de ANS

```
# PromQL: Tempo medio de liquidacao
avg(pix_ans_settlement_duration_ms)

# PromQL: Violacoes ANS
rate(pix_ans_breach_duration_ms_count[5m])

# PromQL: P99 do tempo de liquidacao
histogram_quantile(0.99, rate(pix_ans_settlement_duration_ms_bucket[5m]))
```

#### Dashboard de Banco de Dados

```
# PromQL: Tempo de espera por conexao (pool queue time)
avg(shared_repo_query_queue_time_milliseconds)

# PromQL: Tempo medio de query
avg(shared_repo_query_query_time_milliseconds)
```

## Tracing Distribuido

### Plug TraceContext

O plug `Shared.Plugs.TraceContext` esta conectado em todas as 3 pipelines de servico:

| Servico | Pipeline | Arquivo |
|---------|----------|---------|
| Settlement (4003) | `gateway` | `settlement_service_web/router.ex` |
| Dict (4001) | `api` | `dict_service_web/router.ex` |
| SPI (4002) | `api` | `spi_service_web/router.ex` |

Funcionamento:

1. Se o header `x-trace-id` existe na requisicao, ele e reutilizado
2. Se nao existe, um novo trace ID de 16 bytes hex e gerado
3. O trace ID e adicionado a `Logger.metadata` para correlacao de logs
4. O trace ID e propagado em chamadas HTTP entre servicos (InternalClient)
5. O trace ID e injetado como header NATS em publicacoes JetStream

### Rastreamento End-to-End

Exemplo de rastreamento de uma transacao PIX do inicio ao fim:

```bash
# 1. Identificar o trace_id da transacao
kubectl logs deployment/pix-backend -n pix | grep "end_to_end_id=E123" | head -1
# Saida: trace_id=a1b2c3d4e5f6a7b8

# 2. Seguir o trace_id por todos os servicos
kubectl logs deployment/pix-backend -n pix | grep "a1b2c3d4e5f6a7b8"

# Saida esperada (fluxo completo):
# [Settlement] POST /api/v1/payments trace_id=a1b2c3d4e5f6a7b8
# [Settlement] PaymentController.create: inserting transaction
# [Settlement] NATS publish: monetarie.spi.transaction.created
# [SPI] InboundProcessor: received message trace_id=a1b2c3d4e5f6a7b8
# [SPI] Processing pacs.008 for E123
# [SPI] OutboundSender: sending to BACEN
# [SPI] StatusUpdater: PDNG -> ACSP trace_id=a1b2c3d4e5f6a7b8
# [SPI] StatusUpdater: ACSP -> STLD trace_id=a1b2c3d4e5f6a7b8
```

## Health Checks

### Endpoint /health

O HealthChecker GenServer executa verificacoes a cada 10 segundos:

```bash
curl -s https://pixapi-dev.fluxiq.com.br/health | jq .
```

Resposta:

```json
{
  "overall": "healthy",
  "services": [
    {"name": "dict-service", "status": "healthy", "responseTimeMs": 12},
    {"name": "spi-service", "status": "healthy", "responseTimeMs": 8},
    {"name": "settlement-service", "status": "healthy", "responseTimeMs": 3}
  ],
  "database": {"status": "healthy", "responseTimeMs": 2},
  "redis": {"status": "healthy", "responseTimeMs": 1},
  "nats": {"status": "healthy", "responseTimeMs": 0},
  "lastUpdatedAt": "2026-02-13T14:30:00.123Z"
}
```

Estados possiveis:

| Estado | Significado | Condicao |
|--------|------------|----------|
| `healthy` | Tudo operacional | Todos os servicos e infra saudaveis |
| `degraded` | Funcionalidade reduzida | Redis ou NATS com problemas, ou servico degradado |
| `unhealthy` | Falha critica | Banco de dados ou servico principal fora |

### BACEN Channel Check

A cada 30 segundos, verifica a saude dos canais BACEN (CPM/CSM):

```bash
curl -s https://pixapi-dev.fluxiq.com.br/api/v1/monitoring/health | jq .bacen_channels
```

Resposta:

```json
{
  "primary": {
    "status": "healthy",
    "consecutiveFailures": 0,
    "totalRequests": 4521,
    "lastSuccess": "2026-02-13T14:29:55.000Z"
  },
  "secondary": {
    "status": "healthy",
    "consecutiveFailures": 0,
    "totalRequests": 890,
    "lastSuccess": "2026-02-13T14:28:30.000Z"
  }
}
```

## Recomendacoes de Alertas

### Alertas Criticos (Pager)

| Alerta | Condicao | Severidade | Acao |
|--------|----------|-----------|------|
| Circuit breaker aberto | `pix_circuit_breaker_state_change_count{to_state="open"} > 0` | Critico | Verificar servico downstream |
| Banco de dados inacessivel | Health check database = "unhealthy" por 1 min | Critico | Verificar PostgreSQL |
| Error rate > 5% | `rate(phoenix_endpoint_stop_duration_count{status=~"5.."}[5m]) / rate(phoenix_endpoint_stop_duration_count[5m]) > 0.05` | Critico | Investigar logs de erro |
| Pod CrashLoopBackOff | Pod restarts > 3 em 5 min | Critico | Ver troubleshooting #1 |

### Alertas de Aviso (Email/Slack)

| Alerta | Condicao | Severidade | Acao |
|--------|----------|-----------|------|
| NATS consumer lag > 1000 | Pending messages > 1000 por 5 min | Aviso | Escalar workers |
| Error rate > 1% | Error rate > 1% por 5 min | Aviso | Investigar causa |
| P99 latencia > 2s | `histogram_quantile(0.99, ...) > 2000` | Aviso | Otimizar queries/infra |
| Memoria > 80% | `vm.memory.total > 0.8 * limit` | Aviso | Verificar leaks |
| DLQ recebendo mensagens | `rate(pix_worker_message_processed_count{status="error"}[5m]) > 0` | Aviso | Investigar falhas |
| ANS violado | `pix_ans_breach_duration_ms_count` incrementando | Aviso | Ver troubleshooting #10 |
| Pool queue time > 100ms | `avg(shared_repo_query_queue_time) > 100` | Aviso | Aumentar POOL_SIZE |
| Redis degradado | Health check redis != "healthy" por 2 min | Aviso | Verificar Redis |

### Exemplo de Regra Prometheus (AlertManager)

```yaml
# alerting-rules.yaml
groups:
  - name: monetarie-pix
    rules:
      - alert: CircuitBreakerOpen
        expr: increase(pix_circuit_breaker_state_change_count{to_state="open"}[5m]) > 0
        for: 0m
        labels:
          severity: critical
          team: pix
        annotations:
          summary: "Circuit breaker aberto para {{ $labels.service }}"
          description: "O circuit breaker para o servico {{ $labels.service }} foi aberto apos {{ $labels.failure_count }} falhas."

      - alert: NATSConsumerLagHigh
        expr: pix_worker_poll_queue_depth > 1000
        for: 5m
        labels:
          severity: warning
          team: pix
        annotations:
          summary: "NATS consumer lag alto no worker {{ $labels.worker }}"
          description: "O worker {{ $labels.worker }} tem {{ $value }} mensagens pendentes por mais de 5 minutos."

      - alert: HighErrorRate
        expr: |
          rate(phoenix_endpoint_stop_duration_count{status=~"5.."}[5m])
          /
          rate(phoenix_endpoint_stop_duration_count[5m]) > 0.01
        for: 5m
        labels:
          severity: warning
          team: pix
        annotations:
          summary: "Taxa de erros acima de 1%"
          description: "A taxa de erros HTTP 5xx esta em {{ $value | humanizePercentage }}."

      - alert: ANSBreached
        expr: increase(pix_ans_breach_duration_ms_count[5m]) > 0
        for: 0m
        labels:
          severity: warning
          team: pix
        annotations:
          summary: "Violacao de ANS detectada"
          description: "Transacao {{ $labels.e2e_id }} excedeu o limite de 1600ms para liquidacao."
```

## WebSocket Monitoring (Real-Time)

O HealthChecker e o QueueMonitor transmitem dados em tempo real via Phoenix Channels:

| Canal | Topico | Frequencia | Dados |
|-------|--------|-----------|-------|
| `transactions:live` | Transacoes em tempo real | Por evento | Status, valor, E2E ID |
| `system:health` | Saude do sistema | 10s | Overall, servicos, DB, Redis, NATS |
| `settlement:status` | Liquidacao | Por evento | Ciclo, status, totais |
| `dict:operations` | Operacoes DICT | Por evento | Chaves criadas/removidas |
| `bacen:channels` | Canais BACEN | 30s | CPM/CSM status |
| `queues:depth` | Profundidade de filas | 15s | Pending messages por consumer |

Conectar via JavaScript:

```javascript
import { Socket } from "phoenix";

const socket = new Socket("/socket", {
  params: { token: "jwt-token-aqui" }
});

socket.connect();

const healthChannel = socket.channel("system:health", {});
healthChannel.join()
  .receive("ok", () => console.log("Conectado ao canal de saude"))
  .receive("error", (err) => console.error("Erro:", err));

healthChannel.on("health_update", (payload) => {
  console.log("Status:", payload.overall);
  console.log("Servicos:", payload.services);
});
```

## Resultado Esperado

Ao implementar este guia de observabilidade, voce tera:

- Logs estruturados em JSON com correlacao via x-trace-id em todos os componentes
- Metricas Prometheus cobrindo HTTP, banco de dados, workers, circuit breaker e VM
- Tracing distribuido end-to-end (HTTP + NATS) com IDs de 16 bytes hex
- Health checks automaticos a cada 10s (servicos) e 30s (BACEN)
- Dashboards Grafana para workers, circuit breaker, ANS e banco de dados
- Alertas configurados para condicoes criticas e de aviso
- Monitoramento real-time via WebSocket com 6 canais de dados