# 可观测性

Monetarie PIX 的全面可观测性设置，包括指标、日志、追踪和告警。

## 前提条件

- 已部署 Prometheus 和 Grafana（或云等价物）
- 了解 Telemetry 和指标类型
- 访问日志聚合系统

## 三大支柱

```mermaid
graph TB
    APP[Monetarie PIX] --> METRICS[指标<br/>Prometheus /metrics]
    APP --> LOGS[日志<br/>结构化 JSON]
    APP --> TRACES[追踪<br/>x-trace-id 头]

    METRICS --> GRAFANA[Grafana 仪表板]
    LOGS --> LOKI[Loki / Elasticsearch]
    TRACES --> CORR[日志关联]
```

## 指标（Prometheus）

### 端点

`GET /metrics`（端口 4003）暴露所有基于 Telemetry 的指标。

### 可用指标

| 指标 | 类型 | 标签 | 描述 |
|------|------|------|------|
| `phoenix_http_request_duration_ms` | Histogram | method, route, status | HTTP 请求延迟 |
| `phoenix_http_request_count` | Counter | method, route, status | 请求计数 |
| `ecto_query_duration_ms` | Histogram | source | 数据库查询时间 |
| `ecto_pool_size` | Gauge | -- | 连接池大小 |
| `pix_worker_message_processed_duration_ms` | Histogram | worker, subject, status | Worker 处理时间 |
| `pix_worker_batch_completed_count` | Counter | worker | 已完成批次 |
| `pix_worker_poll_interval_ms` | Gauge | worker | 当前轮询间隔 |
| `pix_circuit_breaker_state_change` | Counter | name, from, to | 断路器状态转换 |
| `pix_circuit_breaker_request` | Counter | name, result | 请求结果 |

### Prometheus 配置

```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'monetarie-pix'
    scrape_interval: 15s
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [pix]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: pix-backend
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        target_label: __address__
        replacement: ${1}:4003
```

### Grafana 仪表板面板

推荐仪表板布局：

1. **TPS**：`rate(phoenix_http_request_count[1m])`
2. **延迟 p95/p99**：`histogram_quantile(0.99, phoenix_http_request_duration_ms_bucket)`
3. **错误率**：`rate(phoenix_http_request_count{status=~"5.."}[5m])`
4. **DB 连接池利用率**：`ecto_pool_size - ecto_pool_available`
5. **Worker 吞吐量**：`rate(pix_worker_batch_completed_count[5m])`
6. **断路器状态**：`pix_circuit_breaker_state_change`

## 日志

### 结构化日志

所有后端组件使用 Elixir Logger 和结构化元数据：

```elixir
Logger.info("Transaction processed",
  trace_id: trace_id,
  request_id: request_id,
  transaction_id: tx_id,
  status: "STLD",
  duration_ms: duration
)
```

### 日志级别

| 级别 | 用途 |
|------|------|
| `error` | 失败操作、ACK/NAK 失败、不可恢复错误 |
| `warning` | 降级状态、静默 rescue 块、审计事件 |
| `info` | 交易生命周期、Worker 启停、健康状态转换 |
| `debug` | 详细处理步骤（生产环境中禁用） |

### 日志聚合（Kubernetes）

```yaml
# Fluentd/Fluent Bit DaemonSet 配置
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [INPUT]
        Name              tail
        Path              /var/log/containers/pix-backend-*.log
        Parser            json
    [OUTPUT]
        Name              loki
        Match             *
        Host              loki.monitoring
        Port              3100
```

## 分布式追踪

### W3C 风格追踪 ID

`Shared.Plugs.TraceContext` 为每个请求生成 16 字节十六进制 `x-trace-id`：

1. HTTP 请求到达 -> TraceContext 插件生成/提取 `x-trace-id`
2. 设置 Logger.metadata 中的 `trace_id`
3. 内部 HTTP 调用传播 `x-trace-id` 头
4. NATS 消息在消息头中携带 `trace_id`
5. 所有日志行包含 `trace_id` 用于关联

### 追踪流程

```mermaid
sequenceDiagram
    Client->>+Settlement: POST /payment (x-trace-id: abc123)
    Settlement->>+Dict: GET /entries/key (x-trace-id: abc123)
    Dict-->>-Settlement: Key data
    Settlement->>+NATS: Publish (header: trace_id=abc123)
    Settlement-->>-Client: 200 OK
    NATS->>+Worker: Consume (header: trace_id=abc123)
    Worker->>Worker: Logger.metadata(trace_id: abc123)
```

## 告警

### 推荐规则

```yaml
# Prometheus 告警规则
groups:
  - name: monetarie-pix
    rules:
      - alert: HighErrorRate
        expr: rate(phoenix_http_request_count{status=~"5.."}[5m]) > 0.05
        for: 2m
        labels:
          severity: critical

      - alert: CircuitBreakerOpen
        expr: pix_circuit_breaker_state_change{to="open"} > 0
        for: 0m
        labels:
          severity: high

      - alert: HighLatency
        expr: histogram_quantile(0.99, phoenix_http_request_duration_ms_bucket) > 2000
        for: 5m
        labels:
          severity: high

      - alert: DBPoolExhausted
        expr: ecto_pool_available == 0
        for: 1m
        labels:
          severity: critical
```

## 监控网络策略

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring
  namespace: pix
spec:
  podSelector:
    matchLabels:
      app: pix-backend
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: monitoring
      ports:
        - port: 4003
```

## 预期结果

配置可观测性后：

- Prometheus 每 15 秒抓取指标
- Grafana 仪表板显示 TPS、延迟、错误率和 Worker 吞吐量
- 结构化日志带追踪 ID 实现端到端关联
- 告警规则在关键条件下触发
- 网络策略允许监控流量