# Deployment On-Premises

Guia para deploy do Monetarie PIX em infraestrutura propria utilizando ProxMox para virtualizacao, OPNsense para firewall/routing, HAProxy para load balancing, PostgreSQL com Patroni para alta disponibilidade, Redis Sentinel e cluster NATS.

## Pre-requisitos

- Servidor(es) fisico(s) ou hipervisor (ProxMox, VMware, Hyper-V)
- Rede local com segmentacao (VLANs recomendadas)
- Acesso a internet para download de imagens Docker e pacotes
- Dominio configurado com DNS apontando para o IP externo
- Docker Engine 24.0+ instalado nos nodes de aplicacao

## Arquitetura On-Premises

```mermaid
graph TB
    subgraph Internet
        CLIENTS[Clientes / Admin Portal]
    end

    subgraph DMZ["DMZ (VLAN 10 — 10.10.10.0/24)"]
        OPNSENSE[OPNsense<br/>Firewall + NAT]
        HAPROXY[HAProxy<br/>Load Balancer + TLS]
    end

    subgraph AppLayer["Aplicacao (VLAN 20 — 10.10.20.0/24)"]
        APP1[App Node 1<br/>Docker]
        APP2[App Node 2<br/>Docker]
        APP3[App Node 3<br/>Docker]
    end

    subgraph DataLayer["Dados (VLAN 30 — 10.10.30.0/24)"]
        PG1[(PostgreSQL Primary<br/>Patroni)]
        PG2[(PostgreSQL Replica<br/>Patroni)]
        PG3[(PostgreSQL Replica<br/>Patroni)]
        ETCD[etcd cluster<br/>3 nos]
    end

    subgraph CacheLayer["Cache (VLAN 40 — 10.10.40.0/24)"]
        REDIS1[Redis Master<br/>Sentinel]
        REDIS2[Redis Replica<br/>Sentinel]
        REDIS3[Redis Replica<br/>Sentinel]
    end

    subgraph MsgLayer["Mensageria (VLAN 50 — 10.10.50.0/24)"]
        NATS1[NATS Node 1<br/>JetStream]
        NATS2[NATS Node 2<br/>JetStream]
        NATS3[NATS Node 3<br/>JetStream]
    end

    CLIENTS -->|HTTPS :443| OPNSENSE
    OPNSENSE -->|:443| HAPROXY
    HAPROXY -->|:4003| APP1
    HAPROXY -->|:4003| APP2
    HAPROXY -->|:4003| APP3
    HAPROXY -->|:80| APP1
    HAPROXY -->|:80| APP2

    APP1 --> PG1
    APP2 --> PG1
    APP3 --> PG1
    APP1 --> REDIS1
    APP1 --> NATS1
    APP2 --> NATS2
    APP3 --> NATS3

    PG1 <--> PG2
    PG1 <--> PG3
    PG1 --- ETCD
    REDIS1 <--> REDIS2
    REDIS1 <--> REDIS3
    NATS1 <--> NATS2
    NATS1 <--> NATS3
```

## Especificacoes de VMs

### Producao (5.000 TPS)

| Funcao | Quantidade | vCPU | RAM | Disco | VLAN |
|--------|-----------|------|-----|-------|------|
| OPNsense (Firewall) | 1 | 2 | 4 GB | 20 GB SSD | 10 |
| HAProxy (Load Balancer) | 1-2 | 4 | 8 GB | 20 GB SSD | 10 |
| App Node (Docker) | 3 | 8 | 16 GB | 50 GB SSD | 20 |
| PostgreSQL (Patroni) | 3 | 8 | 32 GB | 500 GB NVMe | 30 |
| etcd | 3 | 2 | 4 GB | 20 GB SSD | 30 |
| Redis (Sentinel) | 3 | 4 | 8 GB | 20 GB SSD | 40 |
| NATS (JetStream) | 3 | 4 | 8 GB | 100 GB SSD | 50 |
| **Total** | **19-20** | **98-102** | **220-228 GB** | **~2.7 TB** | |

### Single-Node (Desenvolvimento / Teste)

Para desenvolvimento ou POC, todos os componentes em uma unica VM:

| Componente | vCPU | RAM | Disco |
|------------|------|-----|-------|
| Tudo-em-um | 8 | 32 GB | 100 GB SSD |

Use o Docker Compose modo desenvolvimento (ver [Docker Compose](./docker-compose.md)).

## Rede

### Segmentacao de VLANs

| VLAN | Rede | Descricao | Acesso Permitido |
|------|------|-----------|-----------------|
| 10 | 10.10.10.0/24 | DMZ (Firewall + LB) | Internet <--> VLAN 20 |
| 20 | 10.10.20.0/24 | Aplicacao (Docker) | VLAN 10, 30, 40, 50 |
| 30 | 10.10.30.0/24 | Dados (PostgreSQL) | VLAN 20 apenas |
| 40 | 10.10.40.0/24 | Cache (Redis) | VLAN 20 apenas |
| 50 | 10.10.50.0/24 | Mensageria (NATS) | VLAN 20 apenas |

### Regras de Firewall (OPNsense)

| Origem | Destino | Porta | Protocolo | Descricao |
|--------|---------|-------|-----------|-----------|
| Internet | HAPROXY | 443 | TCP | HTTPS externo |
| Internet | HAPROXY | 80 | TCP | HTTP (redirect) |
| HAPROXY | VLAN 20 | 4003 | TCP | Backend API |
| HAPROXY | VLAN 20 | 80 | TCP | Admin Frontend |
| VLAN 20 | VLAN 30 | 5432 | TCP | PostgreSQL |
| VLAN 20 | VLAN 40 | 6379 | TCP | Redis |
| VLAN 20 | VLAN 50 | 4222 | TCP | NATS clientes |
| VLAN 30 | VLAN 30 | 5432, 2379, 2380 | TCP | Patroni + etcd |
| VLAN 40 | VLAN 40 | 6379, 26379 | TCP | Redis + Sentinel |
| VLAN 50 | VLAN 50 | 4222, 6222 | TCP | NATS + cluster |

## Passo 1: PostgreSQL (Patroni HA)

Instale o PostgreSQL 16 com Patroni em 3 nos para alta disponibilidade automatica.

### 1.1 Instalar PostgreSQL e Patroni

Em cada no PostgreSQL (pg1, pg2, pg3):

```bash
# Instalar PostgreSQL 16
sudo apt-get update
sudo apt-get install -y postgresql-16 postgresql-16-contrib

# Instalar Patroni
sudo apt-get install -y python3-pip python3-psycopg2
sudo pip3 install patroni[etcd3]

# Parar PostgreSQL padrao
sudo systemctl stop postgresql
sudo systemctl disable postgresql
```

### 1.2 Instalar etcd

Em cada no etcd (pode compartilhar VMs com PostgreSQL ou ser separado):

```bash
sudo apt-get install -y etcd

# /etc/default/etcd
ETCD_NAME=etcd1
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
ETCD_ADVERTISE_CLIENT_URLS=http://10.10.30.11:2379
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.10.30.11:2380
ETCD_INITIAL_CLUSTER="etcd1=http://10.10.30.11:2380,etcd2=http://10.10.30.12:2380,etcd3=http://10.10.30.13:2380"
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=monetarie-pix

sudo systemctl enable etcd
sudo systemctl start etcd
```

### 1.3 Configurar Patroni

Crie `/etc/patroni/patroni.yml` em cada no (ajuste nome e IP):

```yaml
scope: monetarie-pix
name: pg1

restapi:
  listen: 0.0.0.0:8008
  connect_address: 10.10.30.11:8008

etcd3:
  hosts: 10.10.30.11:2379,10.10.30.12:2379,10.10.30.13:2379

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    postgresql:
      use_pg_rewind: true
      parameters:
        max_connections: 6000
        shared_buffers: 8GB
        effective_cache_size: 24GB
        wal_level: replica
        max_wal_senders: 5
        max_replication_slots: 5
        hot_standby: "on"

  initdb:
    - encoding: UTF8
    - data-checksums

  pg_hba:
    - host replication replicator 10.10.30.0/24 md5
    - host all all 10.10.20.0/24 md5

  users:
    postgres:
      password: "<SENHA_POSTGRES>"
      options:
        - createrole
        - createdb
    replicator:
      password: "<SENHA_REPLICATOR>"
      options:
        - replication

postgresql:
  listen: 0.0.0.0:5432
  connect_address: 10.10.30.11:5432
  data_dir: /var/lib/postgresql/16/main
  bin_dir: /usr/lib/postgresql/16/bin
  authentication:
    replication:
      username: replicator
      password: "<SENHA_REPLICATOR>"
    superuser:
      username: postgres
      password: "<SENHA_POSTGRES>"
```

Iniciar Patroni:

```bash
sudo systemctl enable patroni
sudo systemctl start patroni

# Verificar cluster
patronictl -c /etc/patroni/patroni.yml list
```

Saida esperada:
```
+ Cluster: monetarie-pix --------+---------+---------+----+-----------+
| Member | Host           | Role    | State   | TL | Lag in MB |
+--------+----------------+---------+---------+----+-----------+
| pg1    | 10.10.30.11    | Leader  | running |  1 |           |
| pg2    | 10.10.30.12    | Replica | running |  1 |       0.0 |
| pg3    | 10.10.30.13    | Replica | running |  1 |       0.0 |
+--------+----------------+---------+---------+----+-----------+
```

### 1.4 Criar Banco

Conecte ao lider e crie o banco:

```bash
psql -h 10.10.30.11 -U postgres -c "CREATE DATABASE monetarie;"
```

## Passo 2: Redis Sentinel

### 2.1 Instalar Redis

Em cada no Redis (redis1, redis2, redis3):

```bash
sudo apt-get install -y redis-server
```

### 2.2 Configurar Redis Master (redis1)

```bash
# /etc/redis/redis.conf
bind 0.0.0.0
port 6379
maxmemory 4gb
maxmemory-policy allkeys-lru
appendonly yes
```

### 2.3 Configurar Redis Replicas (redis2, redis3)

```bash
# /etc/redis/redis.conf
bind 0.0.0.0
port 6379
replicaof 10.10.40.11 6379
maxmemory 4gb
maxmemory-policy allkeys-lru
appendonly yes
```

### 2.4 Configurar Sentinel

Em cada no Redis:

```bash
# /etc/redis/sentinel.conf
port 26379
sentinel monitor monetarie-pix 10.10.40.11 6379 2
sentinel down-after-milliseconds monetarie-pix 5000
sentinel failover-timeout monetarie-pix 10000
sentinel parallel-syncs monetarie-pix 1
```

Iniciar Redis e Sentinel:

```bash
sudo systemctl enable redis-server redis-sentinel
sudo systemctl start redis-server redis-sentinel

# Verificar Sentinel
redis-cli -p 26379 sentinel masters
```

Saida esperada:
```
1) "monetarie-pix"
   ...
   "num-slaves" "2"
   "num-other-sentinels" "2"
```

## Passo 3: NATS Cluster

### 3.1 Instalar NATS

Em cada no NATS (nats1, nats2, nats3):

```bash
curl -L https://github.com/nats-io/nats-server/releases/download/v2.10.22/nats-server-v2.10.22-linux-amd64.tar.gz | tar xz
sudo mv nats-server-v2.10.22-linux-amd64/nats-server /usr/local/bin/
```

### 3.2 Configurar NATS

Crie `/etc/nats/nats-server.conf` em cada no (ajuste nome e IPs):

```conf
server_name: nats1
port: 4222
monitor_port: 8222

jetstream {
  store_dir: /var/nats/jetstream
  max_mem: 4G
  max_file: 50G
}

cluster {
  name: monetarie-pix
  port: 6222
  routes: [
    nats-route://10.10.50.11:6222
    nats-route://10.10.50.12:6222
    nats-route://10.10.50.13:6222
  ]
}
```

Iniciar NATS:

```bash
sudo mkdir -p /var/nats/jetstream
sudo useradd -r -s /bin/false nats
sudo chown nats:nats /var/nats -R

# Criar systemd service
sudo tee /etc/systemd/system/nats.service << 'EOF'
[Unit]
Description=NATS Server
After=network.target

[Service]
User=nats
ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats-server.conf
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable nats
sudo systemctl start nats

# Verificar cluster
curl -s http://localhost:8222/routez | python3 -m json.tool
```

Saida esperada:
```json
{
  "num_routes": 2,
  "routes": [...]
}
```

## Passo 4: App Nodes (Docker)

### 4.1 Instalar Docker

Em cada app node (app1, app2, app3):

```bash
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
```

### 4.2 Configurar Ambiente

Em cada app node, crie o `.env`:

```bash
mkdir -p /opt/monetarie-pix
cd /opt/monetarie-pix

cat > .env << 'EOF'
# Database (Patroni leader)
DB_HOST=10.10.30.11
DB_PORT=5432
DB_USER=postgres
DB_PASS=<SENHA_POSTGRES>
DB_NAME=monetarie
DATABASE_URL=ecto://postgres:<SENHA_POSTGRES>@10.10.30.11:5432/monetarie

# Redis (Master)
REDIS_HOST=10.10.40.11
REDIS_PORT=6379

# NATS (qualquer no do cluster)
NATS_HOST=10.10.50.11
NATS_PORT=4222
NATS_ENABLED=true

# Phoenix
SECRET_KEY_BASE=<CHAVE_64_CHARS>
PHX_HOST=pixapi.seudominio.com.br
PORT=4003
MIX_ENV=prod
POOL_SIZE=200

# Auth
JWT_SECRET=<SECRET_COMPARTILHADO>
GUARDIAN_SECRET_KEY=<SECRET_COMPARTILHADO>

# BACEN
BACEN_ENABLED=false
BACEN_ENV=homolog
SIMULATOR_ENABLED=false

# Rate Limiting
RATE_LIMIT_MODE=strict
RATE_LIMIT_BYPASS_AUTH=false
EOF
```

### 4.3 Iniciar Aplicacao

```bash
# Pull imagens
docker pull monetarie/pix-backend:latest
docker pull monetarie/pix-admin:latest

# Backend
docker run -d \
  --name pix-backend \
  --restart=unless-stopped \
  --env-file .env \
  -p 4003:4003 \
  monetarie/pix-backend:latest

# Admin (apenas em app1 e app2)
docker run -d \
  --name pix-admin \
  --restart=unless-stopped \
  -p 8080:80 \
  monetarie/pix-admin:latest
```

### 4.4 Migrations (Apenas uma vez, em app1)

```bash
docker exec pix-backend bin/monetarie_pix eval "Shared.Release.migrate()"
docker exec pix-backend bin/monetarie_pix eval "Shared.Release.seed()"
```

## Passo 5: HAProxy

### 5.1 Instalar HAProxy

```bash
sudo apt-get install -y haproxy
```

### 5.2 Configurar HAProxy

```bash
# /etc/haproxy/haproxy.cfg

global
    maxconn 4096
    log /dev/log local0
    ssl-default-bind-options ssl-min-ver TLSv1.2
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384

defaults
    mode http
    log global
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000
    retries 3

# Frontend HTTPS
frontend ft_https
    bind *:443 ssl crt /etc/haproxy/certs/monetarie.pem
    bind *:80
    redirect scheme https code 301 if !{ ssl_fc }

    # Roteamento por Host
    acl is_api hdr(host) -i pixapi.seudominio.com.br
    acl is_admin hdr(host) -i pixadmin.seudominio.com.br
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_socket path_beg /socket

    use_backend bk_websocket if is_api is_websocket
    use_backend bk_websocket if is_api is_socket
    use_backend bk_api if is_api
    use_backend bk_admin if is_admin
    default_backend bk_api

# Backend API
backend bk_api
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server app1 10.10.20.11:4003 check inter 5s fall 3 rise 2
    server app2 10.10.20.12:4003 check inter 5s fall 3 rise 2
    server app3 10.10.20.13:4003 check inter 5s fall 3 rise 2

# Backend WebSocket
backend bk_websocket
    balance roundrobin
    option httpchk GET /health
    timeout server 7d
    timeout tunnel 7d
    server app1 10.10.20.11:4003 check inter 5s fall 3 rise 2
    server app2 10.10.20.12:4003 check inter 5s fall 3 rise 2
    server app3 10.10.20.13:4003 check inter 5s fall 3 rise 2

# Backend Admin
backend bk_admin
    balance roundrobin
    server app1 10.10.20.11:8080 check inter 10s fall 3 rise 2
    server app2 10.10.20.12:8080 check inter 10s fall 3 rise 2

# Stats (acesso interno apenas)
listen stats
    bind 10.10.10.10:8404
    stats enable
    stats uri /haproxy-stats
    stats auth admin:monetarie2026
```

### 5.3 Configurar Certificado TLS

Combine certificado e chave em formato PEM:

```bash
cat server.crt server.key > /etc/haproxy/certs/monetarie.pem
chmod 600 /etc/haproxy/certs/monetarie.pem
```

Iniciar HAProxy:

```bash
sudo systemctl enable haproxy
sudo systemctl start haproxy

# Verificar status
curl -s http://10.10.10.10:8404/haproxy-stats
```

## Passo 6: OPNsense (Firewall)

### 6.1 Configuracao Basica

Configure o OPNsense com as interfaces:

| Interface | VLAN | IP | Descricao |
|-----------|------|-----|-----------|
| WAN | - | DHCP/Estatico | Internet |
| LAN | 10 | 10.10.10.1/24 | DMZ |
| OPT1 | 20 | 10.10.20.1/24 | Aplicacao |
| OPT2 | 30 | 10.10.30.1/24 | Dados |
| OPT3 | 40 | 10.10.40.1/24 | Cache |
| OPT4 | 50 | 10.10.50.1/24 | Mensageria |

### 6.2 Regras de NAT

| Interface | Origem | Destino | Porta | NAT Para |
|-----------|--------|---------|-------|----------|
| WAN | Any | WAN IP | 443 | 10.10.10.10:443 (HAProxy) |
| WAN | Any | WAN IP | 80 | 10.10.10.10:80 (HAProxy) |

### 6.3 Regras de Firewall

Crie as regras conforme a tabela de regras de firewall na secao de Rede acima.

## Gerenciamento de Certificados TLS

### Opcao 1: Let's Encrypt (Recomendado)

Configure o OPNsense com o plugin ACME para renovacao automatica:

1. Instalar plugin `os-acme-client` no OPNsense
2. Configurar desafio DNS ou HTTP-01
3. Exportar certificado para HAProxy

### Opcao 2: Certificado Comercial

Obtenha certificado de uma CA confiavel e instale no HAProxy.

### Opcao 3: CA Interna

Para ambientes de teste, crie uma CA interna:

```bash
# Criar CA
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
  -out ca.crt -subj "/CN=Monetarie Internal CA"

# Criar certificado do servidor
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr \
  -subj "/CN=pixapi.seudominio.com.br" \
  -addext "subjectAltName=DNS:pixapi.seudominio.com.br,DNS:pixadmin.seudominio.com.br"

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365 -sha256
```

## Estrategia de Backup

### Backup Diario

| Componente | Metodo | Retencao | Horario |
|------------|--------|----------|---------|
| PostgreSQL | `pg_basebackup` + WAL archiving | 30 dias | 03:00 UTC |
| Redis | RDB snapshots | 7 dias | A cada 1 hora |
| NATS JetStream | File store backup | 7 dias | 04:00 UTC |
| Configuracoes | rsync para storage externo | 90 dias | 02:00 UTC |

### Script de Backup PostgreSQL

```bash
#!/bin/bash
# /opt/backup/backup-pg.sh
BACKUP_DIR="/opt/backup/postgresql"
DATE=$(date +%Y%m%d_%H%M%S)

pg_basebackup -h 10.10.30.11 -U postgres -D "$BACKUP_DIR/$DATE" \
  --format=tar --gzip --checkpoint=fast --wal-method=stream

# Manter apenas ultimos 30 backups
ls -dt $BACKUP_DIR/*/ | tail -n +31 | xargs rm -rf
```

### Script de Backup Redis

```bash
#!/bin/bash
# /opt/backup/backup-redis.sh
BACKUP_DIR="/opt/backup/redis"
DATE=$(date +%Y%m%d_%H%M%S)

redis-cli -h 10.10.40.11 BGSAVE
sleep 5
cp /var/lib/redis/dump.rdb "$BACKUP_DIR/dump_$DATE.rdb"

# Manter apenas ultimos 168 backups (7 dias x 24 horas)
ls -t $BACKUP_DIR/dump_*.rdb | tail -n +169 | xargs rm -f
```

## Monitoramento

### Prometheus + Grafana

Instale Prometheus e Grafana em uma VM dedicada ou no app node:

```bash
# Docker Compose para monitoramento
cat > docker-compose-monitoring.yml << 'EOF'
version: '3.9'
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    restart: unless-stopped
EOF
```

Configuracao do Prometheus:

```yaml
# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'monetarie-pix-backend'
    static_configs:
      - targets:
          - '10.10.20.11:4003'
          - '10.10.20.12:4003'
          - '10.10.20.13:4003'
    metrics_path: /metrics

  - job_name: 'haproxy'
    static_configs:
      - targets: ['10.10.10.10:8404']

  - job_name: 'nats'
    static_configs:
      - targets:
          - '10.10.50.11:8222'
          - '10.10.50.12:8222'
          - '10.10.50.13:8222'
    metrics_path: /varz

  - job_name: 'postgresql'
    static_configs:
      - targets:
          - '10.10.30.11:9187'
          - '10.10.30.12:9187'
          - '10.10.30.13:9187'
```

### Dashboards Recomendados

| Dashboard | Metricas | Descricao |
|-----------|---------|-----------|
| Monetarie PIX Overview | TPS, latencia, erros | Visao geral da plataforma |
| PostgreSQL | conexoes, queries/s, replicacao | Saude do banco de dados |
| Redis | memoria, hit rate, conexoes | Performance do cache |
| NATS | mensagens/s, consumers, lag | Saude da mensageria |
| HAProxy | req/s, backends, erros | Load balancer |

## Opcao Single-Node (Desenvolvimento)

Para desenvolvimento ou teste em uma unica maquina:

```bash
# Usar Docker Compose modo dev
cd deploy/docker-compose
cp .env.example .env
# Editar .env

docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d
```

Isso inicia todos os componentes (backend, admin, PostgreSQL, Redis, NATS) em containers Docker na mesma maquina. Consulte [Docker Compose](./docker-compose.md) para detalhes.

## Resultado Esperado

Ao final deste guia, voce tera:

- Cluster PostgreSQL 16 com Patroni (failover automatico, 3 nos)
- Cluster Redis 7 com Sentinel (failover automatico, 3 nos)
- Cluster NATS 2.10 com JetStream (3 nos)
- 3 app nodes Docker com Monetarie PIX backend e admin
- HAProxy com TLS termination e health checks
- OPNsense com segmentacao de rede por VLAN
- Estrategia de backup definida para todos os componentes
- Monitoramento com Prometheus e Grafana