# Monetarie SPB - Deployment & Implementation Guide

**Step-by-step guide for deploying Monetarie SPB at client institutions**

**Version**: 1.0
**Date**: 2026-02-07
**Audience**: Implementation engineers, DevOps, infrastructure team
**Classification**: Internal / Client Engineering

---

## Table of Contents

1. [Prerequisites](#1-prerequisites)
2. [GCP Project Setup](#2-gcp-project-setup)
3. [GKE Cluster Provisioning](#3-gke-cluster-provisioning)
4. [Networking & DNS](#4-networking--dns)
5. [Database Setup (Cloud SQL)](#5-database-setup-cloud-sql)
6. [NATS JetStream Cluster](#6-nats-jetstream-cluster)
7. [TigerBeetle Financial Ledger](#7-tigerbeetle-financial-ledger)
8. [IBM MQ Deployment](#8-ibm-mq-deployment)
9. [Kubernetes Manifests](#9-kubernetes-manifests)
10. [Backend Deployment](#10-backend-deployment)
11. [Frontend Deployment](#11-frontend-deployment)
12. [Database Migration & Seed](#12-database-migration--seed)
13. [SSL/TLS Certificates](#13-ssltls-certificates)
14. [ICP-Brasil & BACEN RSFN](#14-icp-brasil--bacen-rsfn)
15. [Monitoring & Observability](#15-monitoring--observability)
16. [Go-Live Checklist](#16-go-live-checklist)
17. [Rollback Procedures](#17-rollback-procedures)
18. [Troubleshooting](#18-troubleshooting)

---

## 1. Prerequisites

### 1.1 Client Requirements

Before starting deployment, the client must provide:

| Requirement | Description | Source |
|-------------|-------------|--------|
| ISPB Code | 8-digit institution code registered with BACEN | BACEN registration |
| ICP-Brasil Certificate | A1 or A3 digital certificate for XMLDSig | Authorized CA (Serasa, CertiSign, etc.) |
| RSFN Network Access | VPN/dedicated link to BACEN RSFN network | BACEN onboarding |
| GCP Project | Google Cloud project with billing enabled | Client or Monetarie |
| Domain Names | DNS domains for frontend, API, and docs | Client |
| BACEN Onboarding Docs | Queue manager config, domain assignments, MQ credentials | BACEN Manual de Redes v9.3 |

### 1.2 Team Requirements

| Role | Responsibility |
|------|----------------|
| Monetarie Implementation Engineer | Platform deployment, configuration, testing |
| Client Infrastructure / DevOps | GCP access, networking, DNS, firewall rules |
| Client Security | ICP-Brasil certificates, secret management |
| Client BACEN Contact | RSFN coordination, BACEN certification tests |

### 1.3 Software & Tools

Install on your workstation:

```bash
# Google Cloud SDK
curl https://sdk.cloud.google.com | bash
gcloud init

# kubectl (via gcloud)
gcloud components install kubectl

# Helm (for cert-manager)
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Docker (for image builds)
# Install Docker Desktop or Docker Engine

# Elixir/Erlang (for backend compilation)
# Install via asdf or brew
asdf install elixir 1.17.0
asdf install erlang 27.0

# Node.js (for frontend build)
asdf install nodejs 20.11.0
```

---

## 2. GCP Project Setup

### 2.1 Enable Required APIs

```bash
PROJECT_ID="<client-project-id>"
gcloud config set project $PROJECT_ID

gcloud services enable \
  container.googleapis.com \
  sqladmin.googleapis.com \
  compute.googleapis.com \
  artifactregistry.googleapis.com \
  cloudbuild.googleapis.com \
  servicenetworking.googleapis.com \
  dns.googleapis.com \
  certificatemanager.googleapis.com \
  secretmanager.googleapis.com \
  monitoring.googleapis.com \
  logging.googleapis.com
```

### 2.2 Create Artifact Registry Repository

```bash
REGION="southamerica-east1"

gcloud artifacts repositories create monetarie \
  --repository-format=docker \
  --location=$REGION \
  --description="Monetarie SPB container images"
```

### 2.3 Create Service Accounts

```bash
# Backend service account (Cloud SQL access)
gcloud iam service-accounts create bacen-gateway \
  --display-name="BACEN Gateway Service Account"

# Grant Cloud SQL Client role
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:bacen-gateway@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/cloudsql.client"

# Grant Artifact Registry Reader role
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:bacen-gateway@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/artifactregistry.reader"
```

### 2.4 Register GKE Fleet Membership

```bash
# Required for Connect Gateway access
gcloud container fleet memberships register <client>-cluster \
  --gke-cluster=$REGION/<cluster-name> \
  --enable-workload-identity \
  --project=$PROJECT_ID
```

---

## 3. GKE Cluster Provisioning

### 3.1 Create VPC Network

```bash
# Create VPC
gcloud compute networks create spb-vpc --subnet-mode=custom

# Create subnet for GKE nodes
gcloud compute networks subnets create spb-nodes \
  --network=spb-vpc \
  --region=$REGION \
  --range=10.10.20.0/24 \
  --secondary-range pods=10.10.64.0/18,services=10.10.128.0/20

# Create subnet for VMs (NATS, TigerBeetle)
gcloud compute networks subnets create spb-vms \
  --network=spb-vpc \
  --region=$REGION \
  --range=10.10.40.0/24
```

### 3.2 Create Private GKE Cluster

```bash
CLUSTER_NAME="<client>-spb"

gcloud container clusters create $CLUSTER_NAME \
  --region=$REGION \
  --num-nodes=1 \
  --machine-type=e2-standard-4 \
  --node-locations=${REGION}-a,${REGION}-b,${REGION}-c \
  --network=spb-vpc \
  --subnetwork=spb-nodes \
  --cluster-secondary-range-name=pods \
  --services-secondary-range-name=services \
  --enable-private-nodes \
  --master-ipv4-cidr=172.16.0.0/28 \
  --enable-ip-alias \
  --enable-master-authorized-networks \
  --master-authorized-networks=<client-office-ip>/32 \
  --workload-pool=${PROJECT_ID}.svc.id.goog \
  --enable-autorepair \
  --enable-autoupgrade \
  --release-channel=stable \
  --disk-type=pd-ssd \
  --disk-size=100
```

### 3.3 Connect via Connect Gateway

**MANDATORY**: Always use Connect Gateway, never direct `kubectl`:

```bash
gcloud container fleet memberships get-credentials <client>-spb \
  --project=$PROJECT_ID \
  --location=$REGION

# Verify connection
kubectl get nodes
```

### 3.4 Create Node Pool (System)

```bash
gcloud container node-pools create np-system \
  --cluster=$CLUSTER_NAME \
  --region=$REGION \
  --machine-type=e2-standard-4 \
  --num-nodes=1 \
  --node-locations=${REGION}-a,${REGION}-b,${REGION}-c \
  --enable-autorepair \
  --enable-autoupgrade \
  --disk-type=pd-ssd \
  --disk-size=100
```

---

## 4. Networking & DNS

### 4.1 Reserve Static IP

```bash
gcloud compute addresses create spb-static-ip \
  --global

# Get the IP address
gcloud compute addresses describe spb-static-ip --global --format='get(address)'
```

### 4.2 Configure DNS Records

Create these DNS records pointing to the static IP:

| Record | Type | Value | Purpose |
|--------|------|-------|---------|
| `spbadmin-<env>.<domain>` | A | `<static-ip>` | Admin portal (frontend) |
| `spbapi-<env>.<domain>` | A | `<static-ip>` | Backend API |
| `spbdocs-<env>.<domain>` | A | `<static-ip>` | Documentation portal |

Example for dev:
```
spbadmin-dev.client.com.br  →  34.95.x.x
spbapi-dev.client.com.br    →  34.95.x.x
spbdocs-dev.client.com.br   →  34.95.x.x
```

### 4.3 Firewall Rules

```bash
# Allow NATS cluster internal communication
gcloud compute firewall-rules create allow-nats-cluster \
  --network=spb-vpc \
  --allow=tcp:4222,tcp:6222,tcp:8222 \
  --source-ranges=10.10.40.0/24 \
  --target-tags=nats

# Allow TigerBeetle cluster communication
gcloud compute firewall-rules create allow-tigerbeetle \
  --network=spb-vpc \
  --allow=tcp:3001 \
  --source-ranges=10.10.40.0/24,10.10.20.0/24 \
  --target-tags=tigerbeetle

# Allow GKE to NATS/TB VMs
gcloud compute firewall-rules create allow-gke-to-vms \
  --network=spb-vpc \
  --allow=tcp:4222,tcp:3001 \
  --source-ranges=10.10.20.0/24 \
  --target-tags=nats,tigerbeetle

# Allow IBM MQ ports within cluster (internal)
gcloud compute firewall-rules create allow-ibmmq \
  --network=spb-vpc \
  --allow=tcp:1414,tcp:5672,tcp:9443 \
  --source-ranges=10.10.20.0/24
```

---

## 5. Database Setup (Cloud SQL)

### 5.1 Create Cloud SQL Instance

```bash
DB_INSTANCE="<client>-pg"

gcloud sql instances create $DB_INSTANCE \
  --database-version=POSTGRES_16 \
  --region=$REGION \
  --tier=db-custom-8-32768 \
  --storage-type=SSD \
  --storage-size=100GB \
  --storage-auto-increase \
  --availability-type=REGIONAL \
  --network=spb-vpc \
  --no-assign-ip \
  --enable-point-in-time-recovery \
  --backup-start-time=02:00 \
  --retained-backups-count=30 \
  --database-flags=\
max_connections=5000,\
shared_buffers=8192MB,\
effective_cache_size=24576MB,\
work_mem=64MB,\
maintenance_work_mem=2048MB,\
log_statement=ddl,\
log_min_duration_statement=1000
```

### 5.2 Create Database and User

```bash
# Create database
gcloud sql databases create spb_prod --instance=$DB_INSTANCE

# Create user (store password securely)
DB_PASSWORD=$(openssl rand -base64 32)
gcloud sql users create spb_app \
  --instance=$DB_INSTANCE \
  --password=$DB_PASSWORD

echo "Database password: $DB_PASSWORD"
echo "SAVE THIS SECURELY - it will not be shown again"
```

### 5.3 Get Private IP

```bash
gcloud sql instances describe $DB_INSTANCE --format='get(ipAddresses[0].ipAddress)'
```

### 5.4 Cloud SQL Proxy Sidecar

The backend pods include a Cloud SQL Proxy sidecar automatically. The database connection string uses `127.0.0.1:5432` (localhost) because the proxy runs as a sidecar container in the same pod.

**Connection pattern**:
```
Backend Container  →  127.0.0.1:5432  →  Cloud SQL Proxy Sidecar  →  Cloud SQL Instance
```

**IMPORTANT**: Never connect directly to the Cloud SQL private IP from application code. Always use the proxy sidecar.

---

## 6. NATS JetStream Cluster

### 6.1 Provisioning Strategy

NATS runs on **dedicated VMs** (not in Kubernetes) for data persistence and performance isolation.

### 6.2 Create VMs (3-node cluster)

```bash
ZONES=("${REGION}-a" "${REGION}-b" "${REGION}-c")
NAMES=("nats-1" "nats-2" "nats-3")

for i in 0 1 2; do
  gcloud compute instances create ${NAMES[$i]} \
    --zone=${ZONES[$i]} \
    --machine-type=e2-standard-4 \
    --network=spb-vpc \
    --subnet=spb-vms \
    --tags=nats \
    --image-family=ubuntu-2204-lts \
    --image-project=ubuntu-os-cloud \
    --boot-disk-type=pd-ssd \
    --boot-disk-size=50GB
done
```

### 6.3 Install NATS on Each VM

SSH into each VM and run:

```bash
# Install NATS Server
curl -L https://github.com/nats-io/nats-server/releases/download/v2.10.11/nats-server-v2.10.11-linux-amd64.tar.gz -o nats.tar.gz
tar xzf nats.tar.gz
sudo mv nats-server-v2.10.11-linux-amd64/nats-server /usr/local/bin/
```

### 6.4 Configure NATS Cluster

Create `/etc/nats/nats.conf` on each node (adjust `server_name` and `routes`):

```conf
server_name: nats-1
port: 4222
http_port: 8222

jetstream {
  store_dir: /data/jetstream
  max_mem: 2G
  max_file: 50G
}

cluster {
  name: monetarie-nats
  port: 6222
  routes = [
    nats://10.10.40.x:6222    # nats-1
    nats://10.10.40.y:6222    # nats-2
    nats://10.10.40.z:6222    # nats-3
  ]
}

max_connections: 10000
max_payload: 8388608
logtime: true
```

### 6.5 Create Systemd Service

```bash
sudo tee /etc/systemd/system/nats.service <<EOF
[Unit]
Description=NATS JetStream Server
After=network.target

[Service]
Type=simple
User=nats
ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats.conf
Restart=always
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable nats
sudo systemctl start nats
```

### 6.6 Verify Cluster

```bash
# Check cluster members
curl http://10.10.40.x:8222/varz | jq '.server_name, .jetstream'
curl http://10.10.40.x:8222/routez | jq '.routes[].port'
```

---

## 7. TigerBeetle Financial Ledger

### 7.1 Provisioning Strategy

TigerBeetle runs on **dedicated VMs** with NVMe SSDs for deterministic performance. 3-node consensus cluster.

### 7.2 Create VMs

```bash
ZONES=("${REGION}-a" "${REGION}-b" "${REGION}-c")
NAMES=("tb-1" "tb-2" "tb-3")

for i in 0 1 2; do
  gcloud compute instances create ${NAMES[$i]} \
    --zone=${ZONES[$i]} \
    --machine-type=c2-standard-8 \
    --network=spb-vpc \
    --subnet=spb-vms \
    --tags=tigerbeetle \
    --local-ssd=interface=NVME \
    --image-family=ubuntu-2204-lts \
    --image-project=ubuntu-os-cloud
done
```

### 7.3 Install and Configure

SSH into each VM:

```bash
# Format NVMe SSD
sudo mkfs.ext4 /dev/nvme0n1
sudo mkdir -p /data/tigerbeetle
sudo mount /dev/nvme0n1 /data/tigerbeetle

# Download TigerBeetle
curl -LO https://github.com/tigerbeetle/tigerbeetle/releases/latest/download/tigerbeetle-x86_64-linux.zip
unzip tigerbeetle-x86_64-linux.zip
sudo mv tigerbeetle /usr/local/bin/

# Format data file (on each node, with correct replica index)
tigerbeetle format --cluster=0 --replica=<0|1|2> --replica-count=3 /data/tigerbeetle/data.tb

# Start (adjust addresses)
tigerbeetle start --addresses=10.10.40.a:3001,10.10.40.b:3001,10.10.40.c:3001 /data/tigerbeetle/data.tb
```

### 7.4 Create Systemd Service

Similar to NATS, create `/etc/systemd/system/tigerbeetle.service` with the appropriate start command.

---

## 8. IBM MQ Deployment

### 8.1 Mirror IBM MQ Image

**IMPORTANT**: Private GKE clusters cannot pull images from external registries like `icr.io`. You must mirror the image to Artifact Registry first.

```bash
# Pull on a machine with internet access
docker pull --platform linux/amd64 icr.io/ibm-messaging/mq:9.4.2.0-r1

# Tag for Artifact Registry
docker tag icr.io/ibm-messaging/mq:9.4.2.0-r1 \
  ${REGION}-docker.pkg.dev/${PROJECT_ID}/fluxiq/ibm-mq:9.4.2.0-r1

# Push to Artifact Registry
docker push ${REGION}-docker.pkg.dev/${PROJECT_ID}/fluxiq/ibm-mq:9.4.2.0-r1
```

### 8.2 Create IBM MQ Secrets

```bash
kubectl create secret generic ibmmq-secrets \
  --namespace=spb \
  --from-literal=admin-password="$(openssl rand -base64 24)" \
  --from-literal=app-password="$(openssl rand -base64 24)"
```

### 8.3 MQSC Configuration

The MQSC auto-configuration creates 21 BACEN-compliant queues (5 domains x 4 queues + 1 DLQ) and 5 server connection channels.

**CRITICAL RULES for MQSC auto-config**:
- Do NOT use `START SERVICE`, `START LISTENER`, or `REFRESH SECURITY` commands (invalid in auto-config context)
- Do NOT use `SSLCIPH` in dev environments (requires certificates not present)
- Use `CHCKCLNT(OPTIONAL)` for dev; use `CHCKCLNT(REQUIRED)` for production with certificates

Queue naming convention (per BACEN Manual de Redes v9.3):
```
QL.REQ.<ISPB_BACEN>.<ISPB_LOCAL>.<SEQ>   = Local request queue
QL.RSP.<ISPB_BACEN>.<ISPB_LOCAL>.<SEQ>   = Local response queue
QR.REQ.<ISPB_BACEN>.<ISPB_LOCAL>.<SEQ>   = Remote request queue
QR.RSP.<ISPB_LOCAL>.<ISPB_BACEN>.<SEQ>   = Remote response queue
```

| Domain | Seq | Channel | QoS | Message Types |
|--------|-----|---------|-----|---------------|
| SPB01 | 01 | C{ISPB}.{BACEN}X1 | Platina | STR, LPI |
| SPB02 | 02 | C{ISPB}.{BACEN}X2 | Ouro | CIP, Settlement |
| MES01 | 03 | C{ISPB}.{BACEN}X3 | Platina | Reserved |
| MES02 | 04 | C{ISPB}.{BACEN}X4 | Prata | General |
| MES03 | 05 | C{ISPB}.{BACEN}X5 | Bronze | General |

### 8.4 Apply Deployment

```bash
# Use the manifest at k8s/production/ibmmq-deployment.yaml
# Make sure image points to Artifact Registry (not icr.io)
kubectl apply -f k8s/production/ibmmq-deployment.yaml

# Verify
kubectl get pods -n spb -l app=ibmmq
kubectl exec -it deploy/ibmmq -n spb -- dspmq
# Expected: QMNAME(QM.<ISPB>.01)    STATUS(Running)
```

### 8.5 Verify Queues

```bash
kubectl exec -it deploy/ibmmq -n spb -- bash -c \
  'echo "DISPLAY QLOCAL(*)" | runmqsc QM.<ISPB>.01'
# Expected: 21 queues (5 domains x 4 + 1 DLQ)
```

### 8.6 If IBM MQ Pod Crashes

Common causes and fixes:

| Error | Cause | Fix |
|-------|-------|-----|
| `AMQ5776E: Auto configuration MQSC definitions contain invalid syntax` | START/REFRESH/SSLCIPH in MQSC | Remove invalid commands, delete PVC, re-apply |
| `ImagePullBackOff` | Can't reach icr.io from private cluster | Mirror image to Artifact Registry |
| `CrashLoopBackOff` after MQSC fix | Corrupted QM data from previous failure | Delete PVC (`kubectl delete pvc ibmmq-data -n spb`), then re-apply deployment |

---

## 9. Kubernetes Manifests

### 9.1 Apply Order

Apply manifests in this order:

```bash
# 1. Namespace
kubectl apply -f k8s/production/00-namespace.yaml

# 2. Secrets and ConfigMaps
# IMPORTANT: Edit with actual values before applying
kubectl apply -f k8s/production/02-secrets-configmaps.yaml

# 3. Infrastructure (NATS K8s reference, IBM MQ)
kubectl apply -f k8s/production/ibmmq-deployment.yaml

# 4. Backend (BACEN Gateway + Cloud SQL Proxy sidecar)
kubectl apply -f k8s/production/bacen-gateway-deployment.yaml

# 5. Frontend
kubectl apply -f k8s/production/frontend-deployment.yaml

# 6. Ingress and certificates
kubectl apply -f k8s/production/08-ingress.yaml
```

### 9.2 Namespace Configuration

The deployment uses namespace `spb`. Service accounts are created per component:

| Service Account | Purpose |
|----------------|---------|
| `bacen-gateway` | Backend pods, Cloud SQL access via Workload Identity |
| `frontend` | Frontend pods |

### 9.3 ConfigMap: Environment Variables

Key environment variables in `service-config` ConfigMap:

| Variable | Value | Description |
|----------|-------|-------------|
| `NATS_URL` | `nats://10.10.40.x:4222` | NATS cluster primary node |
| `IBM_MQ_HOST` | `ibmmq.spb.svc.cluster.local` | IBM MQ K8s service |
| `IBM_MQ_PORT` | `5672` (AMQP) or `1414` (native) | MQ connection port |
| `IBM_MQ_QMGR` | `QM.<ISPB>.01` | Queue Manager name |
| `IBM_MQ_CHANNEL` | `C<ISPB>.<BACEN>.<seq>` | Connection channel |
| `DATABASE_HOST` | `127.0.0.1` | Cloud SQL Proxy (localhost) |
| `DATABASE_PORT` | `5432` | PostgreSQL port |
| `DATABASE_POOL_SIZE` | `500` | Connection pool per pod |
| `MIX_ENV` | `prod` | Elixir environment |

### 9.4 Secrets Checklist

Before deploying, replace ALL `CHANGE_ME` / `REPLACE` values in:

- [ ] `db-credentials` - Database username, password, connection-name
- [ ] `bacen-gateway-secrets` - SECRET_KEY_BASE, ICP-Brasil certificate and key, IBM MQ password
- [ ] `ibmmq-secrets` - Admin and app passwords
- [ ] `api-gateway-secrets` - JWT secret, Guardian secret

Generate secrets:
```bash
# Phoenix secret key base (64+ chars)
mix phx.gen.secret

# JWT signing key
openssl rand -base64 64

# Database password
openssl rand -base64 32
```

---

## 10. Backend Deployment

### 10.1 Image Naming Convention

```
<region>-docker.pkg.dev/<project-id>/monetarie/spb-backend:<tag>
```

Example: `southamerica-east1-docker.pkg.dev/fluxiqbr/monetarie/spb-backend:55c6887`

**NEVER use `gcr.io` or generic names.**

### 10.2 Build via Cloud Build

```bash
cd services/bacen_gateway

gcloud builds submit --config=cloudbuild.yaml \
  --substitutions=SHORT_SHA=$(git rev-parse --short HEAD) \
  --project=$PROJECT_ID
```

The `cloudbuild.yaml` does:
1. Builds Docker image (linux/amd64, E2_HIGHCPU_8 machine, ~10-15 minutes)
2. Pushes to Artifact Registry with `:latest` and `:<SHORT_SHA>` tags
3. Deploys to GKE via Connect Gateway using `kubectl set image`
4. Waits for rollout completion (180s timeout)

**IMPORTANT**: The `cloud-sdk` Docker image in Cloud Build disables `gcloud components install`. The cloudbuild.yaml uses `apt-get install -y kubectl` instead.

### 10.3 Backend Pod Configuration

Each backend pod runs:
- **Main container**: `spb-backend` (Elixir/Phoenix, port 4000)
- **Sidecar**: `cloud-sql-proxy:2.8.2` (proxies 127.0.0.1:5432 to Cloud SQL)

Resources per pod:
```yaml
# Backend
requests: { cpu: 250m, memory: 512Mi }
limits:   { cpu: 1000m, memory: 2Gi }

# Cloud SQL Proxy sidecar
requests: { cpu: 100m, memory: 128Mi }
limits:   { cpu: 200m, memory: 256Mi }
```

### 10.4 Health Checks

```
Liveness:  GET /health  (port 4000, initial: 30s, period: 10s)
Readiness: GET /ready   (port 4000, initial: 20s, period: 5s)
```

### 10.5 HPA (Horizontal Pod Autoscaler)

```yaml
minReplicas: 2
maxReplicas: 10
targetCPU: 70%
targetMemory: 80%
scaleDown stabilization: 300s
```

---

## 11. Frontend Deployment

### 11.1 Image Naming Convention

```
<region>-docker.pkg.dev/<project-id>/monetarie/spb-frontend:<tag>
```

### 11.2 Build via Cloud Build

```bash
cd frontend-vue

gcloud builds submit --config=cloudbuild.yaml \
  --substitutions=SHORT_SHA=$(git rev-parse --short HEAD) \
  --project=$PROJECT_ID
```

Build steps:
1. `npm install` (node:20)
2. `npm run build` (produces static assets)
3. Docker build (nginx serving static files)
4. Push to Artifact Registry
5. Deploy to GKE (~5 minutes total)

### 11.3 Frontend Configuration

The frontend must have the correct API base URL configured at build time:

```typescript
// src/services/api.ts
// API_BASE_URL must point to the backend API domain
const API_BASE_URL = 'https://spbapi-<env>.<domain>/api'
```

### 11.4 CORS Configuration

The backend CORS plug must allow the frontend domain. Update `BacenGatewayWeb.CORSPlug`:

```elixir
# lib/bacen_gateway_web/plugs/cors_plug.ex
@allowed_origins [
  "https://spbadmin-<env>.<domain>",
  "https://spb-<env>.<domain>"
]
```

---

## 12. Database Migration & Seed

### 12.1 Run Ecto Migrations

```bash
# Via kubectl rpc into a running backend pod
kubectl exec -it deploy/spb-backend -n spb -c spb-backend -- \
  bin/bacen_gateway rpc 'BacenGateway.Release.migrate()'
```

Or via the migration job:
```bash
kubectl apply -f k8s/production/complete-migration-and-seed-job.yaml
kubectl logs job/migrate-and-seed-database -n spb
```

### 12.2 Database Tables (37 total)

The migrations create these tables:

**Core Tables**:
| Table | Purpose |
|-------|---------|
| `bacen_messages` | All BACEN/SPB messages (979 types) |
| `institutions` | Financial institutions (ISPB registry) |
| `users` | System users (admin, operator, auditor roles) |
| `clearings` | Clearing system configuration (STR, LPI, CIP, SELIC, etc.) |
| `bacen_tariffs` | BACEN message tariffs |

**Operation Tables**:
| Table | Purpose |
|-------|---------|
| `str_transfers` | STR reserve transfer operations |
| `lpi_operations` | LPI instant payment operations |
| `settlements` | Settlement records |
| `forex_operations` | Foreign exchange operations |
| `securities_operations` | Securities/SELIC operations |

**Configuration Tables**:
| Table | Purpose |
|-------|---------|
| `system_settings` | System-wide settings |
| `system_config` | Runtime configuration |
| `message_groups` | BACEN message groupings |
| `message_templates` | XML message templates |
| `operation_statuses` | Valid status transitions |

**Security Tables**:
| Table | Purpose |
|-------|---------|
| `sessions` | Active user sessions |
| `audit_logs` | Full audit trail (10-year retention) |
| `api_keys` | API authentication keys |
| `alcadas` | Authorization levels for high-value transactions |
| `vistos` | Approval workflow tracking |

**Reference Tables**:
| Table | Purpose |
|-------|---------|
| `bacen_error_codes` | 5,306+ BACEN error codes with retry policies |
| `bacen_holidays` | BACEN calendar / operating days |
| `bacen_operating_schedule` | Operating hours per clearing system |

### 12.3 Seed Data

Run the seed scripts via Elixir rpc:

```bash
# Connect to a running pod
kubectl exec -it deploy/spb-backend -n spb -c spb-backend -- \
  bin/bacen_gateway rpc 'Code.eval_file("priv/repo/seeds.exs")'
```

Required seed data:
- Financial institutions (BACEN public ISPB registry)
- BACEN error codes (5,306+ codes from BACEN catalog)
- Clearing systems (STR, LPI, CIP, SELIC, etc.)
- BACEN tariffs (per message type pricing)
- Message groups and operation statuses
- Operating schedule and holidays
- Admin user (initial login)

**IMPORTANT**: The `production_seed.exs` file is NOT included in the Docker image (it's in `.gitignore` for security). Copy it to the pod or run via `rpc` with inline code.

### 12.4 Column Name Reference

When writing custom seeds, use the **actual database column names**:

| Common Mistake | Correct Column |
|----------------|----------------|
| `inserted_at` | `created_at` |
| `status` (in bacen_messages) | `state` |
| `payer_cpf` | `payer_cpf_cnpj` |
| `holder_cpf` | `holder_cpf_cnpj` |

Always verify column names:
```sql
SELECT column_name FROM information_schema.columns
WHERE table_name = '<table>' ORDER BY ordinal_position;
```

---

## 13. SSL/TLS Certificates

### 13.1 GKE Managed Certificates

For the frontend and API domains, use GKE Managed Certificates:

```yaml
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: spb-cert
  namespace: spb
spec:
  domains:
  - spbadmin-<env>.<domain>
  - spbapi-<env>.<domain>
  - spbdocs-<env>.<domain>
```

### 13.2 HTTP to HTTPS Redirect

```yaml
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: frontend-config
  namespace: spb
spec:
  redirectToHttps:
    enabled: true
    responseCodeName: "MOVED_PERMANENTLY_DEFAULT"
```

### 13.3 Ingress Configuration

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: spb-ingress
  namespace: spb
  annotations:
    kubernetes.io/ingress.class: "gce"
    networking.gke.io/managed-certificates: "spb-cert"
    kubernetes.io/ingress.global-static-ip-name: "spb-static-ip"
    networking.gke.io/v1beta1.FrontendConfig: "frontend-config"
spec:
  rules:
  - host: spbadmin-<env>.<domain>
    http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: spb-frontend
            port:
              number: 80
  - host: spbapi-<env>.<domain>
    http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: spb-backend
            port:
              number: 4000
```

---

## 14. ICP-Brasil & BACEN RSFN

### 14.1 ICP-Brasil Certificate Setup

The ICP-Brasil digital certificate is **mandatory** for BACEN RSFN communication. It's used for XMLDSig (XML Digital Signature) on every message.

**Certificate Types**:
- **A1**: Software-based, stored as PFX/PEM file, 1-year validity
- **A3**: Hardware token (HSM/smartcard), 3-year validity (recommended for production)

```bash
# Store ICP-Brasil certificate as K8s secret
kubectl create secret tls icp-brasil-certificate \
  --namespace=spb \
  --cert=client.pem \
  --key=client-key.pem

# Store CA chain
kubectl create configmap icp-brasil-ca-chain \
  --namespace=spb \
  --from-file=ca-chain.pem
```

### 14.2 BACEN RSFN Network Connection

BACEN provides RSFN network access via:
- **VPN**: IPSec tunnel to BACEN RSFN network
- **Dedicated Link**: MPLS/leased line (for large institutions)

GCP Cloud VPN configuration:
```bash
# Create VPN gateway
gcloud compute vpn-gateways create bacen-vpn \
  --network=spb-vpc \
  --region=$REGION

# Create VPN tunnel (IKEv2, parameters from BACEN)
gcloud compute vpn-tunnels create bacen-tunnel \
  --vpn-gateway=bacen-vpn \
  --peer-address=<bacen-rsfn-ip> \
  --shared-secret=<bacen-provided-psk> \
  --ike-version=2 \
  --region=$REGION
```

### 14.3 Production IBM MQ Configuration

For production BACEN RSFN connection, enable SSL on MQ channels:

```
DEFINE CHANNEL('C<ISPB>.<BACEN>.1') CHLTYPE(SVRCONN) TRPTYPE(TCP) +
  MAXMSGL(4194304) MCAUSER('app') +
  SSLCIPH(ECDHE_RSA_AES_256_GCM_SHA384) +
  SSLCAUTH(REQUIRED) REPLACE
```

Set `IBM_MQ_ENABLED=true` in the backend environment to activate MQ message routing (instead of NATS-only dev mode).

### 14.4 BACEN Certification Tests

Before go-live, BACEN requires passing certification tests:

1. **Message Exchange**: Send/receive each message type your institution uses
2. **Error Handling**: Verify proper handling of BACEN error codes
3. **Settlement**: Complete full settlement cycle (STR transfer → confirmation)
4. **Stress Test**: Handle production-like volumes
5. **Failover**: Demonstrate recovery from MQ disconnect
6. **Security**: XMLDSig validation, certificate verification

---

## 15. Monitoring & Observability

### 15.1 Health Endpoints

All services expose health endpoints:

| Endpoint | Purpose |
|----------|---------|
| `GET /health` | Basic liveness (returns 200) |
| `GET /ready` | Readiness (DB + NATS connected) |
| `GET /api/dashboard/stats` | Application-level statistics |

### 15.2 Prometheus Metrics

Backend pods expose metrics on port 9090:

```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9090"
  prometheus.io/path: "/metrics"
```

### 15.3 Logging

All logs go to Cloud Logging via GKE integration. Structured JSON logging in production:

```bash
# View backend logs
kubectl logs deploy/spb-backend -n spb -c spb-backend --tail=100 -f

# View Cloud SQL Proxy logs
kubectl logs deploy/spb-backend -n spb -c cloud-sql-proxy --tail=50

# View IBM MQ logs
kubectl logs deploy/ibmmq -n spb --tail=100
```

### 15.4 Alerts (Recommended)

Configure GCP Monitoring alerts for:

| Alert | Condition | Severity |
|-------|-----------|----------|
| Pod restart | RestartCount > 3 in 10 min | Critical |
| High CPU | CPU > 80% for 5 min | Warning |
| High memory | Memory > 85% for 5 min | Warning |
| DB connections | Pool usage > 80% | Warning |
| NATS disconnect | NATS connection lost | Critical |
| IBM MQ queue depth | Queue depth > 10,000 | Warning |
| Failed messages | > 10 failures in 5 min | Critical |
| Certificate expiry | ICP-Brasil cert < 30 days | Critical |

---

## 16. Go-Live Checklist

### Infrastructure

- [ ] GKE cluster created and accessible via Connect Gateway
- [ ] Cloud SQL instance provisioned with backups enabled
- [ ] NATS JetStream 3-node cluster running on VMs
- [ ] TigerBeetle 3-node cluster running on VMs
- [ ] IBM MQ deployed with correct BACEN queues (21 queues verified)
- [ ] VPC, subnets, and firewall rules configured
- [ ] Static IP reserved and DNS records pointing

### Security

- [ ] All K8s secrets populated with real values (no `CHANGE_ME` remaining)
- [ ] ICP-Brasil certificate installed and verified
- [ ] JWT signing keys generated and configured
- [ ] Database passwords are strong (32+ chars, random)
- [ ] IBM MQ passwords set
- [ ] CORS plug configured with correct frontend domains
- [ ] TLS certificates provisioned for all domains

### Application

- [ ] Database migrations ran successfully (37 tables)
- [ ] Seed data loaded (institutions, error codes, clearings, tariffs, users)
- [ ] Backend pods running (2+ replicas)
- [ ] Frontend pods running (2+ replicas)
- [ ] Backend `/health` returns 200
- [ ] Login works (admin user can authenticate)
- [ ] Dashboard loads with real data
- [ ] NATS connection verified (`Process.whereis(:bacen_gateway_nats)` returns PID)
- [ ] All API endpoints responding (test key flows)

### BACEN Integration

- [ ] RSFN network connectivity established (VPN or dedicated link)
- [ ] IBM MQ channels connected to BACEN
- [ ] XMLDSig signing verified with ICP-Brasil certificate
- [ ] BACEN certification tests passed
- [ ] Message types your institution uses are tested
- [ ] Error handling verified for common BACEN error codes
- [ ] Alcada (authorization levels) configured for high-value transactions

### Operations

- [ ] Monitoring alerts configured
- [ ] Log aggregation working (Cloud Logging)
- [ ] Backup strategy documented and tested
- [ ] Rollback procedure documented and tested
- [ ] Operator training completed
- [ ] Emergency contacts documented (BACEN, Monetarie, client)
- [ ] Credential rotation schedule documented

---

## 17. Rollback Procedures

### 17.1 Backend Rollback

```bash
# Get previous image tag
kubectl rollout history deployment/spb-backend -n spb

# Rollback to previous version
kubectl rollout undo deployment/spb-backend -n spb

# Rollback to specific revision
kubectl rollout undo deployment/spb-backend -n spb --to-revision=<N>

# Verify
kubectl rollout status deployment/spb-backend -n spb
```

### 17.2 Frontend Rollback

```bash
kubectl rollout undo deployment/spb-frontend -n spb
kubectl rollout status deployment/spb-frontend -n spb
```

### 17.3 Database Rollback

```bash
# Ecto rollback (one migration)
kubectl exec -it deploy/spb-backend -n spb -c spb-backend -- \
  bin/bacen_gateway rpc 'BacenGateway.Release.rollback(BacenGateway.Repo, 1)'
```

For major database issues, restore from Cloud SQL point-in-time recovery:
```bash
gcloud sql instances clone $DB_INSTANCE ${DB_INSTANCE}-restore \
  --point-in-time=<timestamp>
```

### 17.4 IBM MQ Recovery

```bash
# If QM data is corrupted:
kubectl delete pvc ibmmq-data -n spb
kubectl delete pod -l app=ibmmq -n spb
# PVC will be recreated and QM re-initialized from MQSC config
```

---

## 18. Troubleshooting

### 18.1 Common Issues

| Issue | Diagnosis | Fix |
|-------|-----------|-----|
| Backend 500 on `/api/messages` | Ecto schema mismatch with DB columns | Verify schema matches actual column names |
| `ImagePullBackOff` | Private GKE can't reach external registry | Mirror image to Artifact Registry |
| Cloud SQL connection refused | Cloud SQL Proxy sidecar not running | Check sidecar logs, verify connection-name |
| CORS errors in browser | Frontend domain not in CORS plug | Add domain to `BacenGatewayWeb.CORSPlug` |
| Login returns 401 | Wrong email domain | Query users table for actual email addresses |
| NATS not connected | Wrong NATS_URL env var | Verify VM IPs, test with `curl http://<ip>:8222/varz` |
| IBM MQ CrashLoopBackOff | Invalid MQSC syntax | Check logs for AMQ5776E, fix MQSC, delete PVC, re-apply |
| Database pool exhausted | Pool size too small or connection leak | Increase `POOL_SIZE`, check for leaking connections |
| Route not found (404) | Router route ordering | Ensure specific routes (`/clearings/status`) come before parameterized routes (`/clearings/:id`) |

### 18.2 Diagnostic Commands

```bash
# Pod status
kubectl get pods -n spb -o wide

# Pod logs
kubectl logs deploy/spb-backend -n spb -c spb-backend --tail=200

# Pod describe (events, status)
kubectl describe pod <pod-name> -n spb

# Database connectivity test
kubectl exec -it deploy/spb-backend -n spb -c spb-backend -- \
  bin/bacen_gateway rpc 'BacenGateway.Repo.query!("SELECT 1")'

# NATS connectivity test
kubectl exec -it deploy/spb-backend -n spb -c spb-backend -- \
  bin/bacen_gateway rpc 'Process.whereis(:bacen_gateway_nats) |> Process.alive?()'

# IBM MQ status
kubectl exec -it deploy/ibmmq -n spb -- dspmq

# IBM MQ queue depths
kubectl exec -it deploy/ibmmq -n spb -- bash -c \
  'echo "DISPLAY QLOCAL(*) CURDEPTH" | runmqsc QM.<ISPB>.01'

# Cloud SQL Proxy health
kubectl logs deploy/spb-backend -n spb -c cloud-sql-proxy --tail=50
```

### 18.3 Emergency Procedures

**Database Emergency**: If the database is unresponsive:
1. Check Cloud SQL instance status in GCP Console
2. Check Cloud SQL Proxy sidecar logs
3. Verify network connectivity (VPC peering, firewall rules)
4. If needed, failover to Cloud SQL replica

**BACEN Message Stuck**: If messages are stuck in `pending`:
1. Check IBM MQ queue depths
2. Check NATS connection status
3. Check backend logs for errors
4. Manually retry via API: `POST /api/operations/<id>/send`

**Full System Recovery**:
1. Verify GKE cluster health: `kubectl get nodes`
2. Verify all pods: `kubectl get pods -n spb`
3. Verify database: connect and run `SELECT 1`
4. Verify NATS: check cluster health via monitoring port
5. Verify IBM MQ: `dspmq` should show `Running`
6. Test login via API
7. Test a simple query via dashboard

---

## Appendix A: Environment Matrix

| Component | Dev | Homologation | Production |
|-----------|-----|-------------|------------|
| Namespace | `spb` | `spb` | `spb` |
| Backend replicas | 2 | 2 | 2-10 (HPA) |
| Frontend replicas | 2 | 2 | 2-10 (HPA) |
| DB pool size | 500 | 500 | 500 |
| NATS nodes | 3 (VMs) | 3 (VMs) | 3 (VMs) |
| TigerBeetle nodes | 3 (VMs) | 3 (VMs) | 3 (VMs) |
| IBM MQ | 1 pod (dev license) | 1 pod | 1 pod (production license) |
| SSL | GKE Managed Cert | GKE Managed Cert | GKE Managed Cert + ICP-Brasil |
| IBM MQ SSL | Disabled | Enabled | Enabled (mandatory) |
| BACEN RSFN | Simulator | Homologation | Production |

## Appendix B: Port Reference

| Service | Port | Protocol | Access |
|---------|------|----------|--------|
| Backend API | 4000 | HTTP | Via Ingress |
| Frontend | 80 | HTTP | Via Ingress |
| Cloud SQL Proxy | 5432 | TCP | Pod-internal (127.0.0.1) |
| NATS Client | 4222 | TCP | VPC internal |
| NATS Cluster | 6222 | TCP | VPC internal (VMs only) |
| NATS Monitor | 8222 | HTTP | VPC internal |
| TigerBeetle | 3001 | TCP | VPC internal |
| IBM MQ Native | 1414 | TCP | K8s internal |
| IBM MQ AMQP | 5672 | TCP | K8s internal |
| IBM MQ Console | 9443 | HTTPS | K8s internal |
| Prometheus Metrics | 9090 | HTTP | K8s internal |

## Appendix C: Contact & Support

| Contact | Purpose |
|---------|---------|
| BACEN RSFN Support | `infra.rsfn@bcb.gov.br`, +55 61 3414-3012 |
| ICP-Brasil CA | Contact your certificate authority |
| Monetarie Engineering | Implementation support |
| GCP Support | Infrastructure issues |

---

*Document generated 2026-02-07. This guide covers Monetarie SPB platform v1.0 deployment on GKE.*