# Monetarie SPB -- Testing & QA Guide

**Version:** 1.0
**Date:** 2026-02-07
**Classification:** Internal -- Engineering & QA Teams
**Scope:** Complete testing procedures for Monetarie SPB platform

---

## Table of Contents

1. [Test Environment Overview](#1-test-environment-overview)
2. [Simulator Management](#2-simulator-management)
3. [Test Scenarios](#3-test-scenarios)
4. [Full Test Procedures](#4-full-test-procedures)
5. [Frontend Testing](#5-frontend-testing)
6. [Database Verification](#6-database-verification)
7. [Monitoring Verification](#7-monitoring-verification)
8. [Deployment Verification Checklist](#8-deployment-verification-checklist)
9. [Troubleshooting](#9-troubleshooting)

---

## 1. Test Environment Overview

### 1.1 Environment URLs

| System | URL | Purpose |
|--------|-----|---------|
| Frontend (Admin/Operator) | https://spbadmin-dev.fluxiq.com.br | Vue 3 web application |
| API Gateway | https://spbapi-dev.fluxiq.com.br | REST API entry point (port 4000) |
| BACEN Simulator | https://simulator-dev.fluxiq.com.br | BACEN response simulator |
| Grafana Monitoring | https://monitoring-dev.fluxiq.com.br | Prometheus + Grafana dashboards |

### 1.2 Service Ports (Internal)

| Service | Port | Kubernetes Service Name |
|---------|------|------------------------|
| API Gateway | 4000 | api-gateway.spb.svc.cluster.local |
| Auth Service | 4001 | auth-service.spb.svc.cluster.local |
| BACEN Gateway | 4002 | bacen-gateway.spb.svc.cluster.local |
| Message Processor | 4003 | message-processor.spb.svc.cluster.local |
| User Management | 4004 | user-management.spb.svc.cluster.local |
| Transaction Service | 4005 | transaction-service.spb.svc.cluster.local |
| Securities Service | 4006 | securities-service.spb.svc.cluster.local |
| Settlement Service | 4007 | settlement-service.spb.svc.cluster.local |
| Forex Service | 4008 | forex-service.spb.svc.cluster.local |
| Cash Service | 4009 | cash-service.spb.svc.cluster.local |
| Extract Service | 4010 | extract-service.spb.svc.cluster.local |
| NATS JetStream | 4222 | nats.spb.svc.cluster.local |
| NATS Monitor | 8222 | nats.spb.svc.cluster.local |

### 1.3 Test Credentials

| Account | Email / Username | Password | Role |
|---------|-----------------|----------|------|
| System Admin | admin | Monetarie#Adm@2026 | Full admin access |
| API Admin | admin@monetarie.com.br | Monetarie#Adm@2026 | API full access |
| Grafana | admin | Monetarie#Adm@2026 | Monitoring dashboards |

### 1.4 Infrastructure

| Component | Technology | Location |
|-----------|-----------|----------|
| Kubernetes | GKE (southamerica-east1) | Namespace: spb |
| Database | PostgreSQL 16 (Cloud SQL) | 127.0.0.1:5432 via proxy |
| Message Queue (Internal) | NATS JetStream | nats.spb.svc.cluster.local:4222 |
| Message Queue (BACEN) | IBM MQ (AMQP 1.0) | ibmmq.spb.svc.cluster.local:5672 |
| Image Registry | Artifact Registry | southamerica-east1-docker.pkg.dev/fluxiqbr/fluxiq/ |

### 1.5 GKE Cluster Access

The GKE cluster is private. Always use Connect Gateway for access:

```bash
# Step 1: Get credentials via Connect Gateway
gcloud container fleet memberships get-credentials fluxiq-dev \
  --project=fluxiqbr \
  --location=southamerica-east1

# Step 2: Verify access
kubectl get pods -n spb
```

Never use `kubectl` directly against the cluster IP -- it will not work from local machines or Cloud Build.

---

## 2. Simulator Management

### 2.1 What is the Simulator?

The BACEN Simulator is an Elixir/Phoenix application that emulates BACEN's message processing behavior. It provides:

- Simulated responses for all 1,222 BACEN message types
- 10 configurable test scenarios with different success rates, latencies, and error patterns
- A Phoenix LiveView real-time dashboard for monitoring
- A REST API for programmatic control
- Statistics tracking per scenario

The simulator uses a GenServer-based `ScenarioEngine` that determines, for each incoming message, whether to respond with success or failure based on the active scenario's success rate and error code configuration.

### 2.2 Enable/Disable Simulator

#### Local Development (Docker Compose)

```bash
# Start the simulator
cd /Users/luizpenha/monetarie/spb
docker compose up bacen_simulator -d

# Stop the simulator
docker compose stop bacen_simulator

# Check logs
docker compose logs -f bacen_simulator
```

#### Local Development (Mix)

```bash
cd /Users/luizpenha/monetarie/spb/simulator
mix deps.get
mix compile
mix phx.server
# Simulator available at http://localhost:4001
```

#### Kubernetes (GKE)

```bash
# Enable simulator (scale up)
kubectl scale deployment bacen-simulator -n spb --replicas=1

# Disable simulator (scale down)
kubectl scale deployment bacen-simulator -n spb --replicas=0

# Check status
kubectl get deployment bacen-simulator -n spb

# View logs
kubectl logs -n spb deployment/bacen-simulator --tail=100 -f
```

### 2.3 Simulator Dashboard

The simulator includes a Phoenix LiveView dashboard accessible via browser:

| Page | Path | Description |
|------|------|-------------|
| Dashboard | `/` | Real-time processing statistics, active scenario display, throughput counters |
| Scenarios | `/scenarios` | List all 10 scenarios, activate/deactivate, view configuration |
| Message Log | `/messages` | Live feed of all processed messages with type, result, and response time |
| Configuration | `/config` | Adjust simulator behavior: custom success rates, response delays, error codes |

**Access:**

- Local: http://localhost:4001
- GKE: https://simulator-dev.fluxiq.com.br

### 2.4 Simulator API

All simulator API endpoints accept JSON and XML content types.

#### List Scenarios

```bash
curl -s http://localhost:4001/api/scenarios | jq
```

Returns all 10 scenarios with their configuration.

#### Activate a Scenario

```bash
curl -s -X POST http://localhost:4001/api/scenarios/stress_test/activate | jq
```

Changes the active scenario. Only one scenario can be active at a time.

#### Get Current Scenario

```bash
curl -s http://localhost:4001/api/scenarios/current | jq
```

Returns the currently active scenario and its configuration.

#### Send Test Message

```bash
curl -s -X POST http://localhost:4001/api/mq/send \
  -H "Content-Type: application/json" \
  -d '{
    "message_type": "STR0004",
    "xml_content": "<?xml version=\"1.0\"?><BCMSG><SISMSG><CodMsg>STR0004</CodMsg></SISMSG></BCMSG>",
    "destination_ispb": "00038166"
  }' | jq
```

#### Receive Response Message

```bash
curl -s http://localhost:4001/api/mq/receive | jq
```

Returns the next message from the simulator's response queue.

#### Check Queue Status

```bash
curl -s http://localhost:4001/api/mq/status | jq
```

Returns queue depths and connection status.

#### Clear a Queue

```bash
curl -s -X DELETE http://localhost:4001/api/mq/queues/QL.REQ.00038166.12345678.01
```

Clears all messages from the specified queue.

#### Get Statistics

```bash
curl -s http://localhost:4001/api/statistics | jq
```

Returns:

```json
{
  "total_requests": 0,
  "successful_responses": 0,
  "failed_responses": 0,
  "errors_by_code": {},
  "avg_response_time": 0
}
```

#### Get Router Statistics

```bash
curl -s http://localhost:4001/api/statistics/router | jq
```

Returns per-message-type routing statistics.

#### Get Scenario Statistics

```bash
curl -s http://localhost:4001/api/statistics/scenario | jq
```

Returns statistics for the current scenario.

#### Reset Statistics

```bash
curl -s -X POST http://localhost:4001/api/statistics/reset
```

Resets all counters to zero.

#### Process Message Directly

```bash
curl -s -X POST http://localhost:4001/api/messages/process \
  -H "Content-Type: application/json" \
  -d '{
    "message_type": "GEN0001",
    "xml_content": "<?xml version=\"1.0\"?><BCMSG>...</BCMSG>"
  }' | jq
```

Processes a message through the simulator engine without going through MQ.

#### Get Message Template

```bash
curl -s http://localhost:4001/api/messages/templates/STR0004 | jq
```

Returns the XML template for the specified message type.

#### Health Check

```bash
curl -s http://localhost:4001/api/health | jq
```

---

## 3. Test Scenarios

### 3.1 Scenario Configuration Reference

Each scenario controls the simulator's behavior via these parameters:

| Parameter | Description |
|-----------|-------------|
| `success_rate` | Probability (0.0-1.0) that a message will get a success response |
| `avg_response_time_ms` | Base response delay in milliseconds |
| `response_time_variation_ms` | Random variation added to base delay (+/- this value) |
| `error_codes` | Pool of error codes to randomly inject on failure |

### 3.2 Detailed Scenario Reference

#### Scenario 1: Normal Operations (Default)

**Key:** `normal_operations`
**Purpose:** Default happy-path testing for development and basic integration validation.

| Parameter | Value |
|-----------|-------|
| Success Rate | 98% |
| Avg Response Time | 150ms +/- 50ms |
| Error Codes | None (uses default 5001 on the 2% failure) |

**When to use:** Initial integration testing, development, demo environments, smoke tests.

#### Scenario 2: Stress Test

**Key:** `stress_test`
**Purpose:** Performance testing under load with occasional system-level failures.

| Parameter | Value |
|-----------|-------|
| Success Rate | 95% |
| Avg Response Time | 500ms +/- 200ms |
| Error Codes | 5001 (Internal server error), 5003 (Service unavailable), 7001 (Request timeout) |

**When to use:** Load testing, throughput validation, capacity planning. Send 1,000+ messages to measure system behavior under sustained load.

#### Scenario 3: Error Testing

**Key:** `error_testing`
**Purpose:** Validate error handling across all error categories.

| Parameter | Value |
|-----------|-------|
| Success Rate | 50% |
| Avg Response Time | 100ms +/- 50ms |
| Error Codes | 1001, 1002, 1003 (validation), 2004 (auth), 4001, 4002 (business), 5001 (system) |

**When to use:** Error handling validation, retry logic testing, DLQ verification.

#### Scenario 4: Timeout Testing

**Key:** `timeout_testing`
**Purpose:** Validate timeout detection and recovery.

| Parameter | Value |
|-----------|-------|
| Success Rate | 70% |
| Avg Response Time | 2000ms +/- 1000ms |
| Error Codes | 7001 (Request timeout), 7002 (Response timeout), 7003 (Processing timeout) |

**When to use:** Timeout handling validation, circuit breaker testing, long-running operation handling.

#### Scenario 5: Validation Errors

**Key:** `validation_errors`
**Purpose:** Test schema validation and input checking.

| Parameter | Value |
|-----------|-------|
| Success Rate | 60% |
| Avg Response Time | 100ms +/- 30ms |
| Error Codes | 1001-1006 (Invalid format, missing field, invalid value, size limit, XML structure, schema) |

**When to use:** XML validation testing, schema compliance, input sanitization.

#### Scenario 6: Network Issues

**Key:** `network_issues`
**Purpose:** Test network resilience and reconnection logic.

| Parameter | Value |
|-----------|-------|
| Success Rate | 80% |
| Avg Response Time | 800ms +/- 400ms |
| Error Codes | 6001 (Connection timeout), 6002 (Connection refused), 6003 (Network unreachable) |

**When to use:** Network resilience testing, auto-reconnect validation, failover testing.

#### Scenario 7: Business Errors

**Key:** `business_errors`
**Purpose:** Test domain-specific business rule enforcement.

| Parameter | Value |
|-----------|-------|
| Success Rate | 85% |
| Avg Response Time | 200ms +/- 100ms |
| Error Codes | 4001 (Insufficient funds), 4002 (Duplicate), 4003 (Invalid account), 4004 (Blocked), 4005 (Limit exceeded) |

**When to use:** Business logic validation, account status handling, duplicate detection.

#### Scenario 8: High Performance

**Key:** `high_performance`
**Purpose:** Maximum throughput testing with minimal errors.

| Parameter | Value |
|-----------|-------|
| Success Rate | 99% |
| Avg Response Time | 50ms +/- 20ms |
| Error Codes | None |

**When to use:** Throughput benchmarking, peak load simulation, baseline performance measurement.

#### Scenario 9: Authentication Failures

**Key:** `authentication_failures`
**Purpose:** Test certificate and authentication error handling.

| Parameter | Value |
|-----------|-------|
| Success Rate | 40% |
| Avg Response Time | 150ms +/- 50ms |
| Error Codes | 2001-2005 (Invalid cert, expired, revoked, invalid signature, verification failed) |

**When to use:** Certificate error handling, authentication flow validation, security testing.

#### Scenario 10: Authorization Issues

**Key:** `authorization_issues`
**Purpose:** Test permission and access control error handling.

| Parameter | Value |
|-----------|-------|
| Success Rate | 50% |
| Avg Response Time | 120ms +/- 40ms |
| Error Codes | 3001-3004 (Insufficient permissions, not authorized, service unavailable, not permitted) |

**When to use:** RBAC testing, permission enforcement, institutional access control.

---

## 4. Full Test Procedures

### 4.1 Smoke Test (5 minutes)

A quick verification that all essential system components are operational.

**Prerequisites:** System deployed and accessible.

**Step 1: Verify frontend loads**

```
Open https://spbadmin-dev.fluxiq.com.br in a browser.
Expected: Login page renders with Monetarie branding.
```

**Step 2: Login**

```
Enter credentials: admin / Monetarie#Adm@2026
Click "Entrar" (Login)
Expected: Dashboard loads with statistics panels and charts.
```

**Step 3: Check API health**

```bash
curl -s https://spbapi-dev.fluxiq.com.br/api/health | jq
```

Expected response:

```json
{
  "status": "healthy"
}
```

**Step 4: Authenticate via API**

```bash
TOKEN=$(curl -s -X POST https://spbapi-dev.fluxiq.com.br/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@monetarie.com.br","password":"Monetarie#Adm@2026"}' | jq -r '.token')

echo "Token obtained: ${TOKEN:0:20}..."
```

Expected: A non-empty JWT token.

**Step 5: Send a test message**

```bash
curl -s -X POST https://spbapi-dev.fluxiq.com.br/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message_type": "GEN0001",
    "xml_content": "<?xml version=\"1.0\"?><BCMSG xmlns=\"http://www.bcb.gov.br/SPB\"><IdentdEmissor>12345678</IdentdEmissor><IdentdDesworko>00038166</IdentdDesworko><SISMSG><CodMsg>GEN0001</CodMsg></SISMSG></BCMSG>",
    "destination_ispb": "00038166"
  }' | jq
```

Expected: `201 Created` with status `pending`.

**Step 6: Verify message appears in list**

```bash
curl -s https://spbapi-dev.fluxiq.com.br/api/messages?message_type=GEN0001 \
  -H "Authorization: Bearer $TOKEN" | jq '.data | length'
```

Expected: At least 1 message returned.

**Step 7: Check Prometheus metrics**

```bash
curl -s https://spbapi-dev.fluxiq.com.br/metrics | head -5
```

Expected: Prometheus-format metrics text.

**Smoke Test Pass Criteria:** All 7 steps complete without errors.

### 4.2 API Integration Test

A comprehensive test of all REST API endpoints.

#### Setup

```bash
# Get authentication token
export API_URL="https://spbapi-dev.fluxiq.com.br"
export TOKEN=$(curl -s -X POST $API_URL/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@monetarie.com.br","password":"Monetarie#Adm@2026"}' | jq -r '.token')
```

#### Test 1: Authentication Endpoints

```bash
# Login (already done above)
echo "1.1 Login: PASS (token obtained)"

# Verify token
curl -s -X POST $API_URL/api/auth/verify \
  -H "Content-Type: application/json" \
  -d "{\"token\":\"$TOKEN\"}" | jq '.valid'
# Expected: true
echo "1.2 Verify: PASS"

# Refresh token
NEW_TOKEN=$(curl -s -X POST $API_URL/api/auth/refresh \
  -H "Authorization: Bearer $TOKEN" | jq -r '.token')
echo "1.3 Refresh: ${NEW_TOKEN:+PASS}"
```

#### Test 2: Health Endpoints

```bash
# Health check
STATUS=$(curl -s -o /dev/null -w "%{http_code}" $API_URL/api/health)
echo "2.1 Health: $STATUS (expected 200)"

# Readiness check (BACEN Gateway)
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://bacen-gateway.spb.svc.cluster.local:4002/ready)
echo "2.2 Ready: $STATUS (expected 200)"
```

#### Test 3: Messages CRUD

```bash
# Create a message
MSG_ID=$(curl -s -X POST $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message_type": "STR0004",
    "xml_content": "<?xml version=\"1.0\"?><BCMSG><SISMSG><CodMsg>STR0004</CodMsg></SISMSG></BCMSG>",
    "destination_ispb": "00038166"
  }' | jq -r '.id')
echo "3.1 Create message: ID=$MSG_ID"

# List messages
COUNT=$(curl -s $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" | jq '.data | length')
echo "3.2 List messages: count=$COUNT"

# Get specific message
STATUS=$(curl -s $API_URL/api/messages/$MSG_ID \
  -H "Authorization: Bearer $TOKEN" | jq -r '.status')
echo "3.3 Get message: status=$STATUS"

# List by category
CATS=$(curl -s "$API_URL/api/messages?category=STR" \
  -H "Authorization: Bearer $TOKEN" | jq '.data | length')
echo "3.4 Filter by category: STR count=$CATS"

# List by status
PENDING=$(curl -s "$API_URL/api/messages?status=pending" \
  -H "Authorization: Bearer $TOKEN" | jq '.data | length')
echo "3.5 Filter by status: pending count=$PENDING"

# Get categories
curl -s $API_URL/api/messages/categories \
  -H "Authorization: Bearer $TOKEN" | jq '.categories | length'
echo "3.6 Categories: listed"

# Get states
curl -s $API_URL/api/messages/states \
  -H "Authorization: Bearer $TOKEN" | jq '.states | length'
echo "3.7 States: listed"
```

#### Test 4: Payments Endpoints

```bash
# Create payment
PAY_ID=$(curl -s -X POST $API_URL/api/payments \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "STR",
    "amount": 50000.00,
    "currency": "BRL",
    "sender": {"ispb": "12345678", "account": "00001234-5"},
    "receiver": {"ispb": "00038166", "account": "00009876-1"},
    "purpose_code": "01"
  }' | jq -r '.id')
echo "4.1 Create payment: ID=$PAY_ID"

# List payments
curl -s $API_URL/api/payments \
  -H "Authorization: Bearer $TOKEN" | jq '.data | length'
echo "4.2 List payments: listed"

# Get payment
curl -s $API_URL/api/payments/$PAY_ID \
  -H "Authorization: Bearer $TOKEN" | jq '.status'
echo "4.3 Get payment: retrieved"
```

#### Test 5: Dashboard Endpoints

```bash
# Dashboard stats
curl -s $API_URL/api/dashboard/stats \
  -H "Authorization: Bearer $TOKEN" | jq 'keys'
echo "5.1 Dashboard stats: returned"

# Recent activity
curl -s $API_URL/api/dashboard/recent-activity \
  -H "Authorization: Bearer $TOKEN" | jq '.activities | length'
echo "5.2 Recent activity: returned"
```

#### Test 6: Statistics Endpoints

```bash
# Stats by state
curl -s $API_URL/api/stats/by-state \
  -H "Authorization: Bearer $TOKEN" | jq
echo "6.1 Stats by state: returned"

# Stats by category
curl -s $API_URL/api/stats/by-category \
  -H "Authorization: Bearer $TOKEN" | jq
echo "6.2 Stats by category: returned"
```

#### Test 7: Users CRUD

```bash
# List users
curl -s $API_URL/api/users \
  -H "Authorization: Bearer $TOKEN" | jq 'length'
echo "7.1 List users: returned"

# Create user
USER_ID=$(curl -s -X POST $API_URL/api/users \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Test User",
    "email": "testuser@fluxiq.com",
    "role": "operator",
    "password": "Test@123456"
  }' | jq -r '.id')
echo "7.2 Create user: ID=$USER_ID"

# Get user
curl -s $API_URL/api/users/$USER_ID \
  -H "Authorization: Bearer $TOKEN" | jq '.name'
echo "7.3 Get user: retrieved"

# Update user
curl -s -X PUT $API_URL/api/users/$USER_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated Test User"}' | jq '.name'
echo "7.4 Update user: updated"

# Delete user
curl -s -X DELETE $API_URL/api/users/$USER_ID \
  -H "Authorization: Bearer $TOKEN" -o /dev/null -w "%{http_code}"
echo "7.5 Delete user: deleted"
```

#### Test 8: Error Handling

```bash
# Missing auth header
STATUS=$(curl -s -o /dev/null -w "%{http_code}" $API_URL/api/messages)
echo "8.1 Missing auth: $STATUS (expected 401)"

# Invalid token
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer invalid_token" $API_URL/api/messages)
echo "8.2 Invalid token: $STATUS (expected 401)"

# Non-existent resource
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $TOKEN" $API_URL/api/messages/nonexistent-id)
echo "8.3 Not found: $STATUS (expected 404)"

# Invalid message body
STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"invalid": "body"}' $API_URL/api/messages)
echo "8.4 Invalid body: $STATUS (expected 400 or 422)"
```

### 4.3 Message Flow Test (End-to-End)

This test verifies the complete message lifecycle from API submission through BACEN Gateway to simulator response.

**Prerequisites:** Simulator running with `normal_operations` scenario.

```bash
# Ensure simulator is in normal mode
curl -s -X POST http://localhost:4001/api/scenarios/normal_operations/activate

# Reset simulator statistics
curl -s -X POST http://localhost:4001/api/statistics/reset
```

**Step 1: Submit a STR0004 message**

```bash
export TOKEN=$(curl -s -X POST $API_URL/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@monetarie.com.br","password":"Monetarie#Adm@2026"}' | jq -r '.token')

MSG_ID=$(curl -s -X POST $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message_type": "STR0004",
    "xml_content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?><BCMSG xmlns=\"http://www.bcb.gov.br/SPB\"><IdentdEmissor>12345678</IdentdEmissor><IdentdDesworko>00038166</IdentdDesworko><SISMSG><CodMsg>STR0004</CodMsg><NumCtrlIF>E2ETSRT2026020700001</NumCtrlIF><DtHrMsg>2026-02-07T12:00:00</DtHrMsg><DOC xmlns=\"http://www.bcb.gov.br/SPB/STR0004\"><InfTransf><ISPBIfDeworko>00038166</ISPBIfDeworko><VlrTransf>100000.00</VlrTransf><FinlddSTR>01</FinlddSTR><DtMovto>2026-02-07</DtMovto></InfTransf></DOC></SISMSG></BCMSG>",
    "destination_ispb": "00038166"
  }' | jq -r '.id')

echo "Step 1: Message submitted. ID=$MSG_ID"
```

**Step 2: Poll for state transitions**

```bash
# Wait 1 second for processing
sleep 1

# Check current status
for i in 1 2 3 4 5; do
  STATUS=$(curl -s $API_URL/api/messages/$MSG_ID \
    -H "Authorization: Bearer $TOKEN" | jq -r '.status')
  echo "Step 2 (poll $i): Status=$STATUS"

  if [ "$STATUS" = "confirmed" ] || [ "$STATUS" = "failed" ]; then
    break
  fi
  sleep 1
done
```

**Step 3: Verify complete state history**

```bash
curl -s $API_URL/api/messages/$MSG_ID \
  -H "Authorization: Bearer $TOKEN" | jq '.state_history'
```

Expected state history:

```json
[
  {"status": "pending", "timestamp": "...", "detail": "Message accepted"},
  {"status": "processing", "timestamp": "...", "detail": "Signing and routing"},
  {"status": "sent", "timestamp": "...", "detail": "Sent via SPB01 domain"},
  {"status": "confirmed", "timestamp": "...", "detail": "BACEN confirmation received"}
]
```

**Step 4: Verify simulator statistics**

```bash
curl -s http://localhost:4001/api/statistics | jq
```

Expected: `total_requests` incremented by 1, `successful_responses` incremented by 1.

**Step 5: Verify NATS events (if NATS CLI available)**

```bash
# Subscribe to messages.> in a separate terminal first
nats sub "messages.>" --server nats://nats.spb.svc.cluster.local:4222

# Then submit a message (from another terminal)
# Expected events: messages.new, messages.processing, messages.sent, messages.confirmed
```

### 4.4 IBM MQ Connection Test

This test verifies that the DomainManager connects to all 5 BACEN MQ domains.

**Step 1: Check DomainManager status via BACEN Gateway logs**

```bash
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=200 | \
  grep "DomainManager"
```

Expected log entries for each domain:

```
[DomainManager] Initialized with 5 domains
[DomainManager] Connecting domain spb01 to QM.12345678.01:1414
[DomainManager] Domain spb01 connected successfully (send: QL.REQ.00038166.12345678.01, recv: QL.RSP.00038166.12345678.01)
[DomainManager] Connecting domain spb02 to QM.12345678.05:15422
[DomainManager] Domain spb02 connected successfully ...
[DomainManager] Connecting domain mes01 to QM.12345678.02:12422
[DomainManager] Domain mes01 connected successfully ...
[DomainManager] Connecting domain mes02 to QM.12345678.03:13422
[DomainManager] Domain mes02 connected successfully ...
[DomainManager] Connecting domain mes03 to QM.12345678.04:14422
[DomainManager] Domain mes03 connected successfully ...
```

**Step 2: Send GEN0001 connectivity test**

GEN0001 is the standard BACEN connection test. It routes through SPB01 (Platina priority).

```bash
MSG_ID=$(curl -s -X POST $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message_type": "GEN0001",
    "xml_content": "<?xml version=\"1.0\"?><BCMSG xmlns=\"http://www.bcb.gov.br/SPB\"><IdentdEmissor>12345678</IdentdEmissor><IdentdDesworko>00038166</IdentdDesworko><SISMSG><CodMsg>GEN0001</CodMsg><DtHrMsg>2026-02-07T10:00:00</DtHrMsg><DOC xmlns=\"http://www.bcb.gov.br/SPB/GEN0001\"><InfConect><DtHrBC>2026-02-07T10:00:00</DtHrBC></InfConect></DOC></SISMSG></BCMSG>",
    "destination_ispb": "00038166"
  }' | jq -r '.id')

echo "GEN0001 sent: $MSG_ID"
sleep 2

# Verify GEN0001E response
curl -s "$API_URL/api/messages?message_type=GEN0001E" \
  -H "Authorization: Bearer $TOKEN" | jq '.data[0].status'
# Expected: "confirmed"
```

**Step 3: Per-domain validation**

Send one message per domain to verify all 5 connections:

```bash
# SPB01 - STR (Platina)
curl -s -X POST $API_URL/api/messages -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"STR0004","xml_content":"<BCMSG><SISMSG><CodMsg>STR0004</CodMsg></SISMSG></BCMSG>","destination_ispb":"00038166"}' | jq -r '.domain'
# Expected: "spb01"

# SPB02 - SEL (Ouro)
curl -s -X POST $API_URL/api/messages -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"SEL1020","xml_content":"<BCMSG><SISMSG><CodMsg>SEL1020</CodMsg></SISMSG></BCMSG>","destination_ispb":"00038166"}' | jq -r '.domain'
# Expected: "spb02"

# MES02 - CCS (Prata)
curl -s -X POST $API_URL/api/messages -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"CCS0001","xml_content":"<BCMSG><SISMSG><CodMsg>CCS0001</CodMsg></SISMSG></BCMSG>","destination_ispb":"00038166"}' | jq -r '.domain'
# Expected: "mes02"

# MES03 - PAG (Bronze)
curl -s -X POST $API_URL/api/messages -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"PAG0101","xml_content":"<BCMSG><SISMSG><CodMsg>PAG0101</CodMsg></SISMSG></BCMSG>","destination_ispb":"00038166"}' | jq -r '.domain'
# Expected: "mes03"

# MES03 - CAM (Bronze)
curl -s -X POST $API_URL/api/messages -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"CAM0001","xml_content":"<BCMSG><SISMSG><CodMsg>CAM0001</CodMsg></SISMSG></BCMSG>","destination_ispb":"00038166"}' | jq -r '.domain'
# Expected: "mes03"
```

### 4.5 Stress Test

**Prerequisites:** Simulator running with `stress_test` scenario.

**Step 1: Activate stress test scenario**

```bash
curl -s -X POST http://localhost:4001/api/scenarios/stress_test/activate
curl -s -X POST http://localhost:4001/api/statistics/reset
```

**Step 2: Send 1,000 messages**

```bash
#!/bin/bash
# stress_test.sh - Send 1000 messages with mixed types
API_URL="https://spbapi-dev.fluxiq.com.br"
TOKEN=$(curl -s -X POST $API_URL/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@monetarie.com.br","password":"Monetarie#Adm@2026"}' | jq -r '.token')

TYPES=("STR0004" "PAG0101" "SEL1020" "LDL0001" "CAM0001" "CIR0003" "GEN0001" "DDA0001" "CTP0001" "LPI0001")
SUCCESS=0
FAIL=0
START=$(date +%s)

for i in $(seq 1 1000); do
  TYPE=${TYPES[$((RANDOM % ${#TYPES[@]}))]}
  HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST $API_URL/api/messages \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"message_type\":\"$TYPE\",\"xml_content\":\"<BCMSG><SISMSG><CodMsg>$TYPE</CodMsg></SISMSG></BCMSG>\",\"destination_ispb\":\"00038166\"}")

  if [ "$HTTP_CODE" = "201" ]; then
    ((SUCCESS++))
  else
    ((FAIL++))
  fi

  # Progress every 100 messages
  if [ $((i % 100)) -eq 0 ]; then
    echo "Progress: $i/1000 (success=$SUCCESS, fail=$FAIL)"
  fi
done

END=$(date +%s)
DURATION=$((END - START))
RATE=$(echo "scale=1; 1000 / $DURATION" | bc)

echo ""
echo "=== Stress Test Results ==="
echo "Total messages: 1000"
echo "Successful: $SUCCESS"
echo "Failed: $FAIL"
echo "Duration: ${DURATION}s"
echo "Throughput: ${RATE} msg/sec"
```

**Step 3: Verify results**

```bash
# Check simulator statistics
curl -s http://localhost:4001/api/statistics | jq

# Check for lost messages (API count vs simulator count should match)
API_COUNT=$(curl -s "$API_URL/api/dashboard/stats" \
  -H "Authorization: Bearer $TOKEN" | jq '.total_messages')
SIM_COUNT=$(curl -s http://localhost:4001/api/statistics | jq '.total_requests')
echo "API count: $API_COUNT, Simulator count: $SIM_COUNT"
```

**Pass criteria:**

- Throughput: > 10 msg/sec
- Success rate: > 90% (consistent with scenario's 95% rate)
- No lost messages (API count matches simulator count)
- No OOM or pod restarts

### 4.6 Failure Recovery Test

These tests validate system behavior when individual components fail.

#### Test 6.1: BACEN Gateway Restart

```bash
# Kill the BACEN Gateway pod
kubectl delete pod -n spb -l app=bacen-gateway

# Wait for pod to restart
kubectl wait --for=condition=ready pod -n spb -l app=bacen-gateway --timeout=120s

# Verify DomainManager reconnects
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=50 | grep "DomainManager"
# Expected: "Initialized with 5 domains", followed by connection messages

# Send a test message to verify functionality
curl -s -X POST $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"GEN0001","xml_content":"<BCMSG><SISMSG><CodMsg>GEN0001</CodMsg></SISMSG></BCMSG>","destination_ispb":"00038166"}' | jq '.status'
# Expected: "pending"
```

#### Test 6.2: NATS Restart

```bash
# Kill NATS pod
kubectl delete pod -n spb -l app=nats

# Wait for NATS to come back
kubectl wait --for=condition=ready pod -n spb -l app=nats --timeout=120s

# Verify JetStream is operational
kubectl exec -n spb deployment/nats -- nats-server --version

# Send test message and verify NATS events still flow
# (JetStream persists messages to disk, so no data loss expected)
```

#### Test 6.3: Database Connection Loss

```bash
# Kill Cloud SQL Proxy sidecar (simulates DB loss)
kubectl exec -n spb deployment/bacen-gateway -c cloud-sql-proxy -- kill 1

# Wait for sidecar to restart
sleep 10

# Verify error handling - messages should return 503 during outage
curl -s -X POST $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"GEN0001","xml_content":"<test/>","destination_ispb":"00038166"}' | jq

# After sidecar restarts, verify recovery
sleep 30
curl -s $API_URL/api/health | jq '.services.database'
# Expected: "connected"
```

#### Test 6.4: IBM MQ Disconnection

```bash
# Scale down IBM MQ
kubectl scale deployment ibmmq -n spb --replicas=0

# Wait and check DomainManager logs
sleep 10
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=20 | grep "DomainManager"
# Expected: Connection closed messages, scheduled reconnect messages

# Messages sent during outage should queue or return error
curl -s -X POST $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"STR0004","xml_content":"<test/>","destination_ispb":"00038166"}' | jq '.error'

# Scale MQ back up
kubectl scale deployment ibmmq -n spb --replicas=1
sleep 15

# Verify auto-reconnect
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=20 | grep "connected successfully"
# Expected: All 5 domains reconnect automatically
```

### 4.7 Security Test

#### Test 7.1: JWT Token Tests

```bash
# Expired token (manually craft one with past expiry)
curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1MDAwMDAwMDB9.invalid" \
  $API_URL/api/messages
# Expected: 401

# Missing Bearer prefix
curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: $TOKEN" \
  $API_URL/api/messages
# Expected: 401

# No authorization header at all
curl -s -o /dev/null -w "%{http_code}" $API_URL/api/messages
# Expected: 401

# Malformed token
curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer not.a.jwt" \
  $API_URL/api/messages
# Expected: 401
```

#### Test 7.2: Input Injection

```bash
# SQL injection attempt
curl -s -X POST $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"STR0004; DROP TABLE bacen_messages;--","xml_content":"test","destination_ispb":"00038166"}' | jq
# Expected: 400 or 422 (validation error, not SQL execution)

# XSS attempt in message content
curl -s -X POST $API_URL/api/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message_type":"STR0004","xml_content":"<script>alert(1)</script>","destination_ispb":"00038166"}' | jq
# Expected: 400 or 422 (invalid XML)

# Path traversal
curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $TOKEN" \
  "$API_URL/api/messages/../../etc/passwd"
# Expected: 404 (not 200)
```

#### Test 7.3: Audit Trail Verification

```bash
# Perform several actions, then verify audit records exist
# (Check database directly)
kubectl exec -n spb deployment/bacen-gateway -c bacen-gateway -- \
  /app/bin/bacen_gateway rpc "BacenGateway.Repo.query(\"SELECT COUNT(*) FROM audit_logs WHERE inserted_at > now() - interval '1 hour'\")"
# Expected: Non-zero count
```

---

## 5. Frontend Testing

### 5.1 Admin Portal Test Matrix

The admin portal is accessible at `/admin` after login with an admin role account.

| Route | View | Test Steps | Expected Result |
|-------|------|-----------|----------------|
| /admin/dashboard | Dashboard | Login as admin, navigate to dashboard | Statistics panels load, charts render, recent activity shown |
| /admin/messages | Messages List | Click "Mensagens" in sidebar | Paginated message list with filters (status, type, date) |
| /admin/messages/:id | Message Detail | Click any message row | Full message detail with XML content, state history |
| /admin/payments | Payments | Click "Pagamentos" in sidebar | Payment list with type filter (STR/TED/DOC) |
| /admin/users | Users Management | Click "Usuarios" in sidebar | User list with create/edit/delete actions |
| /admin/users/new | Create User | Click "Novo Usuario" | Form with name, email, role, password fields |
| /admin/users/:id | Edit User | Click edit on any user | Pre-filled form, save updates |
| /admin/groups | Groups Management | Click "Grupos" in sidebar | Group list with CRUD operations |
| /admin/settings | System Settings | Click "Configuracoes" in sidebar | System configuration panels |
| /admin/system-date | System Date | Navigate to System Date | Current BACEN system date display and override |
| /admin/permissions | Permission Tree | Navigate to Permissions | Hierarchical permission tree with module/functionality grouping |
| /admin/balances | Account Balances | Navigate to Balances | STR account balance display, domain connection status |
| /admin/entity-status | Entity Status | Navigate to Entity Status | Financial institution status board |
| /admin/schedule | Schedule/Calendar | Navigate to Schedule | BACEN calendar with holidays, operating windows |
| /admin/integration | Integration Config | Navigate to Integration | MQ domain configuration, NATS settings |
| /admin/alcadas | Alcadas (Approval Tiers) | Navigate to Alcadas | Approval tier configuration by operation type |
| /admin/vistos | Vistos (Approvals) | Navigate to Vistos | Pending approval queue, approve/reject actions |
| /admin/company | Company Management | Navigate to Company | Institution profile, ISPB, CNPJ |
| /admin/branches | Branches | Navigate to Branches | Branch office management |
| /admin/queues | Queue Management | Navigate to Queues | MQ queue monitoring, depth, DLQ status |
| /admin/integrations | External Integrations | Navigate to Integrations | BACEN connection status, webhook config |
| /admin/certificates | Certificates | Navigate to Certificates | ICP-Brasil certificate management, expiry alerts |
| /admin/connected-users | Connected Users | Navigate to Connected Users | Active session list, force logout capability |
| /admin/pending-ops | Pending Operations | Navigate to Pending Ops | Operations awaiting approval or processing |
| /admin/compliance | Compliance | Navigate to Compliance | Audit reports, regulatory compliance status |
| /admin/accounting | Chart of Accounts | Navigate to Accounting | Chart of accounts tree view |
| /admin/accounting/history | Account History | Navigate to Account History | Transaction history with date range filter |
| /admin/accounting/cost-centers | Cost Centers | Navigate to Cost Centers | Cost center management |
| /admin/accounting/events | Accounting Events | Navigate to Events | Accounting event log |
| /admin/reports | Reports | Navigate to Reports | Report generation (18+ types) |
| /admin/exports | Exports | Navigate to Exports | Data export (5 formats: CSV, XLSX, PDF, XML, JSON) |
| /admin/tariffs | Tariff Management | Navigate to Tariffs | Tariff configuration by message type |
| /admin/alerts | Alert Configuration | Navigate to Alerts | Alert rules and notification settings |
| /admin/monitors | System Monitors | Navigate to Monitors | Real-time system health monitors |
| /admin/registration | Registration | Navigate to Registration | Participant registration management |

### 5.2 Operator Portal Test Matrix

The operator portal is accessible at `/operator` after login with an operator role account.

| Route | View | Test Steps | Expected Result |
|-------|------|-----------|----------------|
| /operator/dashboard | Operator Dashboard | Login as operator | Operational statistics, daily summary |
| /operator/messages | Message Operations | Click "Mensagens" | Message list with send/receive capabilities |
| /operator/messages/new | Send Message | Click "Nova Mensagem" | Message composition form with template selection |
| /operator/payments | Payment Operations | Click "Pagamentos" | Payment initiation and tracking |
| /operator/payments/new | New Payment | Click "Novo Pagamento" | Payment form (STR/TED/DOC) |
| /operator/transactions | Transactions | Click "Transacoes" | Transaction list with search |
| /operator/monitor | Operations Monitor | Click "Monitor" | Real-time operations dashboard |
| /operator/schedule | Operating Schedule | Click "Agenda" | BACEN operating hours, window status |
| /operator/config/message-routing | Message Routing Config | Navigate to routing | Message routing rules by type/domain |
| /operator/config/templates | Template Config | Navigate to templates | Message template management |
| /operator/config/notifications | Notification Config | Navigate to notifications | Personal notification preferences |
| /operator/config/display | Display Config | Navigate to display | UI preferences, language, timezone |
| /operator/tools/xml-viewer | XML Viewer | Navigate to XML viewer | XML message viewer/formatter |
| /operator/tools/message-search | Message Search | Navigate to search | Advanced message search with filters |
| /operator/monitors/integration | Integration Monitor | Navigate to integration monitor | MQ domain status, throughput graphs |
| /operator/monitors/files | File Monitor | Navigate to file monitor | File processing status |
| /operator/monitors/tariffs | Tariff Monitor | Navigate to tariff monitor | Tariff application monitoring |

### 5.3 Authentication Tests

| Test Case | Steps | Expected Result |
|-----------|-------|----------------|
| Valid admin login | Enter admin@monetarie.com.br / Monetarie#Adm@2026, click Login | Redirect to /admin/dashboard |
| Valid operator login | Enter operator credentials, click Login | Redirect to /operator/dashboard |
| Invalid password | Enter admin@monetarie.com.br / wrongpassword | Error message: "Credenciais invalidas" |
| Invalid email | Enter nobody@fluxiq.com / Monetarie#Adm@2026 | Error message: "Credenciais invalidas" |
| Empty fields | Click Login with empty email/password | Client-side validation error |
| SSO redirect | Click "SSO Login" button | Redirect to SSO provider |
| Token expiry | Wait 24 hours (or manipulate token) | Auto-redirect to login page |
| Role-based redirect | Login as operator, try /admin/users | Redirect to /operator/dashboard or 403 |
| Logout | Click user menu -> "Sair" | Redirect to login, token cleared |

### 5.4 Cross-Browser Testing

| Browser | Version | Platform | Status |
|---------|---------|----------|--------|
| Chrome | 120+ | Windows/Mac/Linux | Primary |
| Firefox | 120+ | Windows/Mac/Linux | Supported |
| Safari | 17+ | macOS/iOS | Supported |
| Edge | 120+ | Windows | Supported |

### 5.5 Responsive Design Testing

| Viewport | Width | Priority Views to Test |
|----------|-------|----------------------|
| Desktop | 1920px | All views |
| Laptop | 1366px | Dashboard, Messages, Payments |
| Tablet | 768px | Dashboard, Messages (sidebar collapses) |
| Mobile | 375px | Login, Dashboard (limited functionality) |

---

## 6. Database Verification

### 6.1 Migration Verification

Run all database migrations and verify they complete without errors.

```bash
# From the BACEN Gateway service directory
cd /Users/luizpenha/monetarie/spb/services/bacen_gateway

# Run migrations
mix ecto.migrate

# Expected output:
# [info] == Running 20260130 BacenGateway.Repo.Migrations.CreateDlqEntries.change/0
# [info] == Migrated 20260130 in 0.0s
# [info] == Running 20260201 BacenGateway.Repo.Migrations.CreateBacenErrorCodes.change/0
# [info] == Migrated 20260201 in 0.0s
# [info] == Running 20260201 BacenGateway.Repo.Migrations.CreateBacenMessages.change/0
# [info] == Migrated 20260201 in 0.0s
# [info] == Running 20260202 BacenGateway.Repo.Migrations.AddBacenMessageFields.change/0
# [info] == Migrated 20260202 in 0.0s
# [info] == Running 20260202 BacenGateway.Repo.Migrations.CreateFinancialInstitutions.change/0
# [info] == Migrated 20260202 in 0.0s
# [info] == Running 20260205 BacenGateway.Repo.Migrations.CreateMessageTemplates.change/0
# [info] == Migrated 20260205 in 0.0s

# Verify migration status
mix ecto.migrations
# All migrations should show "up"
```

### 6.2 Seed Data Verification

Run all seed files and verify data loads correctly.

```bash
cd /Users/luizpenha/monetarie/spb/services/bacen_gateway

# Run the master seed runner
mix run priv/repo/seeds/run_all_seeds.exs

# This executes seeds in order:
# 001_admin_setup.exs          - Admin users and roles
# 002_financial_entities.exs   - Financial institution base data
# 004_payment_requests.exs     - Sample payment data
# 005_bacen_messages.exs       - Sample BACEN messages
# 006_message_templates.exs    - Message template stubs
# 010_bacen_reference_data.exs - 100 ISPBs, 530+ domains, 1222 templates, 50 error codes
# 011_transaction_seeds.exs    - Sample transactions
# 012_legacy_ispb_domains.exs  - 494 ISPBs, 7541 domains from legacy
# 013_message_catalog.exs      - Full 1222 message types, 10000 realistic messages
# 014_operational_data.exs     - Operational reference data
# 020_security_seeds.exs       - Security users, groups, modules
# 021_message_group_permissions.exs - Message group access control
# 022_alcadas_vistos_seeds.exs - Approval tiers and approval records
# 023_schedule_holidays_seeds.exs   - BACEN calendar and holidays
# 024_external_entities_seeds.exs   - External institution registry
# 025_accounting_seeds.exs     - Chart of accounts, cost centers

# Expected: All seeds run without errors, final summary shows record counts
```

### 6.3 Data Integrity Checks

Run these SQL queries to verify critical data is present.

#### Check 1: Message Types

```sql
-- Should return 1222
SELECT COUNT(*) FROM message_templates;

-- Should return 36 categories
SELECT COUNT(DISTINCT category) FROM message_templates;

-- Top categories by count
SELECT category, COUNT(*) as cnt
FROM message_templates
GROUP BY category
ORDER BY cnt DESC
LIMIT 10;
-- Expected: CAM ~111, CIR ~100, SEL ~92, STR ~77, LDL ~71, CTP ~90, PAG ~60, DDA ~56, etc.
```

#### Check 2: Financial Institutions

```sql
-- Should return 494+ institutions
SELECT COUNT(*) FROM bacen_ispbs;

-- Monetarie must exist
SELECT * FROM bacen_ispbs WHERE ispb = '12345678';
-- Expected: 1 row with name containing 'MONETARIE'

-- BACEN must exist
SELECT * FROM bacen_ispbs WHERE ispb = '00000000';
-- Expected: 1 row with name 'Banco Central do Brasil'

-- Major banks present
SELECT ispb, short_name FROM bacen_ispbs
WHERE ispb IN ('00038166', '60701190', '00360305', '90400888', '60746948')
ORDER BY ispb;
-- Expected: BB, Itau, Caixa, Santander, Bradesco
```

#### Check 3: Domain Values

```sql
-- Should return 7500+ domain values
SELECT COUNT(*) FROM bacen_domains;

-- Priority domain tags
SELECT domain_tag, COUNT(*) as cnt
FROM bacen_domains
GROUP BY domain_tag
ORDER BY cnt DESC
LIMIT 15;
-- Expected: CanPgto, TpPessoa, TpCt, SitLancSTR, CodMoeda, etc.
```

#### Check 4: Error Codes

```sql
-- Should return 95+ error codes (from bacen_error_codes seed)
SELECT COUNT(*) FROM bacen_error_codes;

-- Verify error categories
SELECT category, COUNT(*) as cnt
FROM bacen_error_codes
GROUP BY category
ORDER BY cnt DESC;
-- Expected: validation ~20, business_logic ~20, system ~15, network ~10, timeout ~10, etc.
```

#### Check 5: Admin User

```sql
-- Admin user must exist
SELECT id, email, role FROM users
WHERE email = 'admin@monetarie.com.br';
-- Expected: 1 row with role 'admin'
```

#### Check 6: Message Data

```sql
-- Check message volume from seeds
SELECT COUNT(*) FROM bacen_messages;
-- Expected: 10000+ (from 013_message_catalog.exs)

-- Check distribution by status
SELECT status, COUNT(*) FROM bacen_messages GROUP BY status;
-- Expected: sent ~25%, received ~25%, acknowledged ~20%, confirmed ~20%, failed ~10%

-- Check distribution by category
SELECT SUBSTRING(message_type FROM 1 FOR 3) as cat, COUNT(*) as cnt
FROM bacen_messages
GROUP BY cat
ORDER BY cnt DESC
LIMIT 10;
```

### 6.4 Database Migration Quick Test (Kubernetes)

```bash
# Run migration as a Kubernetes Job
kubectl apply -f /Users/luizpenha/monetarie/spb/k8s/production/complete-migration-and-seed-job.yaml

# Watch job progress
kubectl logs -n spb job/complete-migration-and-seed -f

# Verify job completed
kubectl get jobs -n spb complete-migration-and-seed
# Expected: COMPLETIONS 1/1
```

---

## 7. Monitoring Verification

### 7.1 Health Check Matrix

Verify all service health endpoints are responding correctly.

| Service | Endpoint | Method | Expected Status | Expected Body |
|---------|----------|--------|-----------------|---------------|
| API Gateway | /api/health | GET | 200 OK | `{"status": "healthy"}` |
| API Gateway | /metrics | GET | 200 OK | Prometheus text format |
| Auth Service | /health | GET | 200 OK | `{"status": "ok"}` |
| Auth Service | /ready | GET | 200 OK | `{"ready": true}` |
| BACEN Gateway | /health | GET | 200 OK | `{"status": "ok"}` |
| BACEN Gateway | /ready | GET | 200 OK | `{"ready": true}` |
| BACEN Gateway | /api/health | GET | 200 OK | `{"status": "ok"}` |
| BACEN Gateway | /api/ready | GET | 200 OK | `{"ready": true}` |
| Transaction Service | /health | GET | 200 OK | `{"status": "ok"}` |
| Securities Service | /health | GET | 200 OK | `{"status": "ok"}` |
| Settlement Service | /health | GET | 200 OK | `{"status": "ok"}` |
| Forex Service | /health | GET | 200 OK | `{"status": "ok"}` |
| Cash Service | /health | GET | 200 OK | `{"status": "ok"}` |
| Extract Service | /health | GET | 200 OK | `{"status": "ok"}` |
| User Management | /health | GET | 200 OK | `{"status": "ok"}` |
| Message Processor | /health | GET | 200 OK | `{"status": "ok"}` |
| BACEN Simulator | /api/health | GET | 200 OK | `{"status": "ok"}` |
| NATS | /healthz | GET (port 8222) | 200 OK | `{"status": "ok"}` |

**Automated health check script:**

```bash
#!/bin/bash
# health_check.sh - Verify all service health endpoints

NAMESPACE="spb"
PASS=0
FAIL=0

check_health() {
  SERVICE=$1
  URL=$2
  EXPECTED=$3

  STATUS=$(kubectl exec -n $NAMESPACE deployment/api-gateway -c api-gateway -- \
    curl -s -o /dev/null -w "%{http_code}" "$URL" 2>/dev/null)

  if [ "$STATUS" = "$EXPECTED" ]; then
    echo "PASS: $SERVICE ($URL) -> $STATUS"
    ((PASS++))
  else
    echo "FAIL: $SERVICE ($URL) -> $STATUS (expected $EXPECTED)"
    ((FAIL++))
  fi
}

check_health "API Gateway" "http://api-gateway:4000/api/health" "200"
check_health "Auth Service" "http://auth-service:4001/health" "200"
check_health "BACEN Gateway" "http://bacen-gateway:4002/health" "200"
check_health "Message Processor" "http://message-processor:4003/health" "200"
check_health "User Management" "http://user-management:4004/health" "200"
check_health "Transaction Service" "http://transaction-service:4005/health" "200"
check_health "Securities Service" "http://securities-service:4006/health" "200"
check_health "Settlement Service" "http://settlement-service:4007/health" "200"
check_health "Forex Service" "http://forex-service:4008/health" "200"
check_health "Cash Service" "http://cash-service:4009/health" "200"
check_health "Extract Service" "http://extract-service:4010/health" "200"
check_health "NATS" "http://nats:8222/healthz" "200"

echo ""
echo "=== Results ==="
echo "PASS: $PASS"
echo "FAIL: $FAIL"
echo "TOTAL: $((PASS + FAIL))"
```

### 7.2 Prometheus Metrics Verification

Verify that Prometheus is scraping metrics from all services.

**Step 1: Check metrics endpoint**

```bash
curl -s https://spbapi-dev.fluxiq.com.br/metrics | head -20
```

Expected: Prometheus-format text output with metrics like:

```
# HELP http_request_duration_seconds HTTP request duration in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="GET",path="/api/health",le="0.01"} 1523
```

**Step 2: Check Grafana dashboards**

1. Navigate to https://monitoring-dev.fluxiq.com.br
2. Login: admin / Monetarie#Adm@2026
3. Verify the following dashboards exist and show data:

| Dashboard | Panels to Verify |
|-----------|-----------------|
| SPB Overview | Total messages, success rate, active domains |
| Message Throughput | Messages/minute, by category, by domain |
| Error Analysis | Error rate, errors by code, retry success rate |
| Domain Health | Per-domain connection status, queue depth |
| Service Health | Per-service CPU, memory, request latency |

### 7.3 NATS JetStream Verification

```bash
# Check NATS server info
kubectl exec -n spb deployment/nats -- nats-server --version

# Check NATS monitoring endpoint
kubectl exec -n spb deployment/nats -- curl -s http://localhost:8222/varz | python3 -m json.tool

# Check JetStream status
kubectl exec -n spb deployment/nats -- curl -s http://localhost:8222/jsz | python3 -m json.tool

# Verify streams exist
kubectl exec -n spb deployment/nats -- curl -s http://localhost:8222/jsz | python3 -c "
import json,sys
data=json.load(sys.stdin)
print(f'Streams: {data.get(\"streams\", 0)}')
print(f'Messages: {data.get(\"messages\", 0)}')
print(f'Bytes: {data.get(\"bytes\", 0)}')
"
```

---

## 8. Deployment Verification Checklist

Use this checklist after every deployment to verify the system is fully operational.

### 8.1 Infrastructure

- [ ] All pods running: `kubectl get pods -n spb` (all STATUS: Running, READY: N/N)
- [ ] No pod restarts: Check RESTARTS column (should be 0 for recent deploy)
- [ ] All services exist: `kubectl get services -n spb`
- [ ] Ingress configured: `kubectl get ingress -n spb`
- [ ] Network policies applied: `kubectl get networkpolicies -n spb`
- [ ] Secrets created: `kubectl get secrets -n spb`
- [ ] ConfigMaps created: `kubectl get configmaps -n spb`

### 8.2 Service Health

- [ ] API Gateway health: `curl -s https://spbapi-dev.fluxiq.com.br/api/health` returns 200
- [ ] Auth Service health: Internal /health returns 200
- [ ] BACEN Gateway health: Internal /health returns 200
- [ ] All 11 services healthy (run health check script from Section 7.1)
- [ ] NATS JetStream operational: /healthz returns 200
- [ ] IBM MQ connected: DomainManager logs show 5 domains connected

### 8.3 Frontend

- [ ] Frontend accessible: https://spbadmin-dev.fluxiq.com.br loads
- [ ] Login works: admin / Monetarie#Adm@2026 authenticates successfully
- [ ] Dashboard renders: Statistics panels and charts display
- [ ] Navigation works: All sidebar links load correctly
- [ ] API calls succeed: Network tab shows 200 responses

### 8.4 Data

- [ ] Database migrations applied: `mix ecto.migrations` shows all "up"
- [ ] Seed data loaded: 1,222 message types, 494+ ISPBs, 7,541+ domains
- [ ] Admin user exists: admin@monetarie.com.br can login
- [ ] Message templates available: `/api/message-templates` returns data

### 8.5 Integration

- [ ] Message creation works: POST /api/messages returns 201
- [ ] Message listing works: GET /api/messages returns data
- [ ] Simulator responds: Simulator health check passes
- [ ] Metrics endpoint active: GET /metrics returns Prometheus data
- [ ] CORS configured: Frontend API calls do not get CORS errors

### 8.6 Quick Verification Script

```bash
#!/bin/bash
# deploy_verify.sh - Post-deployment verification
set -e

API_URL="https://spbapi-dev.fluxiq.com.br"
FRONTEND_URL="https://spbadmin-dev.fluxiq.com.br"
PASS=0
FAIL=0

check() {
  DESC=$1
  RESULT=$2
  EXPECTED=$3

  if [ "$RESULT" = "$EXPECTED" ]; then
    echo "PASS: $DESC"
    ((PASS++))
  else
    echo "FAIL: $DESC (got: $RESULT, expected: $EXPECTED)"
    ((FAIL++))
  fi
}

echo "=== Post-Deployment Verification ==="
echo ""

# Infrastructure
echo "--- Infrastructure ---"
POD_COUNT=$(kubectl get pods -n spb --field-selector=status.phase=Running --no-headers 2>/dev/null | wc -l | tr -d ' ')
check "Pods running" "$((POD_COUNT > 5 ? 1 : 0))" "1"

# Health checks
echo "--- Service Health ---"
check "API Gateway health" "$(curl -s -o /dev/null -w '%{http_code}' $API_URL/api/health)" "200"
check "Frontend accessible" "$(curl -s -o /dev/null -w '%{http_code}' $FRONTEND_URL)" "200"
check "Metrics endpoint" "$(curl -s -o /dev/null -w '%{http_code}' $API_URL/metrics)" "200"

# Authentication
echo "--- Authentication ---"
TOKEN=$(curl -s -X POST $API_URL/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@monetarie.com.br","password":"Monetarie#Adm@2026"}' | jq -r '.token // empty')
check "Admin login" "$((${#TOKEN} > 10 ? 1 : 0))" "1"

# API functionality
echo "--- API Functionality ---"
if [ -n "$TOKEN" ]; then
  MSG_STATUS=$(curl -s -o /dev/null -w '%{http_code}' -X POST $API_URL/api/messages \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"message_type":"GEN0001","xml_content":"<BCMSG><SISMSG><CodMsg>GEN0001</CodMsg></SISMSG></BCMSG>","destination_ispb":"00038166"}')
  check "Message creation" "$MSG_STATUS" "201"

  LIST_STATUS=$(curl -s -o /dev/null -w '%{http_code}' \
    -H "Authorization: Bearer $TOKEN" $API_URL/api/messages)
  check "Message listing" "$LIST_STATUS" "200"

  DASH_STATUS=$(curl -s -o /dev/null -w '%{http_code}' \
    -H "Authorization: Bearer $TOKEN" $API_URL/api/dashboard/stats)
  check "Dashboard stats" "$DASH_STATUS" "200"
fi

echo ""
echo "=== Summary ==="
echo "PASS: $PASS"
echo "FAIL: $FAIL"
echo "TOTAL: $((PASS + FAIL))"

if [ "$FAIL" -gt 0 ]; then
  echo "STATUS: DEPLOYMENT VERIFICATION FAILED"
  exit 1
else
  echo "STATUS: DEPLOYMENT VERIFIED SUCCESSFULLY"
  exit 0
fi
```

---

## 9. Troubleshooting

### 9.1 Common Issues

| Issue | Symptoms | Likely Cause | Resolution |
|-------|----------|--------------|------------|
| Login returns 401 | "Unauthorized" on valid credentials | JWT_SECRET mismatch between API Gateway and Auth Service | Verify `MONETARIE_SSO_SECRET` is the same across all services. Check `api-gateway-secrets` and `auth-service-secrets` in K8s. |
| Frontend shows blank page | White screen, no content | Build error or nginx misconfiguration | Check frontend pod logs: `kubectl logs -n spb deployment/frontend --tail=50`. Verify nginx.conf serves index.html for all routes. |
| Messages stuck in "pending" | Messages never transition to "processing" | BACEN Gateway not running or MQ disconnected | Check BACEN Gateway pod: `kubectl get pods -n spb -l app=bacen-gateway`. Check DomainManager logs for connection errors. |
| No BACEN responses | Messages go to "sent" but never "confirmed" | Simulator disabled or MQ disconnected | Enable simulator: `kubectl scale deployment bacen-simulator -n spb --replicas=1`. Check MQ connectivity. |
| Database connection error | "connection refused" or "timeout" in logs | Cloud SQL Proxy sidecar missing or crashed | Check sidecar container: `kubectl describe pod -n spb <pod-name>`. Verify `db-credentials` secret has correct connection-name. |
| CORS errors in browser | "Access-Control-Allow-Origin" errors | CORS_ORIGINS not set or wrong | Check `CORS_ORIGINS` env var on API Gateway. Must include the frontend URL (comma-separated). |
| Pod CrashLoopBackOff | Pod restarts repeatedly | Missing config, bad image, startup failure | Check logs: `kubectl logs -n spb <pod-name> --previous`. Check events: `kubectl describe pod -n spb <pod-name>`. |
| NATS connection refused | Services cannot publish events | NATS pod down or service name wrong | Check NATS: `kubectl get pods -n spb -l app=nats`. Verify `NATS_URL` in service-config ConfigMap. |
| Metrics not showing | Grafana dashboards empty | Prometheus not scraping | Verify /metrics endpoint. Check ServiceMonitor resources. Verify Prometheus target status. |
| Image pull error | ImagePullBackOff on pods | Wrong image name or tag | Verify image naming: `southamerica-east1-docker.pkg.dev/fluxiqbr/monetarie/spb-{component}:{tag}`. Never use `gcr.io`. |

### 9.2 Log Inspection

#### View Service Logs

```bash
# API Gateway
kubectl logs -n spb deployment/api-gateway -c api-gateway --tail=100 -f

# Auth Service
kubectl logs -n spb deployment/auth-service -c auth-service --tail=100 -f

# BACEN Gateway (main container)
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=100 -f

# BACEN Gateway (Cloud SQL Proxy sidecar)
kubectl logs -n spb deployment/bacen-gateway -c cloud-sql-proxy --tail=100

# Message Processor
kubectl logs -n spb deployment/message-processor -c message-processor --tail=100 -f

# Transaction Service
kubectl logs -n spb deployment/transaction-service -c transaction-service --tail=100 -f

# Frontend (nginx access logs)
kubectl logs -n spb deployment/frontend --tail=100 -f

# BACEN Simulator
kubectl logs -n spb deployment/bacen-simulator --tail=100 -f

# NATS
kubectl logs -n spb deployment/nats --tail=100 -f
```

#### Search Logs for Errors

```bash
# Find all errors across BACEN Gateway
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=1000 | grep -i error

# Find DomainManager events
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=500 | grep "DomainManager"

# Find authentication failures
kubectl logs -n spb deployment/auth-service -c auth-service --tail=500 | grep -i "auth\|login\|fail"

# Find database errors
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=500 | grep -i "ecto\|database\|sql\|postgres"
```

#### View Pod Events

```bash
# Get events for a specific pod
kubectl describe pod -n spb <pod-name> | tail -20

# Get all events in the namespace (sorted by time)
kubectl get events -n spb --sort-by=.metadata.creationTimestamp | tail -20
```

### 9.3 Emergency Procedures

#### Procedure 1: Drain Dead Letter Queue

When messages are accumulating in the DLQ and need to be reprocessed or discarded:

```bash
# Check DLQ depth
kubectl exec -n spb deployment/ibmmq -- \
  bash -c "echo 'DISPLAY QLOCAL(QL.DLQ.12345678) CURDEPTH' | runmqsc QM.12345678.01"

# Option A: Reprocess DLQ messages (move back to source queue)
kubectl exec -n spb deployment/ibmmq -- \
  bash -c "/opt/mqm/bin/dmpmqmsg -m QM.12345678.01 -i QL.DLQ.12345678 -o QL.REQ.00038166.12345678.01"

# Option B: Clear DLQ (discard all messages - DESTRUCTIVE)
kubectl exec -n spb deployment/ibmmq -- \
  bash -c "echo 'CLEAR QLOCAL(QL.DLQ.12345678)' | runmqsc QM.12345678.01"
```

#### Procedure 2: Restart All Services Without Data Loss

```bash
# Rolling restart of all deployments (zero-downtime)
kubectl rollout restart deployment -n spb

# Monitor the rollout
kubectl rollout status deployment/api-gateway -n spb
kubectl rollout status deployment/bacen-gateway -n spb
kubectl rollout status deployment/auth-service -n spb

# Verify all pods are Running
kubectl get pods -n spb
```

#### Procedure 3: Switch to Contingency Mode

When BACEN communication is down and messages need to be queued locally:

```bash
# Step 1: Scale down BACEN Gateway to stop sending
kubectl scale deployment bacen-gateway -n spb --replicas=0

# Step 2: Messages will accumulate in the database with status "pending"

# Step 3: When BACEN is back, scale up
kubectl scale deployment bacen-gateway -n spb --replicas=1

# Step 4: The MessageProcessor will automatically process pending messages
# Verify by checking message counts
curl -s "$API_URL/api/messages?status=pending" \
  -H "Authorization: Bearer $TOKEN" | jq '.meta.total'
```

#### Procedure 4: Force Certificate Rotation

When the ICP-Brasil certificate needs emergency replacement:

```bash
# Step 1: Update the secret with the new certificate
kubectl create secret generic bacen-gateway-secrets -n spb \
  --from-file=icp-brasil-cert=/path/to/new-cert.pem \
  --from-file=icp-brasil-key=/path/to/new-key.pem \
  --dry-run=client -o yaml | kubectl apply -f -

# Step 2: Restart BACEN Gateway to pick up new cert
kubectl rollout restart deployment/bacen-gateway -n spb

# Step 3: Verify new certificate is active
kubectl logs -n spb deployment/bacen-gateway -c bacen-gateway --tail=20 | grep -i cert
```

#### Procedure 5: Database Emergency Recovery

```bash
# Step 1: Check Cloud SQL Proxy status
kubectl logs -n spb deployment/bacen-gateway -c cloud-sql-proxy --tail=20

# Step 2: If proxy is down, restart the pod
kubectl delete pod -n spb -l app=bacen-gateway

# Step 3: If Cloud SQL itself is down, check GCP console
gcloud sql instances describe fluxiq-pg-dev --project=fluxiqbr

# Step 4: Verify database connectivity after recovery
kubectl exec -n spb deployment/bacen-gateway -c bacen-gateway -- \
  /app/bin/bacen_gateway rpc "BacenGateway.Repo.query(\"SELECT 1\")"
```

### 9.4 Performance Diagnostics

```bash
# Check pod resource usage
kubectl top pods -n spb

# Check node resource usage
kubectl top nodes

# Check pod memory/CPU limits
kubectl describe pod -n spb -l app=bacen-gateway | grep -A 3 "Limits\|Requests"

# Check NATS JetStream storage
kubectl exec -n spb deployment/nats -- curl -s http://localhost:8222/jsz | python3 -m json.tool

# Check IBM MQ queue depths
kubectl exec -n spb deployment/ibmmq -- \
  bash -c "echo 'DISPLAY QLOCAL(*) CURDEPTH' | runmqsc QM.12345678.01" 2>/dev/null | grep CURDEPTH
```

### 9.5 Useful kubectl Commands Reference

```bash
# Get all resources in SPB namespace
kubectl get all -n spb

# Watch pods in real-time
kubectl get pods -n spb -w

# Get pod details with events
kubectl describe pod -n spb <pod-name>

# Execute command in a pod
kubectl exec -it -n spb deployment/bacen-gateway -c bacen-gateway -- /bin/sh

# Port-forward a service locally
kubectl port-forward -n spb svc/api-gateway 4000:4000
kubectl port-forward -n spb svc/nats 4222:4222
kubectl port-forward -n spb svc/bacen-simulator 4001:4001

# Copy files from/to a pod
kubectl cp -n spb <pod-name>:/path/to/file ./local-file
kubectl cp ./local-file -n spb <pod-name>:/path/to/file

# Scale a deployment
kubectl scale deployment <name> -n spb --replicas=<count>

# View deployment history
kubectl rollout history deployment/<name> -n spb

# Rollback to previous version
kubectl rollout undo deployment/<name> -n spb
```

---

**Document Version History:**

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | 2026-02-07 | Monetarie Engineering | Initial release |

**Related Documents:**

- SPB Integration Guide: `/Users/luizpenha/monetarie/spb/docs/spb-integration-guide.md`
- Deployment Validation Report: `/Users/luizpenha/monetarie/spb/docs/plans/2026-02-06-deployment-validation-report.md`
- BACEN Simulator Config: `/Users/luizpenha/monetarie/spb/simulator/config/scenarios.json`
- Kubernetes Manifests: `/Users/luizpenha/monetarie/spb/k8s/production/`