# Maintenance

Routine maintenance procedures including upgrades, certificate rotation, database maintenance, and operational runbooks.

## Prerequisites

- Admin access to Kubernetes cluster or infrastructure
- Familiarity with rolling update strategies
- Access to backup systems

## Rolling Updates

### Backend Update

```bash
# Build new image
SHORT_SHA=$(git rev-parse --short HEAD)
gcloud builds submit --config=backend/cloudbuild.yaml \
  --substitutions=SHORT_SHA=$SHORT_SHA \
  --machine-type=E2_HIGHCPU_8 --region=southamerica-east1 backend/

# If Cloud Build auto-deploys, pods update automatically
# Otherwise, update manually:
kubectl set image deployment/pix-backend -n pix \
  pix-backend=REGISTRY/pix-backend:$SHORT_SHA

# Monitor rollout
kubectl rollout status deployment/pix-backend -n pix
```

::: tip SAME TAG WARNING
Using the same image tag (e.g., `latest`) will not trigger a pod restart. Always use a unique tag (commit SHA) or force restart:
```bash
kubectl rollout restart deployment/pix-backend -n pix
```
:::

### Frontend Update

```bash
gcloud builds submit --config=frontend/admin/cloudbuild.yaml \
  --substitutions=SHORT_SHA=$SHORT_SHA \
  --machine-type=E2_HIGHCPU_8 --region=southamerica-east1 frontend/admin/
```

### Database Migration

Always migrate after backend updates:

```bash
kubectl exec -n pix deployment/pix-backend -- bin/monetarie_pix eval "Shared.Release.migrate()"
```

## Certificate Rotation

### ICP-Brasil (BACEN)

1. Obtain new certificate before expiration
2. Convert to PEM format
3. Update K8s Secret:
   ```bash
   kubectl create secret generic bacen-certs -n pix \
     --from-file=cpic.pem=new-cpic.pem \
     --from-file=cpic-key.pem=new-cpic-key.pem \
     --from-file=bacen-ca.pem=bacen-ca.pem \
     --dry-run=client -o yaml | kubectl apply -f -
   ```
4. Restart backend (CertificatePool reloads on startup):
   ```bash
   kubectl rollout restart deployment/pix-backend -n pix
   ```

### TLS Certificates

With cert-manager, rotation is automatic. For manual certs:

```bash
kubectl create secret tls monetarie-dev-tls -n pix \
  --cert=new-tls.crt --key=new-tls.key \
  --dry-run=client -o yaml | kubectl apply -f -
```

## Database Maintenance

### Vacuum and Analyze

```bash
# Connect to database
kubectl exec -n pix deployment/pix-backend -- \
  bin/monetarie_pix eval "Shared.Repo.query!('VACUUM ANALYZE monetarie_spi.messages')"
```

### Partition Management

`PartitionManager` GenServer runs daily and:
- Creates partitions 60 days ahead for `activity_log` and `login_history`
- Purges expired audit records via `XmlArchiver.purge_expired/0`

Manual partition check:

```sql
SELECT schemaname, tablename FROM pg_tables
WHERE tablename LIKE '%activity_log%' OR tablename LIKE '%login_history%'
ORDER BY tablename;
```

### Index Maintenance

```sql
-- Check index usage
SELECT schemaname, tablename, indexname, idx_scan
FROM pg_stat_user_indexes
WHERE schemaname IN ('monetarie_spi', 'monetarie_dict', 'monetarie_auth')
ORDER BY idx_scan;

-- Reindex if needed
REINDEX INDEX CONCURRENTLY idx_messages_e2e_trgm;
```

## NATS Maintenance

### Stream Maintenance

```bash
# Check stream status
nats stream ls
nats stream info MONETARIE_SPI

# Purge old messages (if needed)
nats stream purge MONETARIE_SPI --keep=10000

# Check consumer lag
nats consumer info MONETARIE_SPI pix-inbound-processor
```

## Redis Maintenance

```bash
# Check memory usage
redis-cli INFO memory

# Check key count
redis-cli DBSIZE

# Flush expired keys (automatic with TTL)
# Manual flush (use with caution):
redis-cli FLUSHDB
```

## Health Verification Runbook

Run after any maintenance operation:

```bash
# 1. Check all pods
kubectl get pods -n pix

# 2. Health checks
curl -s https://pixapi-dev.fluxiq.com.br/health
curl -s https://pixapi-dev.fluxiq.com.br/metrics | head -20

# 3. Check logs for errors
kubectl logs -n pix deployment/pix-backend --tail=50 | grep -i error

# 4. Verify NATS workers
kubectl logs -n pix deployment/pix-backend | grep "Worker started"

# 5. Test login
curl -X POST https://pixapi-dev.fluxiq.com.br/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"Admin@2026!"}'
```

## Expected Outcome

After following maintenance procedures:

- Rolling updates complete without downtime (PDB ensures min 1 pod)
- Certificates rotated before expiration
- Database performance maintained with regular vacuum and index checks
- Partition management running automatically
- Health verification confirms all services operational