# Onda 2.A — Resilient Transfer (chunked upload + ranged download + ETag)

> **For Claude:** REQUIRED SUB-SKILL: Use `superpowers:executing-plans` to implement this plan task-by-task.

**Branch:** `sta/onda2-resilient-transfer` (this worktree, cut from `main` post Onda 1 merge `301503ec`)
**PR target:** `main`
**Depends on:** Onda 1 merged (PR #24, commit `301503ec`)
**Estimated effort:** 3-4 working days · LOC delta estimate: +800
**Parent plan:** `sta/docs/plans/2026-04-26-sta-conformance-rollout.md` (Onda 2.A brief in §Onda 2)

---

## Index

1. [Task 0 — Manual Verification](#task-0)
2. [Architecture revisions (post Task 0)](#architecture-revisions)
3. [Pre-flight](#pre-flight)
4. [Tasks](#tasks)
5. [Acceptance](#acceptance)
6. [References](#references)

---

<a id="task-0"></a>
## Task 0 — Manual Verification

Per the **Pattern: Manual Verification** mandated for every Ondas 2-4 sub-PR (`sta/docs/plans/2026-04-26-sta-conformance-rollout.md#pattern-manual-verification`). This task was completed before any code work for Onda 2.A.

**Source:** `sta/Manual_STA_Web_Services.pdf` (Manual de utilização dos Web Services do STA, v1.5, jul/2022).

### Verified `[VERIFIED §X.Y]`

| Claim | Manual section | Notes |
|---|---|---|
| `PUT /staws/arquivos/{protocolo}/conteudo` for full upload | §5.2.1 | Verified by Onda 1 already |
| `PUT /staws/arquivos/{protocolo}/conteudo` + `Content-Range: bytes {inicio-fim/total}` for partial upload | §5.6 | "inicio-fim/total: byte inicial, byte final e tamanho total — inicio e fim são obrigatórios" |
| `GET /staws/arquivos/{protocolo}/posicaoupload` for resume position lookup | §5.3.1 | Content-Type must NOT be in request header |
| `GET /staws/arquivos/{protocolo}/conteudo` for full download | §6.1 | Content-Type must NOT be in request header |
| `GET /staws/arquivos/{protocolo}/conteudo` + `Range: bytes={inicio-fim}` for ranged download | §6.4 | NB: format uses `bytes=`, NOT `bytes ` (space) |
| `If-Match: {etag}` request header (download) | §6.4 | **Optional** per Manual ("os cabeçalhos If-Match e If-Unmodified-Since são opcionais") |
| `If-Unmodified-Since: {data_modificacao_arquivo}` request header (download) | §6.4 | **Optional** |
| `ETag: {etag}` response header (full + partial download) | §6.1, §6.4 | Per RFC 7232 specification reference |
| `Last-Modified: {data}` response header (full + partial download) | §6.1, §6.4 | Standard HTTP |
| `X-Content-Hash: SHA-256 {hash}` response header — **non-standard BCB-specific** | §6.1 | "Não é um padrão do HTTP. Foi criado pelo Banco Central para validação da integridade" — **NEW; not in original plan** |
| HTTP **412** Precondition Failed (download) | §6.4 | "Validação dos cabeçalhos If-Match e If-Unmodified-Since falhou" |
| HTTP **416** Range Not Satisfiable (download) | §6.4 | "Range informado é inválido" |
| HTTP **501** Not Implemented (download) | §6.4 | "O range multipart não é suportado" — **NEW; not in original plan** |
| HTTP 200 OK (full download) / 206 Partial Content (partial download) | §6.1, §6.4 | Standard HTTP |
| Parallel upload supported (multiple connections, same protocol) | §5.5 | "Importante: o parâmetro {protocolo} é o mesmo para transmissão completa ou em partes" |
| Parallel download supported (multiple connections, same protocol) | §6.3 | Same mechanism |
| BCB connection limits: max 10 simultaneous transfers, 120/min queries (per institution) | §2.6 | Already used by existing `RateLimiter`; no changes needed for Onda 2.A |

### Misdiagnoses caught (study/plan vs Manual)

These claims in the parent plan (Onda 2.A brief) were wrong or imprecise — corrected here before implementation:

| # | Plan/study claim | Manual reality | Impact |
|---|---|---|---|
| 1 | `posicao_upload` returns `<UltimoByteRecebido>` | §5.3.1: returns `<RangesRecebidos>0-3;5-8</RangesRecebidos>` — semicolon-separated list of byte ranges | **Major.** Chunked upload can have GAPS; client must reconcile arbitrary ranges, not just resume from byte N. New `RangesRecebidos` parser + reconciliation logic required. |
| 2 | "HTTP 410 = protocol expired" (Onda 2.C scope, but referenced here) | §5.3.1 + §5.6: HTTP 410 in upload context = "O protocolo informado foi cancelado pelo Banco Central". §6.1 + §6.4: HTTP 410 in download context = "O arquivo não está disponível para download". Neither says "expired". | Re-frame error semantics: 410 means caller cannot resume — must `request_protocol` again. (Still likely correct fallback behavior; just different name.) |
| 3 | "HTTP 429 with Retry-After for rate limiting" (Onda 2.C scope) | Manual §2.6 states the limits but **does not document** what HTTP code is returned when exceeded. | Mark as `[STUDY-CONJECTURE]` for Onda 2.C; don't pre-implement based on this assumption. |
| 4 | Default chunk threshold 1 MB | Manual specifies no chunk size limit. | `[STUDY-CONJECTURE]` — pick a sensible default (1 MB or 5 MB), make it env-configurable, and document that the default is heuristic. |

### New findings (not in original plan)

1. **`<RangesRecebidos>` shape** (§5.3.1): semicolon-delimited list of dash-delimited byte ranges. Format: `"0-3;5-8"`. Must be parsed into a list of `{start_byte, end_byte}` tuples.
2. **`<Situacao>` enum on `posicaoupload` response** (§5.3.1): "Transmissão não iniciada" | "Transmissão pendente" | "Transmissão finalizada". This is **distinct from `<EstadoAtual><Codigo>`** used in queries (Onda 2.B). They're orthogonal: `Situacao` reports upload-side progress, `Estado` (Tabela 3) reports BCB-side processing state.
3. **`X-Content-Hash: SHA-256 <hex>` response header** on download (§6.1): the canonical integrity check for a downloaded file. The existing `Operations.verify_file_integrity/2` already SHA-256s and compares — but the **comparand** must come from this header (currently expected from a different path).
4. **HTTP 501 on download** (§6.4): "O range multipart não é suportado". Should be a permanent error (`{:error, :range_multipart_unsupported}`) — caller must restart from byte 0 with a single contiguous range.
5. **`Content-Type` MUST NOT be sent** in request headers for: `posicao_upload` (§5.3.1), `download_full` (§6.1), `download_part` (§6.4). For `upload_part` (§5.6) it's optional but **must not be `multipart/form-data` if present**. The existing client may currently send `Content-Type` on these — needs audit.

### Items deferred to operator confirmation `[NEEDS-OPERATOR-CONFIRM]`

- **Concurrency budget for parallel upload/download.** Manual §2.6 says "Máximo de 10 simultâneos" per institution but doesn't specify how to split between read/write. Operator should confirm whether we cap at 5+5 or use a single 10-slot bucket.
- **Chunk size default.** No Manual guidance. Will pick 1 MB as a working default; revisit after first real BCB smoke.

---

<a id="architecture-revisions"></a>
## Architecture revisions (post Task 0)

The original plan listed three new client functions: `upload_part/4`, `posicao_upload/1`, `download_part/3`. Task 0 adjusts each:

### `Client.posicao_upload/1`

- Returns `{:ok, %PosicaoUpload{ranges_received: [{start, end_inclusive}, ...], situacao: :nao_iniciada | :pendente | :finalizada, protocolo: pid}}` on 200
- New struct `StaConnector.Sta.Types.PosicaoUpload` (replaces previously planned `bytes_uploaded` scalar field on `FileInfo`)
- Errors: 400 → `{:error, :bad_request}`, 403 → `{:error, :forbidden}` (only — no 404 or 410 documented for this endpoint)

### `Client.upload_part/2` (signature: map-arg, mirroring Onda 1 convention)

- Map keys: `:protocol_number`, `:content_part`, `:start_byte`, `:end_byte`, `:total_size`
- Builds `PUT /staws/arquivos/{protocol}/conteudo` with `Content-Range: bytes {start_byte}-{end_byte}/{total_size}`
- Body: `content_part` (binary)
- Does **NOT** set `Content-Type` (per §5.6 attention note)
- Returns `{:ok, %UploadResult{}}` on 200

### `Client.download_part/2` (signature: map-arg)

- Map keys: `:protocol_number`, `:start_byte`, `:end_byte`, `:if_match` (optional), `:if_unmodified_since` (optional)
- Builds `GET /staws/arquivos/{protocol}/conteudo` with `Range: bytes={start_byte}-{end_byte}` and (if supplied) `If-Match` / `If-Unmodified-Since`
- Returns `{:ok, %DownloadResult{etag, last_modified, x_content_hash, content}}` on 206
- Errors: 412 → `{:error, :precondition_failed}`, 416 → `{:error, :invalid_range}`, 501 → `{:error, :range_multipart_unsupported}`, 410 → `{:error, :file_unavailable}`

### `Operations.upload_file/3` revision

Sequence becomes:
1. `Client.request_protocol/2` → protocol
2. If `byte_size(content) > chunk_threshold`:
   - `chunks = chunk(content, threshold)`
   - For each chunk: `Client.upload_part/2`; on transient failure for chunk N, `Client.posicao_upload/1` to get `<RangesRecebidos>`, reconcile against expected ranges, **resume only the missing ranges** (NOT just from byte N)
3. Else: `Client.upload_file/2` (full PUT, existing path)
4. Return `{:ok, %{protocol_number: protocol, file_name: file_name}}`

### `Operations.download_file/2` revision

1. `Client.download_file/2` (full GET)
2. On transient failure mid-download: persist ETag from previous attempt, retry via `Client.download_part/2` with `If-Match` from N to end-of-file
3. On 412 (file changed): restart from 0 (no `If-Match`)
4. On 416 (bad range): restart from 0 (single range)
5. On 501 (multipart range): restart from 0 (Manual disallows multipart)
6. **Verify `X-Content-Hash: SHA-256 ...`** vs SHA-256 of received body

### Type additions

- `StaConnector.Sta.Types.PosicaoUpload` — `protocolo, ranges_received: [{integer, integer}], situacao: :nao_iniciada | :pendente | :finalizada`
- `StaConnector.Sta.Types.DownloadResult` — extend with `:etag, :last_modified, :x_content_hash` (only as needed for partial-download responses)
- `StaConnector.Files.OutboundFile` schema — add `etag :string`, `last_modified :utc_datetime`, `bytes_uploaded :bigint default 0`, `protocol_expires_at :utc_datetime` (last is Onda 2.C scope; defer)

**Cut from original plan:** `state_code` field. That's `<EstadoAtual><Codigo>` from query responses (Onda 2.B), not from upload/download flow.

---

<a id="pre-flight"></a>
## Pre-flight

```bash
cd /Users/luizpenha/monetarie/.worktrees/sta-paridade

# 1. Confirm branch state
git status              # On sta/onda2-resilient-transfer, clean
git log --oneline -1    # Should show this Task 0 + plan commit when landed

# 2. Compile baseline (5 warnings expected)
cd sta/backend
mix deps.get
mix compile             # 5 baseline warnings (Logger.info macro, unused alias, ungrouped clauses)

# 3. Touchstone tests
mix test test/sta_connector/application_test.exs        # 3/3 PASS
mix test test/sta_connector/sta/client_test.exs         # 35 tests, 6 pre-existing Soap.parse_response failures
```

---

<a id="tasks"></a>
## Tasks

### Task 1 — Add `Types.PosicaoUpload` struct + `Client.posicao_upload/1`

**Files:**
- Modify: `sta/backend/lib/sta_connector/sta/types.ex` (add `PosicaoUpload`)
- Modify: `sta/backend/lib/sta_connector/sta/client.ex` (add `posicao_upload/1` + behaviour callback)
- Modify: `sta/backend/lib/sta_connector/sta/client_behaviour.ex`
- Test: `sta/backend/test/sta_connector/sta/client_test.exs` (Bypass for `<RangesRecebidos>` parsing happy + edge cases)

**Step 1 (TDD red):** test that the client parses `<RangesRecebidos>0-3;5-8</RangesRecebidos>` into `[{0, 3}, {5, 8}]` and `<Situacao>Transmissão pendente</Situacao>` into `:pendente`. Cover empty ranges (`<RangesRecebidos></RangesRecebidos>` → `[]`) and `não iniciada` / `finalizada`.

**Step 2 (TDD green):** implement parser + signature. NB: `posicao_upload` MUST NOT send `Content-Type` (per §5.3.1).

**Step 3:** add behaviour callback (with proper Manual citation `§5.3.1`).

**Step 4:** commit `feat(sta): Client.posicao_upload + RangesRecebidos parser`.

**Estimate:** 4-6h.

---

### Task 2 — Add `Client.upload_part/2`

**Files:**
- Modify: `sta/backend/lib/sta_connector/sta/client.ex`
- Modify: `sta/backend/lib/sta_connector/sta/client_behaviour.ex`
- Test: `sta/backend/test/sta_connector/sta/client_test.exs`

**Step 1 (TDD red):** test that `upload_part/2` builds `PUT /arquivos/{protocol}/conteudo` with `Content-Range: bytes 1024-2047/8192` and the chunk binary. Test that NO `Content-Type` header is sent (Manual §5.6 attention note). Cover edge: first chunk (0-N), last chunk (where end == total-1), partial last chunk.

**Step 2 (TDD green):** implement; 200 OK happy; 400/403 error mapping.

**Step 3:** behaviour callback + Manual citation §5.6.

**Step 4:** commit.

**Estimate:** 3-4h.

---

### Task 3 — Add `Client.download_part/2` + ETag/`X-Content-Hash` plumbing

**Files:**
- Modify: `sta/backend/lib/sta_connector/sta/client.ex`
- Modify: `sta/backend/lib/sta_connector/sta/client_behaviour.ex`
- Modify: `sta/backend/lib/sta_connector/sta/types.ex` (extend `DownloadResult`)
- Test: `sta/backend/test/sta_connector/sta/client_test.exs`

**Step 1 (TDD red):**
- (a) `download_part/2` builds `GET /arquivos/{protocol}/conteudo` with `Range: bytes=1024-2047`, optional `If-Match: "abc"`, optional `If-Unmodified-Since: "..."`.
- (b) On 206, parses `ETag`, `Last-Modified`, and `X-Content-Hash: SHA-256 <hex>` response headers into `DownloadResult`.
- (c) Errors: 410 → `:file_unavailable`, 412 → `:precondition_failed`, 416 → `:invalid_range`, 501 → `:range_multipart_unsupported`.

**Step 2 (TDD green):** implement.

**Step 3:** behaviour callback + Manual citations §6.4.

**Step 4:** commit.

**Estimate:** 4-6h.

---

### Task 4 — Migration: extend `OutboundFile` with `etag`, `last_modified`, `bytes_uploaded`

**Files:**
- Create: `sta/backend/priv/repo/migrations/<TIMESTAMP>_add_resilient_transfer_columns.exs`
- Modify: `sta/backend/lib/sta_connector/files/outbound_file.ex` (schema + changeset)

**Step 1:** generate migration adding columns; run `mix ecto.migrate` against postgres dev container.

**Step 2:** update `outbound_file.ex` schema + `changeset/2` to include new fields.

**Step 3:** unit-test that changeset accepts/rejects the new fields appropriately.

**Step 4:** commit.

**Estimate:** 2-3h.

---

### Task 5 — `Operations.upload_file/3` learns chunked upload

**Files:**
- Modify: `sta/backend/lib/sta_connector/sta/operations.ex`
- Modify: `sta/backend/lib/sta_connector/outbound/uploader.ex` (or `outbound/worker.ex`, depending on which orchestrates)
- Test: `sta/backend/test/sta_connector/sta/operations_test.exs`

**Step 1 (TDD red):**
- (a) Files ≤ chunk_threshold use existing single PUT path (regression).
- (b) Files > threshold split into N chunks, each PUT'd via `upload_part/2` in order.
- (c) Simulated transient failure on chunk 3: caller calls `posicao_upload/1`, reconciles `<RangesRecebidos>`, and resumes ONLY the missing chunks (not just chunk 3 onward — could have gaps from parallel-upload Onda 2 future).
- (d) After last chunk: emit `set_protocol`/`mark_uploaded` as before.

**Step 2 (TDD green):** implement chunking + reconciliation. Default `:chunk_threshold` = 1 MB (env-tunable as `STA_CHUNK_THRESHOLD_BYTES`).

**Step 3:** persistence: on each successful chunk, update `OutboundFile.bytes_uploaded` and `OutboundFile.protocol_expires_at` (the latter Onda 2.C; for now, just `bytes_uploaded`).

**Step 4:** commit.

**Estimate:** 6-8h.

---

### Task 6 — `Operations.download_file/2` learns ranged GET + ETag persistence

**Files:**
- Modify: `sta/backend/lib/sta_connector/sta/operations.ex`
- Modify: `sta/backend/lib/sta_connector/inbound/worker.ex` (persistence)
- Test: `sta/backend/test/sta_connector/sta/operations_test.exs`

**Step 1 (TDD red):**
- (a) Full download: same as before but persists `etag` + `last_modified` from response headers + verifies `X-Content-Hash` against SHA-256(received_body).
- (b) Transient failure mid-download: client retries via `download_part/2` with `Range: bytes=N-` (where N = bytes received so far) + `If-Match` from earlier attempt's etag.
- (c) On 412: restart from 0 (no `If-Match`).
- (d) On 416: restart from 0.
- (e) On 501: restart from 0 with a single range (we never did multipart, so this is defensive).

**Step 2 (TDD green):** implement.

**Step 3:** commit.

**Estimate:** 6-8h.

---

### Task 7 — Final validation + PR

**Steps:** mirrors Onda 1's Task 6 closeout (mix format, compile, scoped tests, guard rails, push, gh pr create with body referencing this plan).

**Estimate:** 2-3h.

---

<a id="acceptance"></a>
## Acceptance

- All 6 implementation tasks land as commits with TDD coverage
- Bypass-tested scenarios:
  - Full upload < threshold (regression — Onda 1 path unchanged)
  - Chunked upload at threshold + 1 byte
  - Chunked upload with simulated transient failure on middle chunk → reconciles `<RangesRecebidos>` → resumes missing chunks
  - Full download with `X-Content-Hash` integrity verification
  - Ranged download with `If-Match` mismatch (412 → restart from 0)
  - Bad range (416 → restart from 0)
  - 501 multipart range (defensive restart)
- `mix format --check-formatted` clean for all touched files
- `mix compile` clean (5 baseline warnings unchanged)
- `mix test test/sta_connector/sta/operations_test.exs` + `client_test.exs` green
- Migration applies clean against postgres dev container
- Guard rails: `git diff $(git merge-base origin/main HEAD)..HEAD --name-only | grep -v "^sta/"` returns empty
- PR description references this plan, the parent rollout plan, and lists the 4 misdiagnoses caught + 5 new findings from Task 0 (continuing the misdiagnosis-pattern documentation precedent set in Onda 1)
- Real BCB smoke test (when creds available) — deferred to Onda 2 sub-PR with creds, or operator-driven post-merge

---

<a id="references"></a>
## References

- **Manual canônico:** `sta/Manual_STA_Web_Services.pdf` v1.5 jul/2022 — primary spec
  - §2.6 Limites de conexões
  - §5.3 Consulta a situação do envio (`posicao_upload`)
  - §5.4 Retomada do envio
  - §5.5 Envio em paralelo
  - §5.6 Envio de parte de arquivo
  - §6.1 Recebimento completo
  - §6.2 Retomada do recebimento (resume)
  - §6.3 Recebimento em paralelo
  - §6.4 Recebimento de parte do arquivo
- **Parent plan:** `sta/docs/plans/2026-04-26-sta-conformance-rollout.md` (Onda 2.A brief in §Onda 2; Pattern: Manual Verification in §pattern-manual-verification)
- **Onda 1 final handoff:** `sta/docs/handoff/2026-04-26-sta-paridade-onda1-final-handoff.md`
- **Onda 1 PR (merged):** https://github.com/FluxiqBR/monetarie/pull/24
- **RFCs referenced by Manual:**
  - RFC 7233 (HTTP Range Requests) — `Range`, `Content-Range`
  - RFC 7232 (HTTP Conditional Requests) — `If-Match`, `If-Unmodified-Since`, `Last-Modified`, `ETag`
- **SPB Wave 9 decompile-first lesson:** `feedback_decompile_first_when_porting_legacy.md` (auto-memory) — same anti-pattern recovery applied here

---

**Plan version:** 1.0 (Task 0 verification + revised architecture) · **Author:** Claude (under direction of Luiz Marcelo) · **Date:** 2026-04-26