Script to published episode — even when your TTS provider dies.
A self-healing audio-episode factory on Genblaze + Backblaze B2.
Generative-media pipelines are brittle in exactly one place that matters: the vendor call. TTS providers throttle, return 500s, and go down.
CastIron makes one measurable promise — zero dropped episodes — and backs it with four load-bearing mechanisms.
Every narration render tries distinct vendors in order — ElevenLabs → LMNT → Hume. One goes dark mid-render, the ladder steps down a rung, the episode still ships. The manifest records the actual provider used, not the requested one.
The asset manifest is embedded inside the MP3 (ID3). Edit one byte and verify() flips to False — tamper-evidence travels with the episode.
Finished episodes land in Backblaze B2 under a real Object Lock (GOVERNANCE, 30 days). "Published" means provably unaltered.
An AgentLoop quality gate (loudness · silence · duration), transient resume with a single charge, a budget hard-abort, and an always-green OFFLINE fallback keep the run alive.
Flip the chaos toggle: the primary voice dies mid-render. The ladder catches it on rung 0, steps to rung 1 — and the episode still lands, verified and published.
| Pipeline.astream / arun | parallel fan-out + typed event stream | pipeline.py |
| ObjectStorageSink + read_manifest | HIERARCHICAL store, verified read-back | pipeline.py |
| SmartEmbedder | in-file ID3 manifest embed | media.py |
| resume_step / aresume_step | transient resume — single charge | resume.py |
| AgentLoop evaluators | LUFS + silence + duration quality gate | gate.py |
| ObjectLockConfig(GOVERNANCE) | immutable publish | publish.py |
| StorageBackend subclass | + ProviderComplianceTests conformance | backends.py |
| RetryPolicy · for_backblaze | per-rung retries · LIVE B2 swap | ladder.py |
Every run's assets + manifest.json land in B2 under runs/{date}/{run}/… via the S3-compatible API.
HMAC-signed notifications trigger the idempotent render→mix→verify→publish stage machine. Storage events are the workflow.
Immutability isn't claimed — it's read back: get_object_retention confirms the lock on the real bucket.
| Scenario | n | p50 / p95 | Correctness gate |
|---|---|---|---|
| Healthy — provider up | 48 | 119 / 126 ms | 48/48 shipped verified ✓ |
| Failover — primary TTS killed | 48 | 120 / 134 ms | rung steps + still ships ✓ |
| Tamper — 1 byte edited | 1 | — | verify() → False ✓ |
| Budget — spend over cap | 1 | — | BUDGET_ABORT before spend ✓ |
$ OFFLINE=1 .venv/bin/python bench.py seed=42 · mock providers · LocalDirBackend … HEADLINE: 96/96 episodes shipped hash-verified across healthy + forced-outage runs — 0 dropped. failover p50≈120ms · p95≈134ms ALL GREEN — 0 dropped episodes $ echo $? 0 # exits non-zero on any correctness failure
CodeQL + Dependabot + secret scanning all enabled, all clean. Webhook HMAC verified in constant time.
The publish stage machine converges under duplicate and reordered webhook delivery. Typed failures: BUDGET_ABORT, degraded-but-recorded runs.
Mock providers + local backend = zero-network, zero-credential path. It's the dev loop, the regression net, and the demo-day disaster fallback.
The SQLite event log is the source of truth — reconnect-safe SSE; the hub is just a low-latency nudge. State survives the stream.
Real B2 runs archived in-repo: docs/evidence/ — kill-switch drill + live publish with the Object Lock retention read back.
The whole harness runs on a laptop: make ci · make e2e · make bench · make security-scan.
Honest scope: E2E = API/ASGI + the deterministic offline proof — this is a FastAPI backend, not a web UI. Performance gate measures orchestration, not vendor synthesis.
One uniform provider + manifest layer is what a cross-provider failover ladder and manifest-verified resume can exist on top of.
Object Lock is what turns "published" into "provably unaltered" — and Event Notifications make storage the control plane.
The hosted console and the recorded chaos-drill walkthrough ship next — until then, everything on these slides reproduces from the repo with OFFLINE=1 .venv/bin/python bench.py.
Deploy the notification receiver to a public URL so B2 Event Notifications drive publish end-to-end in production — the HMAC verify and stage machine are already proven against synthetic deliveries.
Genblaze's built-in fallback is in-provider only — the cross-vendor gap CastIron fills is filed upstream as a dossier issue, with our ladder as the working reference.
The product claim is operational: a show that publishes every day because vendor outages stop mattering. Zero dropped episodes, indefinitely.
Don't take the deck's word for it — the whole claim reproduces in one command, no keys, no config.
Vendors die. Episodes ship.
$ git clone github.com/edycutjong/castiron $ uv sync # genblaze==0.4.1 + deps $ OFFLINE=1 .venv/bin/python bench.py HEADLINE: 96/96 shipped hash-verified — 0 dropped. ALL GREEN — 0 dropped episodes $ OFFLINE=1 .venv/bin/python -m uvicorn app.main:app # → open /console · flip the chaos toggle yourself