Backblaze Generative Media Hackathon · Devpost

Script to published episode — even when your TTS provider dies.

CastIron is a self-healing audio-episode factory on Genblaze + Backblaze B2. When the primary voice provider goes dark mid-render, a cross-provider failover ladder steps down a rung and the episode still ships — hash-verified, immutably published.

View on GitHub → See the pitch deck 0 dropped episodes· 96/96 verified

← swipe to follow the run →

01 · THE PROBLEM

One vendor call away from dead air.

Generative-media pipelines are brittle in exactly one place that matters: the vendor call. TTS providers throttle, return 500s, and go down — and when they do, a naive pipeline drops the entire episode: narration, music, cover, the run.

On a schedule, one outage = dead air. A daily brief or podcast doesn't get to skip a day because a vendor had a bad hour.
All-or-nothing runs. One failed narration render takes the music and cover art down with it.
No tamper-evidence after publish. Once produced, most pipelines can't prove an episode wasn't silently altered afterward.

NAIVE PIPELINE · RUN LOG

09:00:00run.start daily-brief · 3-stage fan-out

09:00:04tts.render provider: primary … HTTP 500

09:00:04tts.render retry … HTTP 500 — provider down

09:00:05run.abort narration failed → music ✕ · cover ✕

09:00:05EPISODE DROPPED — DEAD AIR

02 · THE SOLUTION

Provider failure is the expected case, not the exception.

Four pillars turn a fragile render script into a factory you'd put on a schedule.

Cross-provider failover ladder

Every narration render tries distinct vendors in order — ElevenLabs → LMNT → Hume. One goes dark mid-render, the ladder steps down a rung and the episode still ships. The manifest records the actual provider used, not the requested one.

Provenance hashed into the file

Each asset's manifest is embedded inside the MP3 (ID3). Edit one byte and verify() flips to False — tamper-evidence travels with the episode, wherever the file goes.

Immutable Object-Lock publish

Finished episodes land in Backblaze B2 under a real Object Lock (GOVERNANCE, 30 days), proven live by reading back get_object_retention. "Published" means "provably unaltered."

Self-healing orchestration

An AgentLoop quality gate (loudness · silence · duration), transient resume that charges once — not resubmits — a budget hard-abort before spend, and an always-green OFFLINE fallback keep the run alive.

03 · THE FAILOVER MOMENT

Kill the primary provider. Watch the episode ship anyway.

This is the hero moment from the live demo: flip the chaos toggle to force a TTS outage. The ladder catches the failure on rung 0, steps down, and the run completes — verified and published.

POST /runs · SCRIPT: daily-brief.md

RUNG 0elevenlabsIDLE

ladder.step ≈120 ms

RUNG 1lmntSTANDBY

RUNG 2humeSTANDBY

episode.mp3 — rendering…B2 · CI-PUBLISHED/ · OBJECT LOCK 30D

SSE · LIVE RUN RAILRUN #96 · IN FLIGHT

FAILOVER TRIALS 48/48 SHIPPED RUNG STEP ≈120 MS MANIFEST RECORDS ACTUAL PROVIDER

04 · PROOF

One measurable promise, reproduced in one command.

Every number below is real, deterministic, and reproducible from the repo — the benchmark exits non-zero if any correctness gate fails.

96/96

Episodes verified

Shipped hash-verified across healthy + forced-outage runs.

Dropped episodes

The primary TTS provider is killed on every failover trial.

120 ms

Failover p50

p95 ≈ 134 ms — OFFLINE orchestration; vendor synth excluded.

129

Tests passing

~92% line coverage on castiron/.

CI stages, green

Quality → Security → Build → E2E → Performance → Deploy.

Open security alerts

CodeQL + Dependabot + secret scanning.

30d

Object Lock, live

GOVERNANCE retention proven by get_object_retention read-back.

1 byte

Tamper trial

Edit one byte of provenance → verify() flips to False.

REPRODUCE — ZERO CONFIG · NO KEYS · FIXED SEED

# mock providers + local backend — the same always-green path the app falls back to
OFFLINE=1 .venv/bin/python bench.py
→ HEADLINE: 96/96 episodes shipped hash-verified … — 0 dropped.
→ ALL GREEN — 0 dropped episodes

05 · THE ENGINE

Why only B2 + Genblaze.

The SDK is the engine, not decoration. The resilience thesis needs both halves — one makes cross-provider failover possible, the other makes "published" provable.

Genblaze 0.4.1 — the pipeline engine

Provider abstraction that makes the ladder possible

▸Pipeline.astream — 3-stage parallel fan-out (narration · music · cover) with a typed event stream
▸ObjectStorageSink + read_manifest(verify=True) — hierarchical layout, verified read-back
▸SmartEmbedder — the in-file ID3 manifest embed
▸resume_step — transient resume: single charge, not resubmit
▸AgentLoop evaluators — the loudness / silence / duration quality gate
▸ObjectLockConfig · StorageBackend subclassing · ProviderComplianceTests

Backblaze B2 — storage plane and control plane

Storage that turns "published" into "provably unaltered"

▸S3 API — hierarchical object layout for every run's assets + manifest.json
▸Event Notifications — HMAC-signed webhooks drive an idempotent render → mix → verify → publish stage machine
▸Object Lock — GOVERNANCE, 30 days; retention verified live via get_object_retention
▸SQLite event log + SSE console ride on top: B2 events are the source of truth

Remove either half and the differentiator collapses. Without Genblaze there's no uniform provider/manifest layer to fail over across; without B2 Object Lock the tamper-evidence stops at the file and never reaches storage.