hyperframes-media Checked

• **Request** (audio_request.json): { provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }. id joins each line back to the caller's model (a frame number, a scene id, …). bgm.mode = retrieve | generate | none; omit for auto (retrieve when credentialed, else generate). An **explicit** retrieve is strict — it skips rather than starting a detached generate (for callers with no wait-bgm step).
• **Output** (audio_meta.json, id-keyed): { tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }.
• --only tts,bgm,sfx runs a subset and **merges** into an existing --out (e.g. TTS+BGM early, SFX once cues exist).
• BGM generate is spawned **detached** (bgm_pending: true) — run scripts/wait-bgm.mjs before assembling.
• scripts/heygen-tts.mjs is a single-shot CLI over the same code (one text → wav + words) for when you just need HeyGen TTS without a request file.

Full flag list + the audio_meta.json schema live in the header of scripts/audio.mjs. The references below cover the provider details and edge cases behind each capability.

Preflight — show sign-in status before any audio

**Always run this before generating voice or BGM — inside a full workflow _or_ a one-off "generate me a BGM/voiceover" request.** No HeyGen credential is **not** a reason to silently fall back to local engines: first recommend signing in and let the user decide. Run the shared preflight and **relay its output verbatim** — don't improvise your own "missing key" prompt, and don't offer to write keys into a per-repo .env:

npx hyperframes auth status

• **Signed in** → it prints the account; proceed.
• **Not signed in** (exit 1 is expected here — "not signed in" is a normal state, not a failure) → it prints registration-first guidance. Recommend signing in: npx hyperframes auth login is browser OAuth — it **signs in and creates an account** (always available through this repo's CLI). To use an existing HeyGen API key (from app.heygen.com/settings/api), run npx hyperframes auth login --api-key — it saves to the shared ~/.heygen (no per-repo .env). The output also lists the local engines voice/BGM will fall back to and a pip hint when deps are missing. **Relay this output as-is — don't paraphrase it into your own wording.** Then **STOP and wait** for the user to choose — sign in, or say "go" / "local" to continue offline — **before generating anything.** This is a real decision point, not a passing note: don't fold it into another question, and don't proceed past it on your own. (Exception: in autonomous / non-interactive mode, note the status and continue offline.)
• npx hyperframes auth status --json returns { configured, recommended_action, offline_engines } for deterministic branching.
• **If the CLI can't run** (not on PATH and npx can't fetch it) → still **recommend signing in** (npx hyperframes auth login) and **STOP for the user's choice** — don't treat "no credential" as a silent green light for local generation.

Credential resolution, full key priority, and the local-dependency list are in references/requirements.md.

Provider chains (the detail behind the engine)

**TTS** — first available provider wins (the engine, or npx hyperframes tts "..."):

| Order | Provider | Detected when | Word timestamps |

| ----- | ----------------------------- | -------------------------------------------- | ---------------------------------------------------------------- |

| 1 | HeyGen (Starfish) | $HEYGEN_API_KEY / hyperframes auth login | **Yes, native** — pass --words narration.words.json to capture |

| 2 | ElevenLabs | $ELEVENLABS_API_KEY set | No — chain transcribe after |

| 3 | Kokoro-82M (local, 54 voices) | always (no key required) | No — chain transcribe after |

> The published hyperframes tts CLI is often the local-only build (its --help says "Kokoro-82M", no --provider/--words) and silently falls back to Kokoro even with $HEYGEN_API_KEY set. That is why the engine's HeyGen path is the self-contained scripts/heygen-tts.mjs (REST), NOT the CLI; the CLI is used only for the Kokoro path. See references/tts.md.

**BGM & SFX** — by default **retrieved** from the HeyGen audio library (/v3/audio/sounds), same credential as HeyGen TTS, with the no-credential fallback from the switch above:

| Asset | HeyGen type | Lands in | Fallback (no credential) |

| ----- | ------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------- |

| BGM | music | assets/bgm/track.mp3 (retrieve) · track.wav (generate) | Lyria / MusicGen generation |

| SFX | sound_effects (min_score 0.4) | assets/sfx/<slug>.mp3 | bundled 21-file library (assets/sfx/* + manifest.json) |

See references/bgm.md and references/sfx.md.

Routing

| Task | Read |

| ------------------------------------------------------------------- | -------------------------------------------- |

| The audio engine — request/meta schema, --only, the switch | scripts/audio.mjs (header comment) |

| npx hyperframes tts / heygen-tts.mjs — providers, voices, words | references/tts.md |

| BGM — HeyGen retrieval + local Lyria / MusicGen generation | references/bgm.md |

| SFX — HeyGen retrieval (min_score 0.4) + bundled local library | references/sfx.md |

| npx hyperframes transcribe — Whisper, model rules, output shape | references/transcribe.md |

| npx hyperframes remove-background — transparent cutouts | references/remove-background.md |

| TTS → transcription → captions (no recorded voiceover) |