Overline
A small Manifest V3 Chrome extension that overlays interactive live Mandarin subtitles on supported Chinese-language AniKoto shows:
https://anikototv.to/watch/link-click-2e0jm/ep-1
https://anikototv.to/watch/the-girl-downstairs-9eddv/ep-1
Live transcription works across all numbered episodes of Link Click and The Girl Downstairs. Each show receives an isolated model prompt and vocabulary list. The optional bundled precomputed dataset remains specific to Link Click episode 1; every other episode uses live transcription. No video is stored by the extension.
Install and demo
- Open
chrome://extensions in Chrome.
- Turn on Developer mode.
- Choose Load unpacked and select the
extension/ folder. If you later click Reload on the extension card while the episode is already open, refresh the episode tab once as well.
- Open a supported episode URL and start playback. Use any working supported SUB server; Vidstream-2 and VidPlay-1 are both supported, and you do not need to choose VidPlay-1.
- Confirm the page diagnostic changes from connecting to video or waiting for video to load to Mandarin connected. Seek to about 2:15, where the first Mandarin cue should begin.
- Hover intentionally over a Chinese word for about 250 ms.
The video pauses and reveals pinyin syllables directly above their corresponding Hanzi, plus an unboxed contextual gloss. Moving away closes the learning detail but leaves playback paused. Press Space to resume.
The initial episode alignment is episode time = capture time + 124 seconds. This is anchored from the live observation that the VidTube player was near 2:18 when the native English subtitle read “I'm paying your debt,” which falls between the captured 12.9s and 15.7s cues. Uncertainty is about ±2.5 seconds. To calibrate, seek to the first line 老租婆又来吸血喽。, then click Sync first line in the top-right player overlay. Use −0.5/+0.5 for fine adjustment; the offset is saved locally by the extension.
For a deterministic cross-origin UI demo, serve the repository root on two ports:
python3 -m http.server 4173
python3 -m http.server 4174
Then open http://127.0.0.1:4173/fixture/. The outer fixture and player iframe use different origins and exercise the same authenticated parent/frame message flow. The fixture is not part of the extension's host permissions.
Capture and transcribe a real excerpt
Chrome requires a direct user gesture for tab capture, so recording starts only when the user opens the extension toolbar popup and clicks Start recording.
- Open the supported AniKoto episode and start playing the desired excerpt.
- Click the extension toolbar icon, then Start recording. Keep the popup open.
- After 60–90 seconds, click Stop and save (capture automatically stops at 90 seconds).
- Chrome downloads one local Opus/WebM audio file and one small metadata JSON file. The schema-v2 metadata records the player’s exact
HTMLMediaElement.currentTime at capture start, every 15 seconds, at capture stop, and whenever playback pauses, seeks, buffers, resumes, or changes speed. The extension captures no video and uploads nothing.
- Run
scripts/transcribe-openai.sh ~/Downloads/<capture>.webm. If OPENAI_API_KEY is not already present, the helper prompts with hidden input and keeps the key only in that process. The script sends only the captured audio clip to OpenAI's /v1/audio/transcriptions endpoint using gpt-transcribe by default; the API key is never echoed, written, or exposed to the extension.
- The script detects silence locally, transcribes bounded speech windows as Mandarin, and writes
work/transcript/transcript.raw.json plus cue-draft.json. Those window offsets provide draft segment timing because this implementation does not request word-level timing. Review the audio, hand-correct Mandarin and timing, segment the sentences into learning targets, then add pinyin and contextual glosses.
The extension build, local fixture, and tab-audio capture do not require an API key. Only the one-time hosted transcription call does.
To validate a capture's duration and local speech-window detection without entering a key or making an API call, run:
scripts/transcribe-openai.sh --validate-only ~/Downloads/link-click-ep1-excerpt-<timestamp>.webm
Chrome WebM recordings may omit container duration metadata. The helper first accepts a matching, trustworthy .capture.json generated by the extension, then falls back to decoding the local audio stream to enforce the same 95-second maximum.
Precomputed cue timing uses the recorded player clock as its primary source. Each transcript timestamp maps from capture time to episode time using the most recent anchor and its playback state/rate. Cues that cross a seek are flagged rather than silently assigned an interpolated time. Run node tests/player-time-mapping.mjs to validate pause, buffering, seek, rate-change, and drift behavior without opening Chrome.
Validate a downloaded anchored capture with npm run capture:validate -- ~/Downloads/<capture>.capture.json. After transcription, map its cue draft directly into episode/player time with node scripts/map-cues-to-player-time.mjs <capture.json> work/transcript/cue-draft.json.
For a finer timing lattice, run npm run transcribe:timestamps -- ~/Downloads/<capture>.webm [output.json]. This makes a separate whisper-1 pass with word and segment timestamps. Keep gpt-transcribe as the text authority: Whisper timing is useful for alignment, but its recognized Mandarin must not silently replace the higher-quality transcript. WHISPER_CONTEXT_PROMPT can override the concise Mandarin vocabulary prompt for a different show.
To combine those outputs conservatively, run npm run transcribe:align -- <gpt-transcript.raw.json> <whisper-timestamps.json> [cue-draft.json]. Exact GPT/corrected text is retained. A phrase receives Whisper timing only when the text match clears the confidence threshold; missing or contradictory timing falls back to a proportional interval inside the original speech window and remains labeled as a fallback for review.
The hidden key prompt trims surrounding paste whitespace and carriage returns in memory. It rejects quotes, embedded line breaks/control characters, non-ASCII punctuation, and incomplete keys before any network request. It does not require one historical key prefix, so current project and service-account key formats remain supported. To test the capture and local Authorization-header preparation without sending audio, use --preflight instead of --validate-only; preflight asks for the key but makes no API call.
Example:
scripts/transcribe-openai.sh ~/Downloads/link-click-ep1-excerpt.webm
The script requires Node 20+ and ffmpeg/ffprobe. An API account with access to gpt-transcribe is required; official OpenAI pricing currently estimates it at $0.0045 per audio minute. Availability and charges still depend on the user's API account. For an explicit comparison or fallback, set a model only for that invocation, for example OPENAI_TRANSCRIBE_MODEL=gpt-4o-transcribe scripts/transcribe-openai.sh ~/Downloads/<capture>.webm. The model ID is validated locally and is recorded in transcript.raw.json.
Data format and provenance
extension/data/link-click-ep1.real.json contains the real captured timed cues. Each cue has explicit pre-segmented words, and every word has text, pinyin, and a short contextual gloss. The prior representative scaffold is retained at fixture/examples/representative-sample.json.
To create or update another real excerpt later:
- While lawfully viewing the episode, use the user-initiated toolbar capture for only the short excerpt needed for testing; do not download or redistribute the episode.
- Manually correct wording and cue timing.
- Segment each cue into meaningful hover targets and add pinyin plus contextual glosses.
- Preserve the current schema and replace
cues; set source and notice to an accurate provenance label.
- Run
npm run check, reload the extension, and verify timing against the actively viewed player.
Dictionary and word grouping
Live transcripts are segmented and annotated locally using a pinned CC-CEDICT release containing 124,880 entries. The engine follows Yomitan's scanning-parser model: at each character boundary it tests progressively shorter prefixes, selects the longest exact dictionary match, and advances by the matched length. General Mandarin vocabulary comes from CC-CEDICT rather than a hand-maintained whitelist.
Names, fictional organizations, and episode-specific readings live in the higher-priority adaptation layer at extension/data/dictionary-adaptations.json. An adaptation wins over a CC-CEDICT entry with the same text, while a longer exact term still wins over a shorter one. The runtime API also exposes addAdaptation, removeAdaptation, and setAdaptations for session-specific additions.
That file is also the editable source of truth for transcription context. Its transcription section supplies isolated show prompts, recurring names, Realtime language hints, and optional additions under episodes. Link Click receives names such as 陆光, 程小时, and 乔苓; The Girl Downstairs receives 袁君瑭, 李诗雅, 朱茱, 崔若霓, and its university setting. Live transcription also sends languages: ["cmn"] and the active episode number. The HTTP file endpoint retains its singular language: "zh" field. Run npm run build:dictionary after editing the context.
The two layers solve different problems: transcription context encourages the model to output the right Hanzi, then the local dictionary groups and annotates that output. The dictionary cannot repair a name that the transcription model wrote with the wrong characters.
Chinese text renders continuously; dictionary boundaries define hover targets but do not insert visible spaces. Single-character lookup is used only when no longer dictionary term matches.
Live turn and pause behavior
Realtime transcription keeps automatic server turn detection disabled because gpt-live-transcribe rejects server VAD. Deltas continue to render as they arrive; after 1.2 seconds without a new transcript delta, the extension manually commits the buffered audio as one turn. A completed item receives terminal punctuation when the model omitted it. This is sentence/turn separation only; it does not attempt speaker identification.
When the video pauses, including an intentional hover pause, the player sends its media state through the service worker to the offscreen capture owner. The Realtime connection remains available for a fast resume, but no tab-audio buffers are submitted while playback is paused. The overlay reports Live ASR Suspended · no audio sent; playback resumes audio submission automatically.
The source archive and generated asset are reproducible:
The build verifies the pinned source checksum before replacing extension/src/cedict-data.js. CC-CEDICT attribution and license details are recorded in THIRD_PARTY_NOTICES.md and included in the extension package.
Behavior and limitations
- The outer content script matches numbered Link Click and The Girl Downstairs episode paths on AniKoto; provider access is separately limited to the two observed player hosts:
vidtube.site and megaplay.buzz.
- You do not choose or navigate to either provider directly. AniKoto's VidPlay-1 button currently embeds
vidtube.site, while Vidstream-2 embeds megaplay.buzz. The extension needs access only so it can read the video clock inside the selected cross-origin player; either frame remains inert unless a supported show activates it.
- A background fallback locates the active embedded frame by Chrome frame ID and injects the same player adapter when declarative frame matching misses a redirect. It runs only when requested by a supported AniKoto episode.
- It follows the largest visible
<video> and repositions on resize/fullscreen changes.
- It hides common native caption containers when found and adds a narrow dark scrim behind the Mandarin line. Player-specific subtitle markup may require a selector update.
- The supplied real cues cover most spoken dialogue from roughly 11–89 seconds of the capture. A low-confidence exchange around 44–50 seconds is intentionally omitted, and timing remains approximate until checked on the live player.
- If the diagnostic says Mandarin ready · waiting for video to load, the extension is connected but AniKoto's selected player has not produced a playable video yet. That is different from an extension connection failure.
- The integration is fail-open: before a real
<video> exists it does not insert an overlay into the player, and only rendered Chinese word buttons accept pointer input. Reloading the extension no longer leaves a polling script throwing Extension context invalidated; the page asks for one refresh instead.
- AniKoto/provider markup may change. Cross-origin fixture success does not by itself prove live-site compatibility.
- Audio capture is user-initiated, capped at 90 seconds, local, and audio-only. The separate companion script sends only the selected clip to OpenAI after an explicit terminal command. There is no video capture, DRM bypass, analytics, or extension storage.
Validation
Run npm test for manifest and JavaScript validation, cue timing, CC-CEDICT segmentation and adaptation precedence, continuous subtitle rendering, cross-origin interaction, live transcription, and latency regressions.
For fast browserless transcription regression testing, run npm run benchmark. It replays fixed Realtime events against a private 38.4-second Mandarin reference fixture and scores character error rate, coverage, insertions, duplicate finals, and event latency. To send the same audio through the live 24 kHz PCM transcription path without opening Chrome, run OPENAI_API_KEY=... npm run benchmark:live. See benchmark/README.md for the reference-review checklist and thresholds.