fix: classification errors surfaced by a TV box-set import #4

Merged

nugget merged 5 commits from lessons/got-s1-import into main

2026-08-20 12:41:59 -05:00

claude commented

2026-08-14 15:36:35 -05:00

Collaborator

`feat: decode MakeMKV StreamFlags in scan_streams`

Attribute 22 of each SINFO row is a bitfield that names relationships we had
been inferring by measurement: which lossless track carries an embedded lossy
core (HasCoreAudio), which track is that core (CoreAudio|DerivedStream),
and which subtitle entries are derived forced-only views (ForcedSubtitles).

This settles a case the published specs cannot. Each episode exposes two English
DD 5.1 640 kb/s tracks; blu-ray.com lists one, because cores are never listed
separately. The flags say which is the Atmos companion — confirmed independently
by correlation (r=0.994) and by the other track failing to null against it.

A set bit is evidence; a clear bit is not. These same discs carry commentary
tracks with a flag value of 0. Forced-view detection previously string-matched
description text and now reads the bit, but the scan-vs-output warning is kept
and sharpened: 26 non-derived entries still became 32 output streams.

`fix: classify forced subtitles against the file, not a constant`

The 500-cue rule was calibrated on features. A 60-minute episode's full tracks
run 380–650 cues, so it reclassified Czech (450), Polish (472), Croatian (383),
Hebrew (431) and Slovenian (383) as forced — five languages spanning the entire
runtime.

Replacing the count with track span fails too, and that is the trap worth
recording: a Hungarian forced track carries four cues spread from 518s to 3694s,
covering nearly the whole episode, because the scenes it exists for are spread
out. Coverage says "full"; the track is forced.

What holds is the ratio — full and forced differ by ~100×, so a cut taken from
the file's own median needs no absolute number. New --survey does this in one
demux pass. --count no longer offers a verdict, because a single track in
isolation genuinely cannot be classified.

`fix: separate same-language alternate mixes from foreign dubs`

A remaster often ships its legacy mix alongside the new one. That track lands at
r=0.95–0.98 — below the companion-core threshold, squarely inside the foreign-dub
band. The 2011 broadcast DD 5.1 measured r=0.974/+3.1 dB against the 2018 Atmos,
and r=0.982/+1.5 dB against the Atmos's own core, where it was reported as a
foreign dub. It is English.

Only the language resolves it, so tags are now read from the file and compared
automatically; when the language is unknown the classifier returns the ambiguity
instead of picking a side.

Also widens the dub lift ceiling from 3 to 6. Gap lift is measured against the
window's quiet quartile and rises as that floor falls — the same sensitivity
already documented for companion cores. Known French and German dubs measured
+3.9 and +4.6 dB against a −53 dB floor, so at the old ceiling the German
control classified as UNCLEAR.

`docs: record failure modes from a TV box-set import`

Six entries, including the one that actually wrote bad data to a file:

Title numbers restart on every disc. Anything keyed on the title tag alone
collides across a set, and because stream indices line up too, the wrong data
reads as plausible. Disc 2's E04 analysed disc 1's E01 subtitle packets; the
only symptom was Italian ending up with a forced track and no full track.
mkvpropedit changes only what you name. A second pass renames tracks but
leaves the first pass's flags in place — correct names over stale bits.
The last audio track is not necessarily a commentary (one episode ends on
Russian DTS because it has none).
A missing track can identify content — the episode one track short is the
one without a commentary, readable from the scan log before decoding anything.
Duplicate titles are not always duplicates: two menu loops, identical
duration and near-identical frames, different video MD5s.

Verification

No unit suite, per AGENTS.md; syntax check plus --help on every script, then
each changed script run against real media:

scan_streams.py on a real scan log — reports HasCoreAudio /
CoreAudio,DerivedStream on the expected pair
audio_identify.py on an episode — all five tracks classify correctly
(alternate mix, core, commentary, two dubs); previously two were wrong
subtitle_inspect.py --survey on two episodes — catches all forced tracks and
misclassifies no full track, including both counter-examples above

scan_streams.py now exits 0 from --help, which AGENTS.md documents as a
verification step. pickthumb.py, probe_mkv.py and scan_titles.py still
fail that check on main — pre-existing, left alone as out of scope for this
change, but worth a follow-up since the repo currently fails its own documented
gate.

Four corrections surfaced while importing a Game of Thrones season 1 4K box set — the first TV material this tooling has been pointed at. Three of them are cases where a script returned a confident wrong answer rather than an error. ### `feat: decode MakeMKV StreamFlags in scan_streams` Attribute 22 of each `SINFO` row is a bitfield that names relationships we had been inferring by measurement: which lossless track carries an embedded lossy core (`HasCoreAudio`), which track *is* that core (`CoreAudio|DerivedStream`), and which subtitle entries are derived forced-only views (`ForcedSubtitles`). This settles a case the published specs cannot. Each episode exposes two English DD 5.1 640 kb/s tracks; blu-ray.com lists one, because cores are never listed separately. The flags say which is the Atmos companion — confirmed independently by correlation (r=0.994) and by the other track failing to null against it. **A set bit is evidence; a clear bit is not.** These same discs carry commentary tracks with a flag value of `0`. Forced-view detection previously string-matched description text and now reads the bit, but the scan-vs-output warning is kept and sharpened: 26 non-derived entries still became 32 output streams. ### `fix: classify forced subtitles against the file, not a constant` The 500-cue rule was calibrated on features. A 60-minute episode's full tracks run 380–650 cues, so it reclassified Czech (450), Polish (472), Croatian (383), Hebrew (431) and Slovenian (383) as forced — five languages spanning the entire runtime. Replacing the count with track *span* fails too, and that is the trap worth recording: a Hungarian forced track carries four cues spread from 518s to 3694s, covering nearly the whole episode, because the scenes it exists for are spread out. Coverage says "full"; the track is forced. What holds is the ratio — full and forced differ by ~100×, so a cut taken from the file's own median needs no absolute number. New `--survey` does this in one demux pass. `--count` no longer offers a verdict, because a single track in isolation genuinely cannot be classified. ### `fix: separate same-language alternate mixes from foreign dubs` A remaster often ships its legacy mix alongside the new one. That track lands at r=0.95–0.98 — below the companion-core threshold, squarely inside the foreign-dub band. The 2011 broadcast DD 5.1 measured r=0.974/+3.1 dB against the 2018 Atmos, and r=0.982/+1.5 dB against the Atmos's own core, **where it was reported as a foreign dub. It is English.** Only the language resolves it, so tags are now read from the file and compared automatically; when the language is unknown the classifier returns the ambiguity instead of picking a side. Also widens the dub lift ceiling from 3 to 6. Gap lift is measured against the window's quiet quartile and rises as that floor falls — the same sensitivity already documented for companion cores. Known French and German dubs measured +3.9 and +4.6 dB against a −53 dB floor, so at the old ceiling the **German control classified as UNCLEAR**. ### `docs: record failure modes from a TV box-set import` Six entries, including the one that actually wrote bad data to a file: - **Title numbers restart on every disc.** Anything keyed on the title tag alone collides across a set, and because stream indices line up too, the wrong data reads as plausible. Disc 2's E04 analysed disc 1's E01 subtitle packets; the only symptom was Italian ending up with a forced track and no full track. - **`mkvpropedit` changes only what you name.** A second pass renames tracks but leaves the first pass's flags in place — correct names over stale bits. - **The last audio track is not necessarily a commentary** (one episode ends on Russian DTS because it has none). - **A missing track can identify content** — the episode one track short is the one without a commentary, readable from the scan log before decoding anything. - **Duplicate titles are not always duplicates**: two menu loops, identical duration and near-identical frames, different video MD5s. --- ### Verification No unit suite, per `AGENTS.md`; syntax check plus `--help` on every script, then each changed script run against real media: - `scan_streams.py` on a real scan log — reports `HasCoreAudio` / `CoreAudio,DerivedStream` on the expected pair - `audio_identify.py` on an episode — all five tracks classify correctly (alternate mix, core, commentary, two dubs); previously two were wrong - `subtitle_inspect.py --survey` on two episodes — catches all forced tracks and misclassifies no full track, including both counter-examples above `scan_streams.py` now exits 0 from `--help`, which `AGENTS.md` documents as a verification step. **`pickthumb.py`, `probe_mkv.py` and `scan_titles.py` still fail that check on `main`** — pre-existing, left alone as out of scope for this change, but worth a follow-up since the repo currently fails its own documented gate.

claude added 4 commits

2026-08-14 15:36:35 -05:00

feat: decode MakeMKV StreamFlags in scan_streams 9621e83e46

Attribute 22 of each SINFO row is a bitfield naming the relationships that
were previously reached only by measurement: which lossless track carries an
embedded lossy core (HasCoreAudio), which track is that core (CoreAudio|
DerivedStream), and which subtitle entries are derived forced-only views
(ForcedSubtitles) rather than real streams.

This settles a case the published disc specs cannot. A Game of Thrones UHD
episode exposes two English DD 5.1 640 kb/s tracks; blu-ray.com lists one,
because cores are never listed separately. The flags identify which is the
Atmos companion, confirmed independently by correlation (r=0.994) and by the
other track failing to null against it (-32.9 dB residual).

Forced-view detection previously string-matched the description text; it now
reads the bit. The scan-vs-output warning is kept and sharpened, because the
count still does not predict extraction: 26 non-derived entries became 32
output streams on the same episode.

Also handles --help, which AGENTS.md documents as a verification step.

fix: classify forced subtitles against the file, not a constant 4bd2561d9c

The 500-cue rule was calibrated on features. A 60 minute episode's full
tracks run 380-650 cues, so on a Game of Thrones disc it reclassified
Czech (450), Polish (472), Croatian (383), Hebrew (431) and Slovenian (383)
as forced-narrative -- five languages that span the entire runtime.

Replacing it with track span does not work either, and that is the trap
worth recording: a Hungarian forced track carries four cues spread from
518s to 3694s, covering nearly the whole episode, because the scenes it
exists for are spread out. Coverage says 'full'; the track is forced.

What holds is the ratio. Full and forced differ by roughly two orders of
magnitude, so a cut taken from the file's own median separates them with
enormous margin and needs no absolute number -- it survived features, a
62 minute episode with 6 forced tracks, and a 52 minute one with 9.

--survey does this in a single demux pass. --count no longer offers a
verdict, because one track in isolation genuinely cannot be classified.

Also records that SDH punctuation is not universal: these discs caption
'(Horses neighing)', not '[horses neighing]'.

fix: separate same-language alternate mixes from foreign dubs 489c14fa70

A remaster often ships its legacy mix alongside the new one. That track
correlates 0.95-0.98 against the remaster -- below the companion-core
threshold, squarely inside the foreign-dub band -- and no measurement here
tells the two apart. A Game of Thrones UHD carries the 2011 broadcast DD 5.1
beside the 2018 Atmos: r=0.974/+3.1 dB against the Atmos, and r=0.982/+1.5 dB
against the Atmos's own AC-3 core, where it was reported as a foreign dub.
It is English.

Only the language resolves it, so the tags are now read from the file and
compared automatically, and the reading is printed with each track's language.
When the language cannot be determined the classifier returns the ambiguity
instead of picking a side.

Also widens the dub lift ceiling from 3 to 6. The gap lift is measured against
the chosen window's quiet quartile and rises as that floor falls -- the same
sensitivity already documented for companion cores. Known French and German
dubs measured +3.9 and +4.6 dB against a -53 dB floor, so at the old ceiling
the German control classified as UNCLEAR. Nothing else reaches r >= 0.88;
descriptive audio topped out at 0.809 on Project Hail Mary.

The phase-inversion null, useless for descriptive audio, does separate a core
from an alternate mix: -32.9 dB residual here against roughly -60 for a
same-encode null.

docs: record failure modes from a TV box-set import 9a6f01b1df

Six entries, each of which produced a wrong answer or wrote wrong data:

- Title numbers restart on every disc, so anything keyed on the title tag
  alone collides across a set. Stream indices line up too, which is what
  makes the wrong data look plausible.
- mkvpropedit changes only what you name, so re-running after a bad pass
  leaves stale flags under correct names.
- The last audio track is not necessarily a commentary.
- A missing track can identify content: the episode one track short is the
  one without a commentary, readable from the scan log alone.
- Two titles with identical duration and near-identical frames were still
  different encodes; compare video-stream MD5, not whole-file hashes.

Also points the usage example at subtitle_inspect --survey.

claude added 1 commit

2026-08-15 15:34:35 -05:00

feat: add rip_result, because makemkvcon exits 0 on total failure ab48f00034

A rip that hit an unreadable sector printed

    Copy complete. 0 titles saved, 1 failed.

and still returned 0. Driving makemkvcon in a loop over a season, $? reports
success for an episode that produced no file, and the season is quietly short
one episode. That is how S01E09 was nearly lost: the wrapper trusted the exit
code and the monitor duly reported 'exit=0' for a title that did not exist.

rip_result.py parses the completion message and confirms the output file
exists, exiting non-zero when either check fails.

It parses message CODES rather than rendered text, since the text is
localisable and a locale change would make a failed rip look successful.
Critically, SUCCESS AND FAILURE USE DIFFERENT CODES: a clean run emits
5036/5005 carrying only a saved count and never emits 5037 at all. The first
version of this handled only the failure pair and therefore reported every
successful rip as a failure -- caught by testing against a real success log
as well as a real failure log.

Read errors are reported but do not by themselves fail the check. The pass
that recovered S01E09 logged 5 of them; MakeMKV retried through, the content
hash table verified the M2TS, and the file decoded start to finish. Repeated
byte offsets are surfaced because a fixed offset across runs indicates a real
defect while a moving one indicates a marginal read worth retrying.