media-import-tooling

Helpers for turning MakeMKV rips into a well-formed media library: identifying what every title on a disc actually is, naming and flagging audio and subtitle tracks, and generating thumbnails for extras.

These are small, single-purpose scripts meant to be run by hand while working a disc. Nothing here is a pipeline — the judgement calls stay with the operator.

Which media server?

Built and used against Jellyfin, and the library-layout advice here reflects that. But most of what these scripts do happens below the media server: they read discs, and they write names and flags into Matroska headers with mkvpropedit. A track labelled Commentary by Ben Affleck and Chris Terrio and flagged flag-commentary reads correctly in Jellyfin, Plex, Emby, Kodi, VLC and anything else that understands Matroska, because the metadata lives in the file rather than in a server's database.

So if you are running something else, the identification and tagging tools should work unchanged. What differs is the layout around the files:

	Jellyfin / Emby	Plex
Extras	extras/, plus typed folders (interviews/, featurettes/, …)	typed folders (Featurettes/, Interviews/, …) or -behindthescenes filename suffixes
Trailers	trailers/	Trailers/ or -trailer suffix
Alternate cuts	labelled files in one folder	Movie (Year) {edition-Extended}.mkv
Extra thumbnails	<basename>.jpg beside the video	handled by the server; sidecars are ignored

Patches that make the layout side configurable for other servers are welcome. The per-file work needs no such accommodation — it is already portable.

Requirements

ffmpeg / ffprobe, mkvtoolnix (mkvpropedit, mkvmerge), Python 3. No third-party Python packages; numpy is deliberately not used so the scripts run anywhere.

The scripts

Script	What it answers
lib/scan_titles.py	What's on this disc? Parses a makemkvcon info log into a title table sorted by runtime.
lib/scan_streams.py	What's in one title? Per-stream detail from the same log, including MakeMKV's stream flags.
lib/rip_disc.py	Rip titles, verifying each output is the title that was actually asked for.
lib/rip_result.py	Did that rip actually save anything? makemkvcon exits 0 even when it saves nothing.
lib/tag_episode.py	Name and flag one episode's tracks: commentaries, companion cores, forced subtitles.
lib/probe_mkv.py	What's in the finished file? Stream list with names, languages and flags.
lib/audio_identify.py	Is this unnamed track a commentary, descriptive audio, a dub, or a lossy companion core?
lib/subtitle_inspect.py	Is this subtitle track full or forced, and is it SDH?
lib/pickthumb.py	Pick a representative thumbnail frame for an extra.
lib/mux_commentary.py	Move a commentary off a companion disc into the feature you keep, verifying sync by chapter marks.
lib/ttml2srt.py	Convert a TTML subtitle sidecar (StreamFab/Amazon) to SubRip so it can be muxed.

Typical run against a disc:

makemkvcon -r --cache=512 --minlength=15 info disc:0 > scan.log
python3 lib/scan_titles.py scan.log            # pick the feature, spot duplicates
python3 lib/scan_streams.py scan.log 4         # inspect the feature's streams
python3 lib/rip_disc.py scan.log ./out 4 7 9   # rips AND verifies it got those titles
python3 lib/rip_result.py ./out/riplog-t4.txt --expect ./out/*_t04.mkv
python3 lib/probe_mkv.py feature.mkv
python3 lib/audio_identify.py feature.mkv --main 0:a:0 \
        --candidates 0:a:2 0:a:18 --controls 0:a:3 0:a:5
python3 lib/subtitle_inspect.py feature.mkv --survey   # full vs forced, all streams
python3 lib/subtitle_inspect.py feature.mkv --render 14 --ordinal 0 \
        --diff-against 15                      # SDH residue vs a translation track

scan_streams.py decodes MakeMKV's per-stream flags, which name the companion core outright and mark the derived forced-subtitle views. Trust a set bit; a clear bit means nothing, since plenty of discs leave real commentaries unflagged.

Things that have burned us

Recorded because each one cost real time or produced a wrong answer.

MakeMKV subtitle scan counts are not stream counts. The scan adds a derived (forced only) entry for streams containing forced segments, so scan totals exceed real counts — but not by a clean factor. Guardians Vol. 3: 14 → 7 (all paired). The Untouchables 4K: 52 → 34 (only 18 paired). Argo 4K: 64 → 40. Compare against the finished file, never arithmetic on the scan. Never re-rip on a raw count mismatch.

Discs expose the feature twice. Usually a playlist (with chapters) and a bare .m2ts (without). Rip the playlist. But check the streams before assuming: on the Argo Blu-ray the second copy carried a DTS Express track, which is the secondary-audio codec — that copy was the picture-in-picture playlist, not a bare duplicate.

A failed phase-inversion null does not mean unrelated content. Descriptive audio is a different mix of the same film, so it cannot cancel even though it contains the film. Use envelope correlation plus the silent-gap lift, and always run known dubs as controls. See the table in audio_identify.py.

Envelope correlation measures timing, not content. A foreign dub correlates highly with the original because dubbed dialogue lands where the original dialogue was. It is the lift in the film's silent gaps that separates dub from descriptive audio.

Lossy cores are exposed as separate selectable tracks. DTS-HD MA carries a DTS core, TrueHD carries an AC-3 core. Name the core identically to its parent so the player doesn't offer a mix that isn't distinct. When channel counts differ (7.1 TrueHD with a 5.1 AC-3 companion), say "companion" explicitly rather than mislabelling the channel count.

PGS emits two packets per subtitle. A display segment carrying the bitmap, then a small "clear screen" segment ending it. Counting every packet roughly doubles the apparent cue count, and picking timestamps without filtering lands on clear segments, which render nothing at all. Filter to packets over ~1 KB.

Evenly spaced sampling can miss SDH entirely. Sixteen cues spread across The Adjustment Bureau's featurettes showed nothing but dialogue, which reads as a plain translation track. They were SDH: the sound captions sat in the 11-cue gap between the English and Spanish counts, and sampling never landed on one.

Sound captions are a small fraction of an SDH track and they cluster where the dialogue is not, so uniform sampling is biased against finding them. Diff the cue TIMES against another language instead (--diff-against) and render only the residue -- on Dead Poets Society that isolated 399 of 1763 English cues and landed on [PEOPLE MURMURING], [LAUGHING], BOY 1 [WHISPERING] immediately.

A near-equal count across languages is itself the negative result: a featurette on the same disc ran 286 English against 287 French, residue 2, and those two were timing jitter rather than captions.

ffmpeg 9 dropped zscale and drawtext from the Homebrew build. Both are common in older recipes here: zscale for tone-mapping an HDR grab down to SDR, drawtext for labelling contact sheets. They now fail with "No such filter". The scale filter gained the colour conversion, so a PQ frame tone-maps with scale=...:out_transfer=bt709:out_primaries=bt709:out_color_matrix=bt709; label tiles by filename or ordering instead of burning text in.

Full-range video breaks the MJPEG encoder. Grabbing a still from a file flagged full-range (some Blu-ray AVC) fails with "Non full-range YUV is non-standard" and writes nothing -- in a loop, that is a silently empty sheet. Add format=yuv420p to the filter chain and -pix_fmt yuvj420p to the output.

Subtitles sit in the letterbox bar. On scope films a crop of the lower third misses them entirely. Render full frames.

Extras open on black. Nearly every disc extra begins with black or an ident, so a player generating its own thumbnail gets nothing useful. pickthumb.py skips the head and tail and scores candidates; write the result as a .jpg sidecar beside the video.

signalstats has no YDEV. It emits YLOW/YHIGH/YDIF. Asking for a metric that doesn't exist silently yields zero and any gate built on it passes nothing.

MakeMKV sanitises the disc name when it builds output filenames. The name it reports can contain characters the name it writes does not. A Monty Python disc reported

MONTY PYTHON’S LIFE OF BRIAN     (U+2019, bytes e2 80 99)

and wrote

MONTY PYTHON'S LIFE OF BRIAN_t01.mkv   (ASCII apostrophe, 0x27)

Anything that predicts the output path by pasting the reported name together with the title number therefore looks for a file that was never created. Four good rips were reported FAILED with "0 read errors" while sitting on disk, and the checker deleted nothing only because there was nothing at the path it guessed. Find the output by its _tNN.mkv suffix instead; the suffix is MakeMKV's own and survives any mangling of the leading name.

A title's container duration can be much longer than its video. A Blu-ray menu asset on the same disc held 20 seconds of video (480 frames at 23.976) over a five-minute audio bed, so the container reported 300s against the scan's 20s. That is not a bad rip and not a wrong title -- it is how menus are authored. Compare durations as advisory only; the playlist is what settles identity.

A child process reading stdin eats the script that is driving it. ffmpeg, whisper and makemkvcon all read standard input, and a subprocess inherits the parent's. When these scripts are driven from a shell heredoc, the script is stdin, so a child that consumes a few bytes deletes them from the commands that have not been read yet. The failure surfaces on a LATER line and looks nothing like its cause: a loop over four titles reported

bash: line 14: rab: command not found

because ffmpeg had swallowed the g of the next grab call, and the third title was silently never processed at all. Nothing errors at the call site, and re-running by hand always works, because an interactive shell is not a heredoc.

Every subprocess.run here now passes stdin=subprocess.DEVNULL. Do the same in any wrapper you write, and pass -nostdin to ffmpeg invoked from a shell.

ffmpeg cannot read TTML at all. StreamFab writes Amazon subtitles beside the video as .ttml, and ffmpeg has no TTML demuxer -- it rejects the file with "Invalid data found when processing input", which reads like a corrupt download rather than an unsupported format. Convert first (ttml2srt.py), then mux.

Two details of that flavour of TTML. Styling is by reference -- a <span> names a style id defined in the head, so italics are only visible after resolving it -- and StreamFab substitutes _ for : in filenames, so an episode arrives as Angola_ Survive to Drive and its real title has to be restored before it is written into the container.

Bracketed paths break shell globbing. [tmdbid-120] is a character class, so Python's glob.glob, and sed/parameter expansion in shell, all silently return nothing. Use os.listdir and pass paths as arguments.

makemkvcon exits 0 even when it saves nothing. A rip that hit an unreadable sector printed Copy complete. 0 titles saved, 1 failed. and still returned 0. Driving it in a loop over a season, $? will happily report success for an episode that produced no file. Check with rip_result.py, which parses the completion message and confirms the output exists.

Parse the message codes, not the text — and note that success and failure use different codes. A clean run emits 5036/5005 carrying only a saved count; a run with failures emits 5037/5004 carrying saved and failed. A checker that knows only the failure pair calls every successful rip a failure.

Read errors alone do not condemn the output. The pass that recovered a failed episode still logged 5 of them; MakeMKV retried through, the content hash table verified the M2TS, and the file decoded start to finish. Errors mean the media is marginal, not that the result is bad — but do compare the reported byte offsets across runs. A repeat at the same offset is a fixed defect; a moving one is a marginal read worth retrying.

makemkvcon mkv does not always enumerate the same titles as makemkvcon info. With identical flags on the same disc, info offered 29 titles and mkv offered 26 -- it silently dropped the episode's playlists because their first sector was unreadable. Every index then pointed at a DIFFERENT title, one or more positions along. Twenty titles ripped, every completion message said success, every output file existed, and not one was the title requested; four were duplicates of each other and the episode was absent entirely.

Neither "did it save something" nor "does the file exist" detects this, because both were true. The check that does is comparing each output's DURATION against the duration the scan recorded for that title -- rip_disc.py does this and aborts on the first mismatch rather than the twentieth. Treat a divergence as a reason to stop, not an off-by-one to correct: the omissions need not be contiguous.

A DVD has no playlists, so the rip verification is inert on one. rip_disc.py settles identity by comparing the playlist each output came from against the playlist the scan recorded. DVDs have no .mpls, so the parse finds none and the run reports

### disc: MB_MegaMovieMyths   (scan offered 0 titles)

then rips correctly with that check doing nothing. The duration comparison still runs and is the only guard left; a DVD title whose container duration drifts from the scan is worth a look rather than a shrug. Nothing is wrong with the output -- but do not read "scan offered 0 titles" as a failure, and do not read a clean DVD run as having been verified the way a Blu-ray run is.

Title numbers restart on every disc. Each disc in a set has its own t00, t01, t05. Anything keyed on the title tag alone — a cache, a temp file, a results dict — collides across discs of the same set, and because the stream indices also line up, the wrong data reads as entirely plausible. On a Game of Thrones box set this made disc 2's E04 analyse disc 1's E01 subtitle packets, and the only visible symptom was Italian ending up with a forced track and no full track, which no disc ships. Key on the full disc-qualified filename.

mkvpropedit changes only what you name. It has no notion of resetting a track to defaults, so a second pass that renames tracks leaves every flag from the first pass in place. After a wrong pass this yields correct names over stale flag-forced / flag-commentary bits — the hardest kind of error to see, because everything you look at reads right. Write every flag explicitly as 0 or 1 on every track, so the operation is idempotent and self-correcting.

The last audio track is not necessarily a commentary. It usually is when a disc carries one, but episodes and cuts within one set differ: on a Game of Thrones disc, two episodes ended with an English commentary and the third ended with Russian DTS, because that episode has no commentary at all. Require the trailing track to be the expected language and past the known film tracks before labelling it, and fail loudly rather than guessing.

A missing track can identify content. The inverse is useful: when published sources say which episodes carry commentary, the episode whose audio count is one short is identifiable from the scan log alone, before decoding anything. Cheap corroboration for a runtime-based guess — and on that disc the runtimes were not monotonic with episode order, so runtime alone would not have ordered them.

Runtime difference is not evidence of drift. A catalogue 4K and the Blu-ray in the same package are separate authorings, and their features routinely differ in length. On Dead Poets Society the Blu-ray ran 84.5 s longer than the 4K -- all of it material after the end credits. The films themselves shared a timeline exactly: all ten CHAPTER MARKS matched within 0.17 s, most within one frame.

Compare chapters, not durations. Chapters are authored against the picture, so they measure the thing that actually matters. A duration gate rejects a good mux, and "correcting" for the difference with an 84.5 s delay destroys a track that was already in sync. mux_commentary.py aborts on chapter drift and only reports the duration delta.

mkvmerge groups tracks by type. A track muxed in from a second file does not land at the end of the output: the new audio goes after the existing audio, and every subtitle shifts down one. A verification that expects the original signature followed by the new track therefore fails on a perfectly good file -- the layout to expect is original[:1+n_audio] + new + original[1+n_audio:].

A remux can silently drop Dolby Vision. The side data is what makes a profile 7 file more than HDR10, and nothing in a track listing shows it missing. Check dv_profile, el_present_flag and rpu_present_flag on the first video frame before and after, and treat any change as a failure.

Duplicate titles are not always duplicates. Two menu loops with the same duration and near-identical frames still differed by video-stream MD5 and by 2 MB — two encodes of one montage, not one file twice. Hash the video stream (ffmpeg -map 0:v:0 -c copy -f md5 -) before calling anything a second copy; whole-file hashes are useless here because MakeMKV writes distinct container metadata on every invocation.

flag-forced alone auto-enables nothing. Jellyfin decides per user: the OnlyForced subtitle mode honours flag-forced, but the default mode, confusingly named Default, only loads a track carrying flag-default. On our server five of six accounts sit on Default, so a correctly forced-flagged track showed no subtitles for everyone but one. Forced-narrative tracks need both flags. Set flag-default on only one subtitle track — several default tracks confuse players.

A cue count of zero is not a forced track. The byte threshold in cue counting exists only to discard PGS clear-screen segments, and while it applied to every codec it counted every SubRip track as 0 — which the below-the-median rule then read as "forced". Known text codecs are now counted one packet per cue (TEXT_SUB_CODECS), so a 1,235-cue DVB track and its 2,372-cue SubRip companion both measure. Anything unrecognised keeps the threshold and reads 0, and 0 still means "not measured", never "forced".

Cue count cannot find the forced track on a bilingual show, and this is not fixable by measurement. The heuristic assumes a forced-narrative track is a small fraction of a full one. That holds when a few lines of signage are translated. It is false whenever the show is substantially in another language: Shōgun is around 70% Japanese, so its forced track subtitles nearly every line and lands at roughly the same count as the full English track. Nothing sits below the median, the heuristic reports no forced track, and it does so silently — the failure looks like a file that simply has none. Declare it with --forced-sub N, which bypasses the heuristic for every track and prints that it did. The operator knows which track the service labelled "forced narrative"; the file does not say.

Jellyfin notes

These are Jellyfin-specific; skip them if you run something else.

Extras cannot own NFO metadata (upstream closed as not-planned), so descriptions have to be pushed through the API and written as sidecars separately. Images are picked up from <basename>.jpg beside the video via the local image provider.

Jellyfin 12.0 disables legacy API authorization. Only the Authorization header with the MediaBrowser scheme and the ApiKey query parameter (capital A, capital K) survive; the lowercase api_key parameter and the X-Emby-* headers are rejected. Also: auto-resolved alternate versions are dropped on upgrade and a full library scan is required afterwards to rebuild them.

Pass userId on every item query in 12.x. The auth change is the visible half; this one is silent. On 10.11 a key-only /Items?Recursive=true&IncludeItemTypes=Movie&Limit=0 returned the whole library. On 12.1 the same call returns a FILTERED SUBSET -- items parented under plain Folder rows are no longer traversed -- while ?userId=<admin> returns the true count. A library that reported 1377 movies before an upgrade and 1173 after lost nothing; the query changed meaning. Counts taken key-only before an upgrade are not comparable with counts taken key-only after it, and "missing" items should be checked against the filesystem rather than against that baseline.

Related: /Environment/DirectoryContents needs includeFiles=true&includeDirectories=true. Both default to false, so it returns [] for every path, which reads exactly like a permissions problem.

License

MIT — see LICENSE.