Scanning

A scan walks a library’s root and writes what it finds into the namespace’s catalog. This guide describes what the walk reads, when it runs, how it removes what is gone, and what happens when a Library is deleted.

What a scan reads

The scanner reads the layout Kodi and Jellyfin read, so a volume those players already organize needs no change.

Movies

One folder per title. A folder is a title folder when it holds movie.nfo or a video file. Any other folder is a grouping folder, and the walk descends through it, up to eight levels deep:

movies-pvc/
  Action/
    Example Movie (2019)/
      Example Movie (2019).mkv
      movie.nfo
      folder.jpg
      fanart.jpg
      Example Movie (2019).trickplay/
      Extras/
        Making Of.mkv
      Trailers/
        Example Movie (2019)-trailer.mkv

A readable movie.nfo with a title names the movie. Without one, the folder name is parsed as Title (Year) or Title [Year], or cut at the first release token such as bluray or x264. A folder with no sidecar and no year is counted in status.unidentified and cataloged under its folder name.

Series

One folder per series, directly under the root. Episodes are the files in the series folder and in its season folders, one level down:

series-pvc/
  Example Series/
    tvshow.nfo
    folder.jpg
    Season 02/
      season02-poster.jpg
      Example Series - S02E05.mkv
      Example Series - S02E05.nfo
      Example Series - S02E05-thumb.jpg
      Example Series - S02E05.en.srt
    Specials/
      Example Series - S00E01.mkv

Season NN is season N, and Specials is season 0. An episode’s number comes from a marker in its file name, s02e05 or 2x05, and a range such as s04e10-e11 names two episodes in one file. Both play the file from the start, because nothing on the volume marks where the second begins. The season comes from the folder first, then from the marker.

Extras and file kinds

A folder named extras, featurettes, trailers, behind the scenes, deleted scenes, interviews, scenes, shorts, clips, or other beside a feature or a season is read one level deep. Videos under trailers are trailers, and the rest are extras.

Every file gets a row. The scanner classifies each one as video, audio, subtitle, image, metadata, trickplay, or other from its name, its folder, and one stat. It opens no file to classify it. A subtitle’s language is the tag in its name, and hi after a language tag marks it hearing-impaired. Dot-named entries, Thumbs.db, desktop.ini, and the trash and service directories of common NAS systems are skipped.

Sidecars and art

The scanner reads movie.nfo, tvshow.nfo, and the .nfo beside each episode, leniently, because Jellyfin writes bare ampersands in URLs. It reads the title, the year, the plot, the genres, the people, the ratings, and the provider ids in uniqueid elements, with imdbid, tmdbid, and tvdbid as fallbacks.

Art uses Kodi’s names: poster.jpg, fanart.jpg, clearlogo.png, clearart.png, banner.jpg, landscape.jpg, and disc.png in the title folder, season02-poster.jpg beside tvshow.nfo, and <episode>-thumb.jpg beside the episode. The scanner also accepts folder.jpg and name-prefixed forms such as <title>-poster.jpg.

The .liken/ directory

Beside a title, a dot-named directory holds what the sidecar has no element for: one YAML file per fact, named for the fact. identity.yaml holds the provider ids, or the candidates left for a person to choose from. arrival.yaml holds when each video file was first seen. Every other <fact>.yaml holds what that fact wrote, which provider answered, and its attempts. One file per writer is what lets several enrichers run at once on a network mount with no locks. The scan reads these files and never writes them.

.contributors/

At the library root, one directory per credited person, sharded by the first two characters of the person’s slug. Each holds contributor.yaml with the name and the provider ids, and, once the enricher fills them, biography.txt and headshot.jpg. The walk reads this directory after the titles. It is the one dot-named directory the walk enters.

When a scan runs

Every scan is a Job. The full walk runs from a CronJob named <library>-scan on spec.scan.schedule, once an hour by default. A walk that runs past its next turn skips that turn, because the catalog claim admits one writer. A webhook runs a one-off Job that rescans one folder.

kubectl -n media get cronjob movies-scan
kubectl -n media get jobs -l library.liken.sh/library=movies,library.liken.sh/worker=scan
kubectl -n media create job movies-scan-now --from=cronjob/movies-scan

A scan Job writes a runs row when it starts and another when it finishes, then waits until the namespace’s reporter echoes that run back over the bus before it exits. So a Job that completed is a Job whose counts reached the Library’s status.

Mark and sweep

Each full walk has an epoch. The walk marks every id, path, and link it reads with that epoch, and a prune pass deletes every row of this library the epoch did not mark, in batches of five hundred.

Two guards keep a bad walk from emptying a library. A walk that could not read every directory, or that found less than half of what the catalog holds, is incomplete: it writes what it read and prunes nothing, and the log reports it:

incomplete walk: could not read the whole volume, keeping the last counts

A prune whose epoch marked nothing at all is refused as an error. status.removedLastSweep reports what the last sweep removed, so a mass delete is visible without a shell.

The walk itself runs eight workers over a shared pool of directories, which keeps a network volume busy without a burst large enough to slow a player.

Deleting a Library

A Library carries a finalizer, and deleting it starts a departure that removes its rows from the namespace’s catalog. The operator deletes the CronJob, waits for any scan or enrich Job to finish, then runs a cleanup Job named <library>-cleanup that deletes the rows in batches through its own catalog agent. The finalizer clears once the cleanup Job succeeded and the reporter echoed its run back.

While this runs, the phase is Departing, and the Departing condition names the step: ScanRunning, EnrichRunning, Sweeping, AwaitingEcho, or Blocked when the cleanup Job keeps failing or the namespace holds two Catalogs. There is no timeout. The operator reports the blocker for as long as the object is deleting.

A namespace with no Catalog releases at once, because nothing there holds the rows. A library whose own catalog claim is already gone gets a fresh, empty one for the cleanup Job, whose agent receives the rows over gossip and then sweeps them.

Reading progress

kubectl -n media get library movies -o jsonpath='{.status.phase} titles={.status.titles} unidentified={.status.unidentified} waiting={.status.waiting} gaps={.status.gaps}{"\n"}'
kubectl -n media logs -l library.liken.sh/library=movies,library.liken.sh/worker=scan -c scanner --tail=100

A finished walk logs its counts:

walk complete: 128 titles from 131 folders, 3 unidentified, 0 removed, in 41s

status.unidentified counts the folders cataloged by name. status.waiting counts the titles a provider returned candidates for, and a scan does not retry those until a person names the right uniqueid in the .nfo. status.gaps counts, per fact, the rows the enricher still has to fill.