diff --git a/.github/instructions/ai-doc-attribution.instructions.md b/.github/instructions/ai-doc-attribution.instructions.md new file mode 100644 index 0000000..8b88b1d --- /dev/null +++ b/.github/instructions/ai-doc-attribution.instructions.md @@ -0,0 +1,18 @@ +--- +description: "Use when writing or updating documentation (Markdown, README, docs pages, guides). Require explicit top-of-document labeling when a document is fully AI-generated." +name: "AI Documentation Attribution" +applyTo: "**/*.md" +--- +# AI Documentation Attribution + +- When documentation is fully AI-generated, include an explicit attribution note. +- The attribution must be visible in the document body and easy to find by readers. +- Acceptable labels include one of: + 1. "AI-generated documentation" +- Place the attribution at the top of the document by default. +- If only parts are AI-assisted, attribution is optional unless you want to disclose assistance. +- Do not imply fully human authorship for content produced by AI. + +Example attribution lines: + +- `> Note: This document was AI-generated and reviewed by a maintainer.` diff --git a/README.md b/README.md index cbe8b2b..724d32e 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,8 @@ This repo supports `x86_64-linux` and (theorically) `aarch64-linux`. Please see [our setup guide](./docs/setting-up.md) for more information on how to onboard a new user or system. +For the media request stack on palatine-hill, see [the media stack guide](./docs/media-stack.md). + ## For Those Interested Although we are not actively looking for new members to join in on this repo, diff --git a/docs/media-stack.md b/docs/media-stack.md new file mode 100644 index 0000000..72a40e9 --- /dev/null +++ b/docs/media-stack.md @@ -0,0 +1,422 @@ +# Media Request Stack Setup + +> Note: This is AI-generated documentation and was reviewed by a maintainer. + +This page documents the setup needed to make media requests flow from Jellyseerr to the Starr apps to qBittorrent and finally into a Jellyfin library. + +It is based on the services defined for palatine-hill in: + +- `systems/palatine-hill/docker/arr.nix` +- `systems/palatine-hill/docker/torr.nix` +- `systems/palatine-hill/postgresql.nix` +- `systems/palatine-hill/vars.nix` + +The guidance here follows the same hardlink principles used by TRaSH Guides: keep downloads and library folders separate, but make sure they live on the same filesystem and appear under the same container path. + +## What Exists In This Repo + +The media-request side currently defines these containers on palatine-hill: + +- Jellyseerr on port `5055` +- Prowlarr on port `9696` +- Radarr on port `7878` +- Sonarr on port `8989` +- Lidarr on port `8686` +- Bazarr on port `6767` +- qBittorrent variants in `docker/torr.nix` + +Related supporting details: + +- The Starr apps and qBittorrent both mount `/data` from `vars.primary_torr`. +- PostgreSQL is enabled locally and used by the arr stack. + +Two caveats matter before expecting the flow to work: + +1. Jellyfin is not currently defined on palatine-hill in this repo, so this guide treats Jellyfin as the destination media server you will point at the finished library. +2. qBittorrent is using host-exposed or gluetun-attached networking rather than `arrnet`, so the Starr apps should connect to qBittorrent through the host and published port. + +## Required Hardlink Layout + +For hardlinks and atomic moves to work reliably, these rules need to be true: + +- qBittorrent and the Starr apps must see the same underlying host filesystem and the same ZFS dataset. +- qBittorrent and the Starr apps should use the same in-container prefix, ideally `/data`. +- Downloads and the final library must be separate directories. +- Jellyfin should only read the final media library, not the download directories. + +For ZFS specifically, sibling child datasets in the same pool are not enough. Hardlinks do not cross dataset boundaries, so `/data/torrents` and `/data/media` must be directories inside the same dataset. + +Recommended logical layout inside containers: + +```text +/data +├── torrents +│ ├── movies +│ ├── music +│ └── tv +└── media + ├── movies + ├── music + └── tv +``` + +This repo draft uses one shared host root from `vars.primary_torr` and mounts that as `/data` for qBittorrent, Radarr, Sonarr, Lidarr, Bazarr, Unpackerr, and Notifiarr. + +### What Matters + +The exact host path is less important than this invariant: + +```text +same host filesystem + same container path prefix + separate downloads/media folders +``` + +If you split torrents and media across different datasets, imports may still be made to work with copies or path fixes, but hardlinks and instant moves will not be dependable. + +## Suggested Host Layout + +Once you choose a shared host root, create a structure like this beneath it: + +```text +data/ +├── torrents/ +│ ├── movies/ +│ ├── music/ +│ └── tv/ +└── media/ + ├── movies/ + ├── music/ + └── tv/ +``` + +In this repo draft, the shared host root is `vars.primary_torr`, with container mounts set to `"${vars.primary_torr}/data:/data"`. + +The matching container paths should then be: + +- qBittorrent download root: `/data/torrents` +- Radarr root folder: `/data/media/movies` +- Sonarr root folder: `/data/media/tv` +- Lidarr root folder: `/data/media/music` +- Jellyfin library roots: `/data/media/movies`, `/data/media/tv`, `/data/media/music` + +Do not point any Starr app root folder at `/data/torrents`. + +## Service Roles + +### Jellyseerr + +Jellyseerr is the user-facing request layer. It should: + +- connect to Jellyfin for users, authentication, and media availability +- connect to Radarr for movies +- connect to Sonarr for series + +Jellyseerr does not talk directly to qBittorrent for normal request flow. + +### Prowlarr Values + +Prowlarr should be the single source of indexers. Configure indexers there, then sync them to: + +- Radarr +- Sonarr +- Lidarr + +This avoids duplicating indexer setup in every Starr app. + +### Radarr, Sonarr, Lidarr + +These apps should: + +- receive requests from Jellyseerr +- search indexers via Prowlarr +- send downloads to qBittorrent +- import completed downloads from `/data/torrents/...` into `/data/media/...` + +### qBittorrent + +qBittorrent should only download into `/data/torrents/...` and should not write directly into `/data/media/...`. + +### Jellyfin + +Jellyfin should only read the final library under `/data/media/...`. + +## Configuration Order + +Set the stack up in this order: + +1. Shared path layout +2. qBittorrent +3. Prowlarr +4. Radarr, Sonarr, Lidarr +5. Jellyfin +6. Jellyseerr +7. Bazarr + +That order keeps each layer pointing at services that already exist. + +## qBittorrent Setup + +The repo defines these Web UI ports: + +- `8082` for `qbit` +- `8081` for `qbitVPN` +- `8083` for `qbitPerm` + +Choose one instance for the Starr apps to use and keep that consistent. + +Recommended qBittorrent settings: + +- Default save path: `/data/torrents` +- Category mode: enabled +- Automatic torrent management: enabled +- Incomplete directory: optional, but avoid a different filesystem if you want cheap moves +- Listening port: use the instance-specific torrent port if applicable + +Recommended categories: + +- `radarr` -> `/data/torrents/movies` +- `sonarr` -> `/data/torrents/tv` +- `lidarr` -> `/data/torrents/music` + +This matches the TRaSH pattern and keeps imports predictable. + +## Prowlarr Setup + +In Prowlarr: + +1. Add your indexers. +2. Add app connections for Radarr, Sonarr, and Lidarr. +3. Sync indexers from Prowlarr into each Starr app. + +Use the container hostnames from the repo when apps share the `arrnet` network: + +- `http://radarr:7878` +- `http://sonarr:8989` +- `http://lidarr:8686` + +If you are configuring through host-exposed ports in a browser from outside Docker, use the server host and published ports instead. + +## Radarr Setup + +In Radarr: + +1. Add a root folder: `/data/media/movies` +2. Add qBittorrent as the download client +3. Set the category to `radarr` +4. Prefer completed download handling on +5. Do not use a movie root inside the downloads tree + +For qBittorrent, use the chosen instance endpoint. + +Examples: + +- preferred for this repo draft: `http://:8082` +- VPN-backed alternative if you intentionally use that instance: `http://:8081` + +The important part is that the path qBittorrent writes must still be visible to Radarr as `/data/torrents/movies`. + +## Sonarr Setup + +In Sonarr: + +1. Add a root folder: `/data/media/tv` +2. Add qBittorrent as the download client +3. Set the category to `sonarr` +4. Enable completed download handling + +Keep the same shared-path rule: Sonarr must be able to see qBittorrent output directly at `/data/torrents/tv`. + +## Lidarr Setup + +In Lidarr: + +1. Add a root folder: `/data/media/music` +2. Add qBittorrent as the download client +3. Set the category to `lidarr` +4. Enable completed download handling + +## Jellyfin Setup + +Jellyfin should be pointed only at the final library paths: + +- Movies: `/data/media/movies` +- TV: `/data/media/tv` +- Music: `/data/media/music` + +Do not add `/data/torrents` as a Jellyfin library. + +If Jellyfin runs in Docker, mount only the media sub-tree if you want a tighter boundary: + +- `host-shared-root/media:/data/media` + +If Jellyfin runs directly on the host, point it at the equivalent host paths. + +## Jellyseerr Setup + +Jellyseerr in this repo runs on port `5055` and joins both `arrnet` and `haproxy-net`. + +Configure it with: + +1. Jellyfin server URL +2. Jellyfin API key +3. Radarr server URL and API key +4. Sonarr server URL and API key + +Suggested internal URLs when services share `arrnet`: + +- Radarr: `http://radarr:7878` +- Sonarr: `http://sonarr:8989` + +Jellyseerr request defaults should map: + +- Movies -> Radarr root `/data/media/movies` +- Series -> Sonarr root `/data/media/tv` + +After that, user flow is: + +1. User requests media in Jellyseerr +2. Jellyseerr hands the request to Radarr or Sonarr +3. The Starr app searches via Prowlarr indexers +4. The Starr app sends the download to qBittorrent with its category +5. qBittorrent writes into `/data/torrents/...` +6. The Starr app imports into `/data/media/...` +7. Jellyfin scans or detects the new item in the final library + +## Bazarr Setup + +Bazarr is optional for the request-to-library path, but it fits after Radarr and Sonarr are stable. + +Point Bazarr at: + +- Radarr +- Sonarr +- the final media library visible under `/data/media` + +It does not need the download tree for ordinary subtitle management. + +## Remote Path Mappings + +If you align the mounts properly, you should not need remote path mappings. + +That is the preferred setup. + +Only use remote path mappings if the downloader and the importing app see different absolute paths for the same files. +In a Docker-only setup with shared `/data`, that is a sign the mounts are wrong rather than a feature you should rely on. + +## ZFS Notes + +For a hardlink-safe media layout on ZFS: + +- Keep `/data/torrents` and `/data/media` in the same dataset. +- Do not split them into separate child datasets if you want hardlinks. +- It is fine to keep qBittorrent config, Jellyfin metadata, and other appdata in separate datasets because those do not need hardlinks with payload files. + +For `ZFS-primary/torr`, a better baseline for bulk media than a small-record, high-compression profile is: + +- `recordsize=1M` +- `compression=zstd-3` or `lz4` +- `sync=standard` +- `logbias=throughput` +- `primarycache=metadata` +- `dnodesize=auto` + +These are new-write behavior settings. `recordsize` only affects newly written data. + +## Repo-Specific Notes + +- Arr containers use `PUID=600` and `PGID=100`. +- qBittorrent containers also use `PUID=600` and `PGID=100`. +- The arr stack uses the local PostgreSQL service via `/var/run/postgresql`. +- `jellyseerr` stores config under `${vars.primary_docker}/overseerr` even though the container is Jellyseerr. +- The hardlink draft in this repo chooses `vars.primary_torr` as the shared `/data` root. + +- `systems/palatine-hill/docker/default.nix` imports `torr.nix`, so the downloader stack is part of the host configuration. + +## Deployment Checklist (Exact Values) + +Use this checklist when configuring the stack so every app matches the current draft. + +### Shared Paths + +- Shared container path for arr + downloader: `/data` +- Download root: `/data/torrents` +- Media roots: +- Movies: `/data/media/movies` +- TV: `/data/media/tv` +- Music: `/data/media/music` + +### qBittorrent (Primary Instance) + +- Web UI URL for Starr apps: `http://:8082` +- Web UI port: `8082` +- Torrent port: `29432` (TCP/UDP) +- Default save path: `/data/torrents` +- Category save-path mode: enabled +- Automatic torrent management: enabled + +Category paths: + +- `radarr` -> `/data/torrents/movies` +- `sonarr` -> `/data/torrents/tv` +- `lidarr` -> `/data/torrents/music` + +### Radarr + +- URL: `http://radarr:7878` (inside arr network) +- Root folder: `/data/media/movies` +- Download client: qBittorrent at `http://:8082` +- qBittorrent category: `radarr` +- Completed download handling: enabled + +### Sonarr + +- URL: `http://sonarr:8989` (inside arr network) +- Root folder: `/data/media/tv` +- Download client: qBittorrent at `http://:8082` +- qBittorrent category: `sonarr` +- Completed download handling: enabled + +### Lidarr + +- URL: `http://lidarr:8686` (inside arr network) +- Root folder: `/data/media/music` +- Download client: qBittorrent at `http://:8082` +- qBittorrent category: `lidarr` +- Completed download handling: enabled + +### Prowlarr + +- URL: `http://prowlarr:9696` (inside arr network) +- App sync targets: +- `http://radarr:7878` +- `http://sonarr:8989` +- `http://lidarr:8686` + +### Jellyseerr Values + +- URL: `http://jellyseerr:5055` (internal) or via your reverse proxy externally +- Radarr target: `http://radarr:7878` +- Sonarr target: `http://sonarr:8989` +- Request defaults: +- Movies root: `/data/media/movies` +- Series root: `/data/media/tv` + +### Jellyfin Values + +- Library roots only: +- `/data/media/movies` +- `/data/media/tv` +- `/data/media/music` +- Do not add `/data/torrents` as a library. + +## Validation Checklist + +Use this after setup: + +1. qBittorrent can create files in `/data/torrents/movies`, `/data/torrents/tv`, and `/data/torrents/music`. +2. Radarr, Sonarr, and Lidarr can browse both `/data/torrents/...` and `/data/media/...`. +3. A test download lands in the expected category folder. +4. The corresponding Starr app imports the item into `/data/media/...` without copy-delete behavior. +5. Jellyfin can see the imported file in the final library. +6. Jellyseerr shows the item as available after import and scan. + +If imports fail or hardlinks do not work, check the mount design before changing app logic. diff --git a/systems/palatine-hill/configuration.nix b/systems/palatine-hill/configuration.nix index f3d3eb4..11f7f7e 100644 --- a/systems/palatine-hill/configuration.nix +++ b/systems/palatine-hill/configuration.nix @@ -34,8 +34,7 @@ loader.grub.device = "/dev/sda"; useSystemdBoot = true; kernelParams = [ - "i915.force_probe=56a5" - "i915.enable_guc=2" + "xe.force_probe=56a5" ]; kernel.sysctl = { "vm.overcommit_memory" = lib.mkForce 1; diff --git a/systems/palatine-hill/docker/arr.nix b/systems/palatine-hill/docker/arr.nix index 0e61e7c..f188d43 100644 --- a/systems/palatine-hill/docker/arr.nix +++ b/systems/palatine-hill/docker/arr.nix @@ -5,6 +5,7 @@ }: let vars = import ../vars.nix; + shared_data_path = "${vars.primary_torr}/data"; arr_postgres_config = container_type: let @@ -62,7 +63,7 @@ in ]; volumes = [ "${vars.primary_docker}/bazarr:/config" - "${vars.primary_plex_storage}/data:/data" + "${shared_data_path}:/data" "/var/run/postgresql:/var/run/postgresql" ]; extraOptions = [ @@ -110,7 +111,7 @@ in ]; volumes = [ "${vars.primary_docker}/radarr:/config" - "${vars.primary_plex_storage}/data:/data" + "${shared_data_path}:/data" "/var/run/postgresql:/var/run/postgresql" ]; extraOptions = [ @@ -134,7 +135,7 @@ in ]; volumes = [ "${vars.primary_docker}/sonarr:/config" - "${vars.primary_plex_storage}/data:/data" + "${shared_data_path}:/data" "/var/run/postgresql:/var/run/postgresql" ]; extraOptions = [ @@ -158,7 +159,7 @@ in ]; volumes = [ "${vars.primary_docker}/lidarr:/config" - "${vars.primary_plex_storage}/data:/data" + "${shared_data_path}:/data" "/var/run/postgresql:/var/run/postgresql" ]; extraOptions = [ @@ -176,7 +177,7 @@ in }; volumes = [ "${vars.primary_docker}/unpackerr:/config" - "${vars.primary_plex_storage}:/data" + "${shared_data_path}:/data" "/var/run/postgresql:/var/run/postgresql" ]; extraOptions = [ "--network=arrnet" ]; @@ -194,7 +195,7 @@ in environmentFiles = [ config.sops.secrets."docker/notifiarr".path ]; volumes = [ "${vars.primary_docker}/notifiarr:/config" - "${vars.primary_plex_storage}:/data" + "${shared_data_path}:/data" "/var/run/postgresql:/var/run/postgresql" ]; extraOptions = [ "--network=arrnet" ]; diff --git a/systems/palatine-hill/docker/default.nix b/systems/palatine-hill/docker/default.nix index b365625..9aab11d 100644 --- a/systems/palatine-hill/docker/default.nix +++ b/systems/palatine-hill/docker/default.nix @@ -1,9 +1,4 @@ -{ - config, - lib, - pkgs, - ... -}: +{ ... }: { imports = [ @@ -20,7 +15,7 @@ ./nextcloud.nix # ./postgres.nix # ./restic.nix - #./torr.nix + ./torr.nix # ./unifi.nix ]; diff --git a/systems/palatine-hill/docker/nextcloud.nix b/systems/palatine-hill/docker/nextcloud.nix index ea3fe70..9881792 100644 --- a/systems/palatine-hill/docker/nextcloud.nix +++ b/systems/palatine-hill/docker/nextcloud.nix @@ -58,6 +58,7 @@ in volumes = [ "${nextcloud_path}/nc_data:/var/www/html:ro" ]; extraOptions = [ "--device=/dev/dri:/dev/dri" + "--network=nextcloud_default" ]; }; collabora-code = {