Refactor patches, add companion patches, modify readme

README.md

@@ -27,13 +27,13 @@ If you NEVER used Invidious before, you can safely start with my fork, but you w
 
 You'll have to set these up yourself:
 
-- Docker Compose V2
+- Docker Compose V2 (this is still not included on many default docker installations so make sure you actually have the plugin version 2).
-- PostgreSQL 16 Server
+- PostgreSQL 16 Server.
-- Nginx Web Server (1.26 or higher for quic support)
+- Nginx Web Server (1.26 or higher for quic support).
-- HAProxy 2.x with my config under etc/haproxy/haproxy.conf
+- HAProxy 2.x with my config under `etc/haproxy/haproxy.conf`.
-- Redis Server for the video cache
+- Redis Server for the video cache.
-- (optional) PGBouncer, recommended if you want to host a public instance to pool connections to PSQL, otherwise you might run into DB pool errors
+- (optional) PGBouncer, recommended if you want to host a public instance to pool connections to PSQL, otherwise you might run into DB pool errors.
-- (optional) Anubis https://github.com/TecharoHQ/anubis, recommended if you want to host a public instance and not get DDoS'ed from bots and ai
+- (optional) Anubis https://github.com/TecharoHQ/anubis, recommended if you want to host a public instance and not get DDoS'ed from bots and ai.
 
 # For public instances
 
@@ -43,32 +43,49 @@ I will not give any recommendations here because this is something you should re
 
 A few important infos:
 
-- Datacenter IPv4 proxies will not work
+- Datacenter IPv4 proxies will not work.
-- Static residential IPv4 proxies will not work
+- Static residential IPv4 proxies will not work.
-- Most VPN providers that do not allow dynamically switching target server or target IP will not work
+- Most VPN providers that do not allow dynamically switching target server or target IP will not work.
-- Datacenter IPv6 proxies with at most a /40 CIDR subnet will work if they allow unlimited IP rotation in that subnet
+- Datacenter IPv6 proxies with at most a /40 CIDR subnet will work if they allow unlimited IP rotation in that subnet.
-- Rotating residential proxies will work but they're really expensive (I'm talking about upwards 50€ / months just for one)
+- Rotating residential proxies will work but they're really expensive (I'm talking about upwards 50€ / months just for one).
-- You need at least 500 GiB traffic / month without DASH
+- You need at least 500 GiB traffic / month without DASH.
-- You need at least 30 TiB traffic / month with DASH and/or local proxy. The more users use your instance, the more traffic you'll need
+- You need at least 30 TiB traffic / month with DASH and/or local proxy. The more users use your instance, the more traffic you'll need.
 
 # Build instructions
 
-Do this in a separate dir (for example /srv/invidious) under non-root permissions
+Do this in a separate dir (for example /srv/invidious) under non-root permissions:
+
+First, clone this repo with all subrepos:
 
 1. `git clone --recurse-submodules https://github.com/Sommerwiesel/invidious-nerdvpn .`
-2. Modify the patches under invidious-nerdvpn/patches to your needs (i.e. change branding) and then run `./patch.sh`
+
-3. `./build.sh development` for the development image or `./build.sh release` for the production image
+Build invidious:
-4. `build.sh companion` to build the invidious companion
+
+1. `git clone --recurse-submodules https://github.com/Sommerwiesel/invidious-nerdvpn .`
+2. `./build.sh development` for the development image or `./build.sh release` for the production image
+
+Build invidious-companion:
+
+1. `patch.sh companion`
+2. `build.sh companion` to build the invidious companion
 
 # Update instructions
 
-1. `./revert.sh` to remove all patch-modifications
+For invidious:
-2. `git pull --recurse-submodules`
-3. Run the steps 3 and 4 again from the run instructions
 
-# Run instructions (docker compose)
+1. `./update.sh invidious` to get the latest upstream
+2. `./patch.sh invidious`
+3. `./build.sh development` for the development image or `./build.sh release` for the production image
 
-- Simply run `docker compose up -d`
+For invidious-companion:
+
+1. `./update.sh companion` to get the latest upstream
+2. `./patch.sh companion`
+3. `build.sh companion` to build the invidious companion
+
+# Run instructions (docker compose V2)
+
+- `docker compose up -d`
 
 # Run instructions (systemd)
 

build.sh

@@ -1,11 +1,12 @@
 #!/usr/bin/env bash
 
 if [ $# -eq 0 ]; then
-    echo "Usage: build [development|release]"
+    echo "Usage: build [development|release|companion]"
     exit 1
 fi
 
 mode=$1
+
 export COMPOSE_BAKE=true
 
 if [ "$mode" == "release" ]; then

patch.sh

@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 
-cd build
+if [ $# -eq 0 ]; then
+    echo "Usage: build [invidious|companion]"
+    exit 1
+fi
 
-PATCHES=../patches/*.patch
+mode=$1
 
-for p in $PATCHES; do
+if [ "$mode" == "invidious" ]; then
-	echo "applying ${p}..."
+        cd build
-	git apply --check $p && git apply $p
-done
 
+        PATCHES=../patches/invidious/*.patch
+
+        for p in $PATCHES; do
+                echo "applying ${p}..."
+                git apply --check $p && git apply $p
+        done
+fi
+
+if [ "$mode" == "companion" ]; then
+        cd companion
+
+        PATCHES=../patches/companion/*.patch
+
+        for p in $PATCHES; do
+                echo "applying ${p}..."
+                git apply --check $p && git apply $p
+        done
+fi

patches/companion/0001-feat-add-resolution-limit-on-DASH-streams-to-save-ba.patch

@@ -0,0 +1,54 @@
+From 04eb4c82d81a4ff183d2e94eff474136841ff1a1 Mon Sep 17 00:00:00 2001
+From: Fijxu <fijxu@nadeko.net>
+Date: Mon, 24 Mar 2025 19:02:01 -0300
+Subject: [PATCH 01/12] feat: add resolution limit on DASH streams to save
+ bandwidth
+
+---
+ config/config.example.toml                  | 1 +
+ src/lib/helpers/config.ts                   | 3 +++
+ src/routes/invidious_routes/dashManifest.ts | 3 ++-
+ 3 files changed, 6 insertions(+), 1 deletion(-)
+
+diff --git a/config/config.example.toml b/config/config.example.toml
+index 2a49d0f..d711ca0 100644
+--- a/config/config.example.toml
++++ b/config/config.example.toml
+@@ -17,6 +17,7 @@
+ # verify_requests = false # env variable: SERVER_VERIFY_REQUESTS
+ # encrypt_query_params = false # env variable: SERVER_ENCRYPT_QUERY_PARAMS
+ # enable_metrics = false # env variable: SERVER_ENABLE_METRICS
++# max_dash_resolution = 1080
+  
+ # [cache]
+ # enabled = true # env variable: CACHE_ENABLED
+diff --git a/src/lib/helpers/config.ts b/src/lib/helpers/config.ts
+index 6ec7112..793265e 100644
+--- a/src/lib/helpers/config.ts
++++ b/src/lib/helpers/config.ts
+@@ -17,6 +17,9 @@ export const ConfigSchema = z.object({
+         enable_metrics: z.boolean().default(
+             Deno.env.get("SERVER_ENABLE_METRICS") === "true" || false,
+         ),
++        max_dash_resolution: z.number().default(
++            Number(Deno.env.get("SERVER_MAX_DASH_RESOLUTION")),
++        ),
+     }).strict().default({}),
+     cache: z.object({
+         enabled: z.boolean().default(
+diff --git a/src/routes/invidious_routes/dashManifest.ts b/src/routes/invidious_routes/dashManifest.ts
+index 059b4ed..2ebb783 100644
+--- a/src/routes/invidious_routes/dashManifest.ts
++++ b/src/routes/invidious_routes/dashManifest.ts
+@@ -55,7 +55,8 @@ dashManifest.get("/:videoId", async (c) => {
+         videoInfo.streaming_data.adaptive_formats = videoInfo
+             .streaming_data.adaptive_formats
+             .filter((i) =>
+-                i.has_video === false || i.mime_type.includes("mp4")
++                //@ts-ignore: 'i.height' is possibly 'undefined'.
++                i.has_video === false || (i.mime_type.includes("mp4") && (i.height <= config.server.max_dash_resolution))
+             );
+ 
+         const player_response = videoInfo.page[0];
+-- 
+2.49.0

patches/companion/0004-add-proxy-retries-on-innertube-error.patch

@@ -0,0 +1,78 @@
+From 8d35596d51350b959d724638fb703661992e1cc4 Mon Sep 17 00:00:00 2001
+From: Fijxu <fijxu@nadeko.net>
+Date: Tue, 25 Mar 2025 00:04:47 -0300
+Subject: [PATCH 04/12] add proxy retries on innertube error
+
+---
+ config/config.example.toml               |  1 +
+ src/lib/helpers/config.ts                |  3 +++
+ src/lib/helpers/youtubePlayerHandling.ts | 23 ++++++++++++++++++++++-
+ 3 files changed, 26 insertions(+), 1 deletion(-)
+
+diff --git a/config/config.example.toml b/config/config.example.toml
+index 84c4f02..2112564 100644
+--- a/config/config.example.toml
++++ b/config/config.example.toml
+@@ -33,6 +33,7 @@
+ # ump = false # env variable: NETWORKING_VIDEOPLAYBACK_UMP
+ # # size of chunks to request from google servers for rate limiting reductions
+ # video_fetch_chunk_size_mb = 5 # env variable: NETWORKING_VIDEOPLAYBACK_VIDEO_FETCH_CHUNK_SIZE_MB
++# max_proxy_retries = 2 # env variable: NETWORKING_MAX_PROXY_RETRIES
+  
+ ###
+ # Network call timeouts when talking to YouTube. 
+diff --git a/src/lib/helpers/config.ts b/src/lib/helpers/config.ts
+index 3990214..01dda24 100644
+--- a/src/lib/helpers/config.ts
++++ b/src/lib/helpers/config.ts
+@@ -70,6 +70,9 @@ export const ConfigSchema = z.object({
+                 ) || 5,
+             ),
+         }).strict().default({}),
++        max_proxy_retries: z.number().default(
++            Number(Deno.env.get("NETWORKING_MAX_PROXY_RETIRES") || 2),
++        ),
+     }).strict().default({}),
+     jobs: z.object({
+         youtube_session: z.object({
+diff --git a/src/lib/helpers/youtubePlayerHandling.ts b/src/lib/helpers/youtubePlayerHandling.ts
+index 6565d06..d809c09 100644
+--- a/src/lib/helpers/youtubePlayerHandling.ts
++++ b/src/lib/helpers/youtubePlayerHandling.ts
+@@ -43,12 +43,33 @@ export const youtubePlayerParsing = async ({
+     if (videoCached != null && cacheEnabled) {
+         return JSON.parse(new TextDecoder().decode(decompress(videoCached)));
+     } else {
+-        const youtubePlayerResponse = await youtubePlayerReq(
++        let youtubePlayerResponse = await youtubePlayerReq(
+             innertubeClient,
+             videoId,
+             config,
+             tokenMinter,
+         );
++
++        const maxRetries = config.networking.max_proxy_retries;
++        for (let retries = 1; retries <= (maxRetries as number); retries++) {
++            if (
++                !youtubePlayerResponse.data.playabilityStatus?.errorScreen
++                    ?.playerErrorMessageRenderer?.subreason?.runs?.[0]?.text
++                    ?.includes("This helps protect our community")
++            ) {
++                break;
++            }
++            console.log(
++                `[DEBUG] Got 'This helps protect our community', retrying request for ${videoId}. Retry ${retries} of ${maxRetries}`,
++            );
++            youtubePlayerResponse = await youtubePlayerReq(
++                innertubeClient,
++                videoId,
++                config,
++                tokenMinter,
++            );
++        }
++
+         const videoData = youtubePlayerResponse.data;
+ 
+         if (videoData.playabilityStatus.status === "ERROR") {
+-- 
+2.49.0

patches/companion/0009-fix-check-for-formats.url-since-most-videos-will-hav.patch

@@ -0,0 +1,20 @@
+diff --git a/src/lib/jobs/potoken.ts b/src/lib/jobs/potoken.ts
+index f6b9aee..aca59a4 100644
+--- a/src/lib/jobs/potoken.ts
++++ b/src/lib/jobs/potoken.ts
+@@ -172,13 +172,13 @@ async function checkToken({
+             instantiatedInnertubeClient,
+             youtubePlayerResponseJson,
+         );
+-        const validFormat = videoInfo.streaming_data?.adaptive_formats[0];
++        const validFormat = videoInfo.streaming_data?.adaptive_formats[0].url || videoInfo.streaming_data?.formats[0].url;
+         if (!validFormat) {
+             throw new Error(
+                 "failed to find valid video with adaptive format to check token against",
+             );
+         }
+-        const result = await fetchImpl(validFormat?.url, { method: "HEAD" });
++        const result = await fetchImpl(validFormat, { method: "HEAD" });
+         if (result.status !== 200) {
+             throw new Error(
+                 `did not get a 200 when checking video, got ${result.status} instead`,

revert.sh

@@ -1,8 +1,26 @@
 #!/usr/bin/env bash
 
-cd build
+if [ $# -eq 0 ]; then
+    echo "Usage: build [invidious|companion]"
+    exit 1
+fi
 
-git checkout .
+mode=$1
-git restore .
+
-git reset
+if [ "$mode" == "invidious" ]; then
-git clean -fd
+        cd build
+
+        git checkout .
+        git restore .
+        git reset
+        git clean -fd
+fi
+
+if [ "$mode" == "companion" ]; then
+        cd companion
+
+        git checkout .
+        git restore .
+        git reset
+        git clean -fd
+fi

update.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+
+mode=$1
+
+if [ "$mode" == "invidious" ]; then
+        cd build
+
+        git checkout .
+        git restore .
+        git reset
+        git clean -fd
+
+        git pull --recurse-submodules
+fi
+
+if [ "$mode" == "companion" ]; then
+        cd companion
+
+        git checkout .
+        git restore .
+        git reset
+        git clean -fd
+
+        git pull
+fi
+