diff --git a/README.md b/README.md index b778c9e..3b76002 100644 --- a/README.md +++ b/README.md @@ -27,13 +27,13 @@ If you NEVER used Invidious before, you can safely start with my fork, but you w You'll have to set these up yourself: -- Docker Compose V2 -- PostgreSQL 16 Server -- Nginx Web Server (1.26 or higher for quic support) -- HAProxy 2.x with my config under etc/haproxy/haproxy.conf -- Redis Server for the video cache -- (optional) PGBouncer, recommended if you want to host a public instance to pool connections to PSQL, otherwise you might run into DB pool errors -- (optional) Anubis https://github.com/TecharoHQ/anubis, recommended if you want to host a public instance and not get DDoS'ed from bots and ai +- Docker Compose V2 (this is still not included on many default docker installations so make sure you actually have the plugin version 2). +- PostgreSQL 16 Server. +- Nginx Web Server (1.26 or higher for quic support). +- HAProxy 2.x with my config under `etc/haproxy/haproxy.conf`. +- Redis Server for the video cache. +- (optional) PGBouncer, recommended if you want to host a public instance to pool connections to PSQL, otherwise you might run into DB pool errors. +- (optional) Anubis https://github.com/TecharoHQ/anubis, recommended if you want to host a public instance and not get DDoS'ed from bots and ai. # For public instances @@ -43,32 +43,49 @@ I will not give any recommendations here because this is something you should re A few important infos: -- Datacenter IPv4 proxies will not work -- Static residential IPv4 proxies will not work -- Most VPN providers that do not allow dynamically switching target server or target IP will not work -- Datacenter IPv6 proxies with at most a /40 CIDR subnet will work if they allow unlimited IP rotation in that subnet -- Rotating residential proxies will work but they're really expensive (I'm talking about upwards 50€ / months just for one) -- You need at least 500 GiB traffic / month without DASH -- You need at least 30 TiB traffic / month with DASH and/or local proxy. The more users use your instance, the more traffic you'll need +- Datacenter IPv4 proxies will not work. +- Static residential IPv4 proxies will not work. +- Most VPN providers that do not allow dynamically switching target server or target IP will not work. +- Datacenter IPv6 proxies with at most a /40 CIDR subnet will work if they allow unlimited IP rotation in that subnet. +- Rotating residential proxies will work but they're really expensive (I'm talking about upwards 50€ / months just for one). +- You need at least 500 GiB traffic / month without DASH. +- You need at least 30 TiB traffic / month with DASH and/or local proxy. The more users use your instance, the more traffic you'll need. # Build instructions -Do this in a separate dir (for example /srv/invidious) under non-root permissions +Do this in a separate dir (for example /srv/invidious) under non-root permissions: + +First, clone this repo with all subrepos: 1. `git clone --recurse-submodules https://github.com/Sommerwiesel/invidious-nerdvpn .` -2. Modify the patches under invidious-nerdvpn/patches to your needs (i.e. change branding) and then run `./patch.sh` -3. `./build.sh development` for the development image or `./build.sh release` for the production image -4. `build.sh companion` to build the invidious companion + +Build invidious: + +1. `git clone --recurse-submodules https://github.com/Sommerwiesel/invidious-nerdvpn .` +2. `./build.sh development` for the development image or `./build.sh release` for the production image + +Build invidious-companion: + +1. `patch.sh companion` +2. `build.sh companion` to build the invidious companion # Update instructions -1. `./revert.sh` to remove all patch-modifications -2. `git pull --recurse-submodules` -3. Run the steps 3 and 4 again from the run instructions +For invidious: -# Run instructions (docker compose) +1. `./update.sh invidious` to get the latest upstream +2. `./patch.sh invidious` +3. `./build.sh development` for the development image or `./build.sh release` for the production image -- Simply run `docker compose up -d` +For invidious-companion: + +1. `./update.sh companion` to get the latest upstream +2. `./patch.sh companion` +3. `build.sh companion` to build the invidious companion + +# Run instructions (docker compose V2) + +- `docker compose up -d` # Run instructions (systemd) diff --git a/build.sh b/build.sh index d67c471..f75f4b5 100755 --- a/build.sh +++ b/build.sh @@ -1,11 +1,12 @@ #!/usr/bin/env bash if [ $# -eq 0 ]; then - echo "Usage: build [development|release]" + echo "Usage: build [development|release|companion]" exit 1 fi mode=$1 + export COMPOSE_BAKE=true if [ "$mode" == "release" ]; then diff --git a/patch.sh b/patch.sh index da33ffc..f90b26c 100755 --- a/patch.sh +++ b/patch.sh @@ -1,11 +1,30 @@ #!/usr/bin/env bash -cd build +if [ $# -eq 0 ]; then + echo "Usage: build [invidious|companion]" + exit 1 +fi -PATCHES=../patches/*.patch +mode=$1 -for p in $PATCHES; do - echo "applying ${p}..." - git apply --check $p && git apply $p -done +if [ "$mode" == "invidious" ]; then + cd build + PATCHES=../patches/invidious/*.patch + + for p in $PATCHES; do + echo "applying ${p}..." + git apply --check $p && git apply $p + done +fi + +if [ "$mode" == "companion" ]; then + cd companion + + PATCHES=../patches/companion/*.patch + + for p in $PATCHES; do + echo "applying ${p}..." + git apply --check $p && git apply $p + done +fi diff --git a/patches/companion/0001-feat-add-resolution-limit-on-DASH-streams-to-save-ba.patch b/patches/companion/0001-feat-add-resolution-limit-on-DASH-streams-to-save-ba.patch new file mode 100644 index 0000000..ec1fcfa --- /dev/null +++ b/patches/companion/0001-feat-add-resolution-limit-on-DASH-streams-to-save-ba.patch @@ -0,0 +1,54 @@ +From 04eb4c82d81a4ff183d2e94eff474136841ff1a1 Mon Sep 17 00:00:00 2001 +From: Fijxu +Date: Mon, 24 Mar 2025 19:02:01 -0300 +Subject: [PATCH 01/12] feat: add resolution limit on DASH streams to save + bandwidth + +--- + config/config.example.toml | 1 + + src/lib/helpers/config.ts | 3 +++ + src/routes/invidious_routes/dashManifest.ts | 3 ++- + 3 files changed, 6 insertions(+), 1 deletion(-) + +diff --git a/config/config.example.toml b/config/config.example.toml +index 2a49d0f..d711ca0 100644 +--- a/config/config.example.toml ++++ b/config/config.example.toml +@@ -17,6 +17,7 @@ + # verify_requests = false # env variable: SERVER_VERIFY_REQUESTS + # encrypt_query_params = false # env variable: SERVER_ENCRYPT_QUERY_PARAMS + # enable_metrics = false # env variable: SERVER_ENABLE_METRICS ++# max_dash_resolution = 1080 + + # [cache] + # enabled = true # env variable: CACHE_ENABLED +diff --git a/src/lib/helpers/config.ts b/src/lib/helpers/config.ts +index 6ec7112..793265e 100644 +--- a/src/lib/helpers/config.ts ++++ b/src/lib/helpers/config.ts +@@ -17,6 +17,9 @@ export const ConfigSchema = z.object({ + enable_metrics: z.boolean().default( + Deno.env.get("SERVER_ENABLE_METRICS") === "true" || false, + ), ++ max_dash_resolution: z.number().default( ++ Number(Deno.env.get("SERVER_MAX_DASH_RESOLUTION")), ++ ), + }).strict().default({}), + cache: z.object({ + enabled: z.boolean().default( +diff --git a/src/routes/invidious_routes/dashManifest.ts b/src/routes/invidious_routes/dashManifest.ts +index 059b4ed..2ebb783 100644 +--- a/src/routes/invidious_routes/dashManifest.ts ++++ b/src/routes/invidious_routes/dashManifest.ts +@@ -55,7 +55,8 @@ dashManifest.get("/:videoId", async (c) => { + videoInfo.streaming_data.adaptive_formats = videoInfo + .streaming_data.adaptive_formats + .filter((i) => +- i.has_video === false || i.mime_type.includes("mp4") ++ //@ts-ignore: 'i.height' is possibly 'undefined'. ++ i.has_video === false || (i.mime_type.includes("mp4") && (i.height <= config.server.max_dash_resolution)) + ); + + const player_response = videoInfo.page[0]; +-- +2.49.0 diff --git a/patches/companion/0004-add-proxy-retries-on-innertube-error.patch b/patches/companion/0004-add-proxy-retries-on-innertube-error.patch new file mode 100644 index 0000000..651075d --- /dev/null +++ b/patches/companion/0004-add-proxy-retries-on-innertube-error.patch @@ -0,0 +1,78 @@ +From 8d35596d51350b959d724638fb703661992e1cc4 Mon Sep 17 00:00:00 2001 +From: Fijxu +Date: Tue, 25 Mar 2025 00:04:47 -0300 +Subject: [PATCH 04/12] add proxy retries on innertube error + +--- + config/config.example.toml | 1 + + src/lib/helpers/config.ts | 3 +++ + src/lib/helpers/youtubePlayerHandling.ts | 23 ++++++++++++++++++++++- + 3 files changed, 26 insertions(+), 1 deletion(-) + +diff --git a/config/config.example.toml b/config/config.example.toml +index 84c4f02..2112564 100644 +--- a/config/config.example.toml ++++ b/config/config.example.toml +@@ -33,6 +33,7 @@ + # ump = false # env variable: NETWORKING_VIDEOPLAYBACK_UMP + # # size of chunks to request from google servers for rate limiting reductions + # video_fetch_chunk_size_mb = 5 # env variable: NETWORKING_VIDEOPLAYBACK_VIDEO_FETCH_CHUNK_SIZE_MB ++# max_proxy_retries = 2 # env variable: NETWORKING_MAX_PROXY_RETRIES + + ### + # Network call timeouts when talking to YouTube. +diff --git a/src/lib/helpers/config.ts b/src/lib/helpers/config.ts +index 3990214..01dda24 100644 +--- a/src/lib/helpers/config.ts ++++ b/src/lib/helpers/config.ts +@@ -70,6 +70,9 @@ export const ConfigSchema = z.object({ + ) || 5, + ), + }).strict().default({}), ++ max_proxy_retries: z.number().default( ++ Number(Deno.env.get("NETWORKING_MAX_PROXY_RETIRES") || 2), ++ ), + }).strict().default({}), + jobs: z.object({ + youtube_session: z.object({ +diff --git a/src/lib/helpers/youtubePlayerHandling.ts b/src/lib/helpers/youtubePlayerHandling.ts +index 6565d06..d809c09 100644 +--- a/src/lib/helpers/youtubePlayerHandling.ts ++++ b/src/lib/helpers/youtubePlayerHandling.ts +@@ -43,12 +43,33 @@ export const youtubePlayerParsing = async ({ + if (videoCached != null && cacheEnabled) { + return JSON.parse(new TextDecoder().decode(decompress(videoCached))); + } else { +- const youtubePlayerResponse = await youtubePlayerReq( ++ let youtubePlayerResponse = await youtubePlayerReq( + innertubeClient, + videoId, + config, + tokenMinter, + ); ++ ++ const maxRetries = config.networking.max_proxy_retries; ++ for (let retries = 1; retries <= (maxRetries as number); retries++) { ++ if ( ++ !youtubePlayerResponse.data.playabilityStatus?.errorScreen ++ ?.playerErrorMessageRenderer?.subreason?.runs?.[0]?.text ++ ?.includes("This helps protect our community") ++ ) { ++ break; ++ } ++ console.log( ++ `[DEBUG] Got 'This helps protect our community', retrying request for ${videoId}. Retry ${retries} of ${maxRetries}`, ++ ); ++ youtubePlayerResponse = await youtubePlayerReq( ++ innertubeClient, ++ videoId, ++ config, ++ tokenMinter, ++ ); ++ } ++ + const videoData = youtubePlayerResponse.data; + + if (videoData.playabilityStatus.status === "ERROR") { +-- +2.49.0 diff --git a/patches/companion/0009-fix-check-for-formats.url-since-most-videos-will-hav.patch b/patches/companion/0009-fix-check-for-formats.url-since-most-videos-will-hav.patch new file mode 100644 index 0000000..f413fb3 --- /dev/null +++ b/patches/companion/0009-fix-check-for-formats.url-since-most-videos-will-hav.patch @@ -0,0 +1,20 @@ +diff --git a/src/lib/jobs/potoken.ts b/src/lib/jobs/potoken.ts +index f6b9aee..aca59a4 100644 +--- a/src/lib/jobs/potoken.ts ++++ b/src/lib/jobs/potoken.ts +@@ -172,13 +172,13 @@ async function checkToken({ + instantiatedInnertubeClient, + youtubePlayerResponseJson, + ); +- const validFormat = videoInfo.streaming_data?.adaptive_formats[0]; ++ const validFormat = videoInfo.streaming_data?.adaptive_formats[0].url || videoInfo.streaming_data?.formats[0].url; + if (!validFormat) { + throw new Error( + "failed to find valid video with adaptive format to check token against", + ); + } +- const result = await fetchImpl(validFormat?.url, { method: "HEAD" }); ++ const result = await fetchImpl(validFormat, { method: "HEAD" }); + if (result.status !== 200) { + throw new Error( + `did not get a 200 when checking video, got ${result.status} instead`, diff --git a/patches/001-nerdvpn-branding.patch b/patches/invidious/001-nerdvpn-branding.patch similarity index 100% rename from patches/001-nerdvpn-branding.patch rename to patches/invidious/001-nerdvpn-branding.patch diff --git a/patches/002-multiple-companion-backends.patch b/patches/invidious/002-multiple-companion-backends.patch similarity index 100% rename from patches/002-multiple-companion-backends.patch rename to patches/invidious/002-multiple-companion-backends.patch diff --git a/patches/003-hide-feed-views.patch b/patches/invidious/003-hide-feed-views.patch similarity index 100% rename from patches/003-hide-feed-views.patch rename to patches/invidious/003-hide-feed-views.patch diff --git a/patches/003-proxy-csp.patch.d b/patches/invidious/003-proxy-csp.patch.d similarity index 100% rename from patches/003-proxy-csp.patch.d rename to patches/invidious/003-proxy-csp.patch.d diff --git a/patches/003-pubsub-innertube.patch.d b/patches/invidious/003-pubsub-innertube.patch.d similarity index 100% rename from patches/003-pubsub-innertube.patch.d rename to patches/invidious/003-pubsub-innertube.patch.d diff --git a/patches/004-donate-page.patch b/patches/invidious/004-donate-page.patch similarity index 100% rename from patches/004-donate-page.patch rename to patches/invidious/004-donate-page.patch diff --git a/patches/007-range-header.patch b/patches/invidious/007-range-header.patch similarity index 100% rename from patches/007-range-header.patch rename to patches/invidious/007-range-header.patch diff --git a/patches/009-restrict-playback.patch.d b/patches/invidious/009-restrict-playback.patch.d similarity index 100% rename from patches/009-restrict-playback.patch.d rename to patches/invidious/009-restrict-playback.patch.d diff --git a/patches/010-use-redis-for-video-cache.patch b/patches/invidious/010-use-redis-for-video-cache.patch similarity index 100% rename from patches/010-use-redis-for-video-cache.patch rename to patches/invidious/010-use-redis-for-video-cache.patch diff --git a/patches/011-remove-psql-mat-views.patch b/patches/invidious/011-remove-psql-mat-views.patch similarity index 100% rename from patches/011-remove-psql-mat-views.patch rename to patches/invidious/011-remove-psql-mat-views.patch diff --git a/patches/013-increase-db-timeout.patch b/patches/invidious/013-increase-db-timeout.patch similarity index 100% rename from patches/013-increase-db-timeout.patch rename to patches/invidious/013-increase-db-timeout.patch diff --git a/patches/014-rewrite-obnoxious-blocked-message.patch b/patches/invidious/014-rewrite-obnoxious-blocked-message.patch similarity index 100% rename from patches/014-rewrite-obnoxious-blocked-message.patch rename to patches/invidious/014-rewrite-obnoxious-blocked-message.patch diff --git a/patches/015-status-code.patch b/patches/invidious/015-status-code.patch similarity index 100% rename from patches/015-status-code.patch rename to patches/invidious/015-status-code.patch diff --git a/patches/016-hide-shorts.patch b/patches/invidious/016-hide-shorts.patch similarity index 100% rename from patches/016-hide-shorts.patch rename to patches/invidious/016-hide-shorts.patch diff --git a/patches/017-add-feeds-menu.patch b/patches/invidious/017-add-feeds-menu.patch similarity index 100% rename from patches/017-add-feeds-menu.patch rename to patches/invidious/017-add-feeds-menu.patch diff --git a/patches/018-add-watch-message.patch.d b/patches/invidious/018-add-watch-message.patch.d similarity index 100% rename from patches/018-add-watch-message.patch.d rename to patches/invidious/018-add-watch-message.patch.d diff --git a/patches/019-add-backend-health-status-page.patch b/patches/invidious/019-add-backend-health-status-page.patch similarity index 100% rename from patches/019-add-backend-health-status-page.patch rename to patches/invidious/019-add-backend-health-status-page.patch diff --git a/patches/020-disable-switching-to-unhealthy-backends.patch b/patches/invidious/020-disable-switching-to-unhealthy-backends.patch similarity index 100% rename from patches/020-disable-switching-to-unhealthy-backends.patch rename to patches/invidious/020-disable-switching-to-unhealthy-backends.patch diff --git a/patches/021-add-option-to-force-local-proxy.patch b/patches/invidious/021-add-option-to-force-local-proxy.patch similarity index 100% rename from patches/021-add-option-to-force-local-proxy.patch rename to patches/invidious/021-add-option-to-force-local-proxy.patch diff --git a/patches/022-change-default-dash-quality-options.patch b/patches/invidious/022-change-default-dash-quality-options.patch similarity index 100% rename from patches/022-change-default-dash-quality-options.patch rename to patches/invidious/022-change-default-dash-quality-options.patch diff --git a/revert.sh b/revert.sh index ea8ff92..eabbb4c 100755 --- a/revert.sh +++ b/revert.sh @@ -1,8 +1,26 @@ #!/usr/bin/env bash -cd build +if [ $# -eq 0 ]; then + echo "Usage: build [invidious|companion]" + exit 1 +fi -git checkout . -git restore . -git reset -git clean -fd +mode=$1 + +if [ "$mode" == "invidious" ]; then + cd build + + git checkout . + git restore . + git reset + git clean -fd +fi + +if [ "$mode" == "companion" ]; then + cd companion + + git checkout . + git restore . + git reset + git clean -fd +fi diff --git a/update.sh b/update.sh new file mode 100755 index 0000000..30106e2 --- /dev/null +++ b/update.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash + +mode=$1 + +if [ "$mode" == "invidious" ]; then + cd build + + git checkout . + git restore . + git reset + git clean -fd + + git pull --recurse-submodules +fi + +if [ "$mode" == "companion" ]; then + cd companion + + git checkout . + git restore . + git reset + git clean -fd + + git pull +fi +