revert: drop the YouTube cookies/PO-token/proxy workarounds (back to pre-saga state)
None of it got downloads working on the flagged datacenter IP (web_music media 403, then 429 rate-limiting), so revert the whole lot to keep things clean: - config.js: remove potProviderUrl + ytdlpProxy - ytdlp.js: remove potArgs/proxyArgs; player_client back to web_music,default - novasonic-server.js: remove pot/proxy args from importer resolve - docker-compose.yml: remove the bgutil-provider service + backend env/plugin/depends_on - CHANGELOG: drop the [2.2] section Kept the one genuine fix: a failed direct-link download now reports failure (moved its note into [2.1]). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
1 parent
31d4a72e62
commit
635ba8d353
5 files changed
+8
-68
No files matched your search
+3
-22
@@ -64,28 +64,6 @@ Versioning: two-part `MAJOR.MINOR` — each release bumps the minor (features an
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## [2.2] — 2026-08-14
|
|
||||||
|
|
||||||
Keeps downloads working after YouTube began blocking datacenter IPs, and fixes a related
|
|
||||||
false-success in the UI.
|
|
||||||
|
|
||||||
### Added
|
|
||||||
- **PO token provider (bgutil)** — YouTube now 403s media downloads from datacenter IPs without a
|
|
||||||
**PO ("Proof of Origin") token** that yt-dlp can't compute itself (seen first as "Sign in to
|
|
||||||
confirm you're not a bot", then `HTTP 403` on the audio stream). A self-hosted **`bgutil-provider`**
|
|
||||||
container (`brainicism/bgutil-ytdlp-pot-provider`) mints the tokens and the matching
|
|
||||||
`bgutil-ytdlp-pot-provider` plugin (installed in the backend) hands them to yt-dlp — **no cookies or
|
|
||||||
account required**. Wired into the download cascade, the YTM search probe and the importer resolve,
|
|
||||||
guarded by `POT_PROVIDER_URL` (empty → yt-dlp runs exactly as before). Deploy: `docker compose up -d`.
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
- **A failed direct-link download reported success** — the direct-URL path returned `success` even
|
|
||||||
when no audio was produced (e.g. the YouTube 403), so the job showed "Fertig!" and the UI (web +
|
|
||||||
Android) said the track was added. It now reports failure with a clear message ("Download
|
|
||||||
fehlgeschlagen — YouTube hat den Abruf blockiert (403)").
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## [2.1] — 2026-08-14
|
## [2.1] — 2026-08-14
|
||||||
|
|
||||||
Builds on the 2.0 listening archive with a **head-to-head "Musik-Match"**, brings the **artist pin**
|
Builds on the 2.0 listening archive with a **head-to-head "Musik-Match"**, brings the **artist pin**
|
||||||
@@ -110,6 +88,9 @@ to Android at parity with web, and fixes a batch of same-name-artist and player
|
|||||||
as avatar chips, matching the Android app (previously plain text).
|
as avatar chips, matching the Android app (previously plain text).
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
- **A failed direct-link download reported success** — the direct-URL path returned `success` even
|
||||||
|
when no audio was produced (e.g. a YouTube 403), so the job showed "Fertig!" and the UI (web +
|
||||||
|
Android) said the track was added. It now reports failure with a clear message.
|
||||||
- **Artist pin — search & similar artists (Web + Android)** — a pinned artist now shows the correct
|
- **Artist pin — search & similar artists (Web + Android)** — a pinned artist now shows the correct
|
||||||
Deezer photo in library search (not the wrong same-name person's), and its "Similar Artists" use
|
Deezer photo in library search (not the wrong same-name person's), and its "Similar Artists" use
|
||||||
Deezer's **related** for the pinned artist instead of Last.fm's list (which was keyed to the
|
Deezer's **related** for the pinned artist instead of Last.fm's list (which was keyed to the
|
||||||
|
|||||||
@@ -8,14 +8,6 @@ const CONFIG = {
|
|||||||
maxParallel: parseInt(process.env.MAX_PARALLEL || '2', 10),
|
maxParallel: parseInt(process.env.MAX_PARALLEL || '2', 10),
|
||||||
downloadBackend: (process.env.DOWNLOAD_BACKEND || 'ytdlp').toLowerCase(),
|
downloadBackend: (process.env.DOWNLOAD_BACKEND || 'ytdlp').toLowerCase(),
|
||||||
ticketmasterKey: process.env.TICKETMASTER_API_KEY || '',
|
ticketmasterKey: process.env.TICKETMASTER_API_KEY || '',
|
||||||
// YouTube 403s media downloads from datacenter IPs without a PO ("Proof of Origin") token, which
|
|
||||||
// yt-dlp can't compute itself. The bgutil-ytdlp-pot-provider plugin fetches tokens from a small
|
|
||||||
// provider service; set its URL here (e.g. http://bgutil-provider:4416). Empty → no PO token used
|
|
||||||
// (yt-dlp runs exactly as before). See docker-compose.yml.
|
|
||||||
potProviderUrl: process.env.POT_PROVIDER_URL || '',
|
|
||||||
// Optional proxy for yt-dlp (e.g. a residential/rotating proxy) — the durable fix when the
|
|
||||||
// datacenter IP stays flagged (403/429) even with a PO token. Empty → yt-dlp runs directly.
|
|
||||||
ytdlpProxy: process.env.YTDLP_PROXY || '',
|
|
||||||
};
|
};
|
||||||
|
|
||||||
module.exports = CONFIG;
|
module.exports = CONFIG;
|
||||||
+4
-21
@@ -134,18 +134,6 @@ async function downloadTrack({ jobId, artist, rawArtist, albumArtist, title, alb
|
|||||||
// YTM ranking toward a same-named artist's full discography.
|
// YTM ranking toward a same-named artist's full discography.
|
||||||
const query = `${ytmArtist} ${searchTitle}`;
|
const query = `${ytmArtist} ${searchTitle}`;
|
||||||
|
|
||||||
// YouTube 403s media downloads from datacenter IPs without a PO token. When a provider URL is
|
|
||||||
// configured, point the bgutil-ytdlp-pot-provider plugin at it so yt-dlp fetches a token. Empty
|
|
||||||
// config → no arg → unchanged behaviour.
|
|
||||||
const potArgs = () =>
|
|
||||||
CONFIG.potProviderUrl
|
|
||||||
? ['--extractor-args', `youtubepot-bgutilhttp:base_url=${CONFIG.potProviderUrl}`]
|
|
||||||
: [];
|
|
||||||
|
|
||||||
// Optional proxy — the durable fix when a datacenter IP stays flagged (403/429) even with a PO
|
|
||||||
// token. Route yt-dlp through a residential/rotating proxy via YTDLP_PROXY. Empty → not used.
|
|
||||||
const proxyArgs = () => (CONFIG.ytdlpProxy ? ['--proxy', CONFIG.ytdlpProxy] : []);
|
|
||||||
|
|
||||||
const baseArgs = [
|
const baseArgs = [
|
||||||
'--extract-audio',
|
'--extract-audio',
|
||||||
'--audio-format', CONFIG.audioFmt,
|
'--audio-format', CONFIG.audioFmt,
|
||||||
@@ -156,13 +144,10 @@ async function downloadTrack({ jobId, artist, rawArtist, albumArtist, title, alb
|
|||||||
'--newline',
|
'--newline',
|
||||||
// Hard upper size limit — prevents saving a wrong 28-min video as a ~52 MB mp3
|
// Hard upper size limit — prevents saving a wrong 28-min video as a ~52 MB mp3
|
||||||
'--max-filesize', '30M',
|
'--max-filesize', '30M',
|
||||||
// For the DOWNLOAD, prefer the default clients (their media URLs go through with a PO token from
|
// Use the web_music player client (falling back to default) so YouTube Music
|
||||||
// a datacenter IP) and keep web_music only as a fallback for Art Tracks that only it can extract.
|
// "Art Track" / topic uploads are reachable. The android_vr client returns
|
||||||
// web_music's own media URLs 403 from datacenter IPs even with a valid PO token. (The YTM *search*
|
// "This video is not available" for many YTM tracks that play fine in the browser.
|
||||||
// in getYtmVideoUrls still leads with web_music, which is where Art-Track discovery matters.)
|
'--extractor-args', 'youtube:player_client=web_music,default',
|
||||||
'--extractor-args', 'youtube:player_client=default,web_music',
|
|
||||||
...potArgs(),
|
|
||||||
...proxyArgs(),
|
|
||||||
];
|
];
|
||||||
|
|
||||||
const spawnYtdlp = (args) => new Promise((resolve, reject) => {
|
const spawnYtdlp = (args) => new Promise((resolve, reject) => {
|
||||||
@@ -235,8 +220,6 @@ async function downloadTrack({ jobId, artist, rawArtist, albumArtist, title, alb
|
|||||||
// generic YouTube search and returns unrelated videos instead of the
|
// generic YouTube search and returns unrelated videos instead of the
|
||||||
// native Songs shelf (where the actual artist track lives).
|
// native Songs shelf (where the actual artist track lives).
|
||||||
'--extractor-args', 'youtube:player_client=web_music,default',
|
'--extractor-args', 'youtube:player_client=web_music,default',
|
||||||
...potArgs(),
|
|
||||||
...proxyArgs(),
|
|
||||||
], { env: { ...process.env, PYTHONUNBUFFERED: '1' } });
|
], { env: { ...process.env, PYTHONUNBUFFERED: '1' } });
|
||||||
proc.stdout.on('data', d => {
|
proc.stdout.on('data', d => {
|
||||||
const text = d.toString();
|
const text = d.toString();
|
||||||
|
|||||||
@@ -533,9 +533,6 @@ app.post('/api/youtube/resolve', async (req, res) => {
|
|||||||
execFile(CONFIG.ytdlpBin, [
|
execFile(CONFIG.ytdlpBin, [
|
||||||
'--dump-json', '--flat-playlist', '--no-warnings',
|
'--dump-json', '--flat-playlist', '--no-warnings',
|
||||||
'--playlist-items', '1-50',
|
'--playlist-items', '1-50',
|
||||||
// PO token provider (if configured) to get past YouTube's datacenter-IP block. Empty → omitted.
|
|
||||||
...(CONFIG.potProviderUrl ? ['--extractor-args', `youtubepot-bgutilhttp:base_url=${CONFIG.potProviderUrl}`] : []),
|
|
||||||
...(CONFIG.ytdlpProxy ? ['--proxy', CONFIG.ytdlpProxy] : []),
|
|
||||||
url,
|
url,
|
||||||
], { timeout: 45000, maxBuffer: 20 * 1024 * 1024 }, (err, out, stderr) => {
|
], { timeout: 45000, maxBuffer: 20 * 1024 * 1024 }, (err, out, stderr) => {
|
||||||
if (err && !out?.trim()) return reject(new Error(stderr?.trim() || err.message));
|
if (err && !out?.trim()) return reject(new Error(stderr?.trim() || err.message));
|
||||||
|
|||||||
+1
-14
@@ -33,28 +33,15 @@ services:
|
|||||||
- MAX_PARALLEL=${MAX_PARALLEL:-2}
|
- MAX_PARALLEL=${MAX_PARALLEL:-2}
|
||||||
- DOWNLOAD_BACKEND=${DOWNLOAD_BACKEND:-ytdlp}
|
- DOWNLOAD_BACKEND=${DOWNLOAD_BACKEND:-ytdlp}
|
||||||
- TICKETMASTER_API_KEY=${TICKETMASTER_API_KEY}
|
- TICKETMASTER_API_KEY=${TICKETMASTER_API_KEY}
|
||||||
# PO token provider (bgutil) — lets yt-dlp download from datacenter IPs without cookies/account.
|
|
||||||
- POT_PROVIDER_URL=http://bgutil-provider:4416
|
|
||||||
depends_on:
|
|
||||||
- bgutil-provider
|
|
||||||
command: >
|
command: >
|
||||||
sh -c "
|
sh -c "
|
||||||
apt-get update && apt-get install -y --no-install-recommends python3 python3-pip ffmpeg curl git && rm -rf /var/lib/apt/lists/* &&
|
apt-get update && apt-get install -y --no-install-recommends python3 python3-pip ffmpeg curl git && rm -rf /var/lib/apt/lists/* &&
|
||||||
curl -sL https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp && chmod a+rx /usr/local/bin/yt-dlp &&
|
curl -sL https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp && chmod a+rx /usr/local/bin/yt-dlp &&
|
||||||
pip install --break-system-packages spotapi 'git+https://github.com/TzurSoffer/spotify-downloader' bgutil-ytdlp-pot-provider &&
|
pip install --break-system-packages spotapi 'git+https://github.com/TzurSoffer/spotify-downloader' &&
|
||||||
mkdir -p /deps && cd /deps && npm install express kuromoji &&
|
mkdir -p /deps && cd /deps && npm install express kuromoji &&
|
||||||
cd /app && NODE_PATH=/deps/node_modules node novasonic-server.js
|
cd /app && NODE_PATH=/deps/node_modules node novasonic-server.js
|
||||||
"
|
"
|
||||||
|
|
||||||
# ── PO token provider (bgutil) — mints YouTube "Proof of Origin" tokens so yt-dlp can download
|
|
||||||
# from a datacenter IP without cookies or an account. Paired with the bgutil-ytdlp-pot-provider
|
|
||||||
# plugin installed in the backend above. Port 4416 is internal only (no reverse proxy needed). ──
|
|
||||||
bgutil-provider:
|
|
||||||
image: brainicism/bgutil-ytdlp-pot-provider:latest
|
|
||||||
container_name: bgutil-provider
|
|
||||||
restart: unless-stopped
|
|
||||||
init: true
|
|
||||||
|
|
||||||
# ── Scrobble service (isolated timestamped play-log → stats + Wrapped) ────────
|
# ── Scrobble service (isolated timestamped play-log → stats + Wrapped) ────────
|
||||||
novasonic-scrobble:
|
novasonic-scrobble:
|
||||||
image: node:22-slim
|
image: node:22-slim
|
||||||
|
|||||||
Reference in new issue
Block a user