Tubearchivist Versions Save

Project Updates

• This will trigger a playlist migration in ElasticSearch, adding a playlist_type field. As always that happens at first start and should be automatic.
• This updates yt-dlp to the latest release. After a change on YT, comment extraction needs to be modified, that is tracked here. Please be patient.

Added

• Added user created custom playlists, by @greg321321, docs and docs
• Added better error message for cookie errors, by @Gentoli

Changed

• Changed disable video autoplay on the video detail page to avoid too many continue watching items.
• Changed first watch progress is now only sent after 10 seconds to avoid too many continue watching items.

Fixed

• Fixed failed empty subtitle return from YT, #663
• Fixed empty comment author return for some old comments, by @dot-mike
• Fixed edge cases for failed channel title building
• Fixed download progress bar for very small videos, by guarantee sending 100% progress
• Fixed download progress messages getting called multiple times in DOM
• Fixed failed legacy channel migration trigger by removing ta_migpath from startup, #687

Project Updates

• This is a small maintenance release fixing some bugs
• This has a workaround for a memory usage problem in Celery, the task scheduler

Fixed

• Fixed a memory leak in celery by reducing the worker life time, by @PhuriousGeorge
• Fixed an issue where deleting a channel wouldn't clean up subtitles correctly
• Fixed reindexing item that has since been deleted locally after queue start
• Fixed throttling video redirect when reindexing by validating IDs and raising error on mismatch

Project Updates

• This is a small release, mainly shipping small but crucial fixes
• Getting this out, then we can take our time for the next additions

Changed

• Changed temporary queue building interface, use redis sorted set instead of lists

Fixed

• Fixed version check comparison over multiple version diff
• Fixed an issue where version_check schedule builder will reset unexpectedly, #611, by @lamusmaser
• Fixed upstream channel about page parser, #614
• Fixed upstream like_count extraction for videos
• Fixed mobile scaling issue in chrome based browsers, #616
• Fixed auto delete day parsing incompatible data types, #622
• Fixed author_is_uploader extraction in comments after yt-dlp API change
• Fixed incompatibility between celery and redis library versions by downgrading redis

Hotfix

If you have updated within approximately the first 24h under certain circumstances you may have gotten an error during the version check at startup. Pull the image again, if you are affected.

Project Updates

• This fixes various mapping problems, triggering a rebuild of the ta_video and ta_playlist index
  • As always, this should be mostly automatic, if not see here.
• This fixes an unfortunate problem where the periodic refresh task wouldn't add outdated videos

Added

• Added additional sort by duration and file size, #559, by @spechter
• Added fallback handling for missing keys in manual import, #581, by @lamusmaser
• Added additional tiles to the dashboard, changed dashboard layout
• [API] Added video stats endpoint
• [API] Added channel stats endpoint
• [API] Added playlist stats endpoint
• [API] Added download stats endpoint

Changed

• [API] Removed previous /api/stats/primary/ endpoint in favor of more detailed dedicated endpoints
• Changed footer text alignment, by @derekslenk

Fixed

• Fixed query not matching outdated documents for reindex
• Fixed reindex outdated pagination error
• Fixed comment extraction for members only videos not passing cookie correctly, #596
• Fixed auto delete watched on new blank index without watched videos, #592
• Fixed mapping problem for some playlists, #605
• Fixed various spelling errors on settings pages, by @pairofcrocs

Project Updates

• This is the second step in moving away from redis persistence, refactoring the environment variable parsing
• This ships a workaround for the new channel about popup parsing failure

Added

• Added read only user functionality, docs, #529 by @stratus-ss
• Added store for volume change in session
• Added CSS styling functionality, docs, by @extome9000
• Added daily download media size to dashboard
• [API] Added zip file backup endpoints
• [API] Added user config endpoints

Changed

• Changed env var handling, Refactored Environment settings, by @anonamouslyginger
• Changed dashboard tile building, by @MerlinScheurer
• Changed daily download stats to be timezone aware
• Touch up the icons, fix coloring for theme, by @pairofcrocs

Fixed

• Fixed channel about page parsing, #587
• Fixed direct video download button, #566 by @OmarLaham
• Fixed off by one message in filesystem rescan

Project update

• First step towards moving away from redis plugins and redis persistence. At application startup this will migrate your user configurations from Redis to ES.

Added

• Added search score to full text search results, docs
• Added mark as unwatched button for playlists and channels, docs and docs, #495 by @froseph
• Added env vars for wider range of ES configurations, docs, #462 by @ajgon
  • ES_SNAPSHOT_DIR: File path for where the snapshots will get stored
  • ES_DISABLE_VERIFY_SSL: Disable SSL verification for ES
• Added health check endpoint, example in compose, by @ajgon
• Added forward authentication to support auth proxies, docs, #403 by @ajgon
• Added years to duration strings for aggregations

Changed

• Changed biggest channel stats to three different sorted tables, by @MerlinScheurer
• Changed and consolidated duration parser to produce human readable string
• Changed and completed search processor and serializer to use API throughout
• Changed user configuration to store in ES, first step moving away from Redis plugins, by @anonamouslyginger
• Changed ES wrapper for better dynamic args builder
• Changed json backup to split into multiple smaller files, #406
• Old wiki is now removed, in favor of docs

Fixed

• Fixed URLs to settings docs, #545 by @MerlinScheurer
• Fixed yt-dlp playlist extraction
• Fixed various yt-dlp errors with newest release, #522

Project updates

• Due to an unfortunate misunderstanding between epoch seconds and milliseconds, this will trigger an index rebuild
  • Depending on your archive size, this can take anywhere from a few seconds to a few minutes to complete.
  • Be patient, follow the logs and let it do its thing.
• Tube Archivist Companion, the browser extension also got an update to v0.2.0 with lots of great additions. See the release notes there.

Added

• Added archive statistic dashboard
• [API] Added playlist delete endpoint
• [API] Added channel ID search endpoint
• [API] Added archive aggregation endpoints
• Added f hotkey for fullscreen to video player, by @dmynerd78

Changed

• Changed settings pages, are now split up for easier navigation, by @anonamouslyginger
• Changed redis config save to trigger bgsave
• Changed to ignore eaDir files from synology, #523
• Changed to reset autostart videos when stopping queue, #520

Fixed

• Fixed final thumbnail error handling
• Fixed validation error when subscribing to video in playlist, #299
• Fixed download folder list not using ingore_filelist
• [API] Fixed sponsorblock endpoint not returning 404, #526
• Fixed wrong index of epoch seconds/milliseconds in ES

Project updates, potentially Breaking Changes

• After trying and repeatedly failing to correctly handle Title and Channel Names for media files, we have decided to standardize that to <channel-id>/<video-id>.mp4.
  • Before updating, make sure Tube Archivist has all media files at the expected file system location.
  • There is a migration at startup that will reorganize all your files.
  • Only the default naming convention is supported for the migration. If you made changes there, this will likely fail and require you to migrate your files manually.
  • Monitor your startup logs for a progress message. Any errors will log there. If you see messages similar to did not find expected video/subtitle at ..., rename that manually to the expected path and restart for the migration to complete.
  • Also see FAQ.
• There is now a Jellyfin integration released: tubearchivist/tubearchivist-jf
  • This repos is renamed from tubearchivist/jellyfin to avoid confusion, that this is not a fork of jellyfin...
  • Shoutout to @jonasrosland for taking the initiative for the docker build
• The Docs now got a dedicated URLs page to document how TA treats URLs from YouTube, docs.
• As mentioned previously, Redis now publishes ARM builds for all it's extensions, including RedisJSON. To avoid confusion, I'll delete our now outdated custom build bbilly1/rejson, use redis/redis-stack-server instead.

Added

• Added Apprise integration for task notifications, docs
• Added days to duration converter
• Added error message for failed downloads with additional controls, docs
• Added new screenshots and showcasing, by @pairofcrocs
• Added support for case insensitive manual import, by @lamusmaser, #502
• [API] Added version number to ping API, by @lamusmaser

Changed

• Changed file system organization to <channel-id>/<video-id>.mp4.
• Changed trusted host configuration to add localhost and tubearchivist to defaults.
• Changed comment extraction to ignore errors by default, so the rest of comments can continue.
• [API] Date strings are now formatted in ISO format for consistency.

Fixed

• Fixed rescan process initiation without task
• Fixed startup race condition error in some cases
• Fixed retry logic for failing download videos, #477
• Fixed reindex progress message when dynamically adding to queue, #478
• Fixed handling missing tv art in thumbnail validator, #479
• Fixed redis retry logic at application start, #485
• Fixed missing auto subtitle extraction error handling, #496
• Fixed download cache file name building for better resume logic, #498

Project Updates

• This update will automatically change and rebuild the video, channel and download queue indexes.
• Tube Archivist Companion browser extension also got an update to control auto start behavior: Release Notes
• If you are a sponsor, the real time monitor client also got an update to control auto start behavior: Release Notes
• It is recommend to run Rescan Filesystem to validate all media file paths before updating, monitor the logs to make sure, this does nothing unexpected.
• At first start, there is a migration command running to index additional metadata from your media files. That includes file size, codec, bitrate, resolution.
  • That process can take some time, depending on various factors, expect this to take around 1 minute per 1000 videos.
  • Log output will show progress and any error messages.
  • The interface will become available again, after that completes.
  • Be patient and grab some popcorn to watch the logs fly by.

Added

• Added video stream metadata indexing like codecs, bitrate, filesize
• Added channel metadata aggregation like total file size, total videos, total playback.
• Added start now for adding to download queue, docs
• Added auto start for subscriptions, docs
• Added extractor language configuration, docs
• Added --format-sort configuration, docs, by @dsander
• Added channel tags indexing for better search results
• [API] Added endpoints to control auto start behavior

Changed

• Changed channel metadata extraction to use yt-dlp instead of custom scraper for better reliability.
• Removed the limit_count config field, use queue control instead

Fixed

• Fixed backup run issue when not initiated with task
• Fixed playlist ID parser for members only playlists, by @mglinski

Hotfix

• I've pushed a quick hotfix dealing with #476, if you encounter that, please pull again.

Project Updates

• All documentation has moved to a more flexible and easier to extend platform: docs.tubearchivist.com. This includes the old wiki pages, detailed installation instructions, API documentation and more. Shout out to @jonasrosland and @micah686 for doing the bulk of the transfer.
• Upon popular request, there is now a proof of concept Jellyfin integration: tubearchivist/jellyfin linking these two platforms together. This allows you to sync metadata from Tube Archivist to Jellyfin. Please test it and provide feedback.
• This fixes an inconsistent file and folder permission handling, additional to the media files, this will now also chown new channel folders and subtitle files with your HOST_UID and HOST_GID values, if you have set them. To retroactively apply that for all channel files run:

docker exec -it tubearchivist chown -R $UID:$GID /youtube

• The parsing of the TA_HOST variable is improved: This now should correctly translate most values that were previously through an error. If you are running TA behind SSL, you’ll have to specify the protocol in the variable, docs.
• To be able to offer something additional to my very real eternal gratitude for becoming a sponsor of this project, I created members.tubearchivist.com. This allows you to get realtime notifications from YouTube directly into your Tube Archivist to add new videos to your download queue.

Added

• Added a brand new task management backend, solving various pain points like falsely locking tasks, lack of notification to the frontend and more. This can be extended further in the future.
• [API] Added reset token endpoint, by @GavinPalmer1984
• [API] Added task by name endpoint
• [API] Added task by id endpoint
• [API] Added notifications endpoint
• Added task result backend using redis
• Added notification hooks for all background task
• Added additional TV artwork for channels
• Added recently published videos to daily refresh metadata task, docs

Changed

• [API] removed the old incomplete task endpoint in favor of new complete solution
• Changed how to interact with background tasks using Redis signal
• Changed error handling on startup checks, use a delay before restarting for better log feedback for some platforms
• Changed when yt-dlp tests the selected formats, speeding up add to queue, channel indexing and post processing
• Changed add to queue task to add videos while scanning for increased reliability.

Fixed

• Fixed arrow key shortcut jumping in fullscreen, by @bakkot
• Fixed inconsistent permission handling, new channel folders and subtitles follow env vars
• Fixed incorrect and error prone handling of TA_HOST environment var parsing, solution for https proxies, #441