Revised TMDB data fetching with improved retry logic and error handling. Refactored metadata management for movies and series, simplifying pipelines and introducing extensibility. Introduced `meta_fetcher` for centralized metadata fetching, replacing scattered IMDb and TMDB logic.
Introduced endpoints to check and submit cache statuses for info hashes. Enhanced cache helpers to integrate with MediaFusion Public Host for status retrieval and syncing. Added new schemas and configurations to support these operations.
Introduced a new Jackett scraper to support torrent indexers and refactored Prowlarr scraper for consistency. Enhanced support for advanced individual indexer searching, searching capability with category validation, and healthy indexer management across both scrapers.
This commit introduces the BT4G scraper to assist in scraping torrent streams for both movies and series. It includes new configuration options, caching mechanisms, and processing logic tailored for the BT4G platform. Additional settings around scraper limits and filtering have been added to ensure robust and efficient integration.
Introduced a new `YTSScraper` to fetch and process movie torrents from YTS. Updated configuration, runtime constants, and utilities to support the scraper. This enhancement extends scraping capabilities to include YTS as a source for movie streams.
Introduced `MoviesTVTgxSpider` for parsing TGx movie/TV torrents and added relevant pipelines for processing. Enhanced scheduler, constants, and configurations to support the new functionality. Made updates to improve torrent detail parsing and handle dynamic content loading using Playwright.
Introduced TMDB API support to fetch and update movie and series metadata, replacing parts of IMDb-based functionality. Enhanced the sports parser pipeline to retrieve TMDB metadata for events, ensuring consistency in handling posters, backgrounds, and descriptions.
Replaced custom proxy implementation and `curl-cffi` with unified `httpx` and `aiohttp` proxy support using `requests_proxy_url` config from settings. Simplified and standardized HTTP client instantiations across various modules for improved maintainability and consistency.
* Improve StremThru error handling
* Allow access to http response object
* Add configure time StremThru store name validation
* Store StremThru magnet cache in redis
* Add settings toggle for storing StremThru magnet cache
* Update streaming_providers/debrid_client.py
* Support using public StremThru instance
---------
Co-authored-by: Mohamed Zumair <mhdzumair@gmail.com>
Introduce a system for toggling streaming providers, allowing specific providers to be disabled via configuration. Enhance error handling in user data decryption by adding validation checks and returning precise error messages in JSON responses. These changes improve flexibility for administrators and provide better feedback to users.
Moved Redis client setup to a new module `redis_database` and updated all references across the codebase. Also added a new configuration setting for redis_max_connections for better resource management.
Enhanced the `update_rd_cache_status` function to check both RealDebrid and Zilean for cache status. Added configurations and HTTP request logic to include Zilean's cache check API using an optional authentication key. This improves the accuracy and coverage of cache status updates.
* Add support for comprehensive scraper metrics tracking
Renamed scrape_and_parse to _scrape_and_parse, adding comprehensive error handling and logging metrics for various scenarios like timeouts, HTTP errors, and validation issues. Incorporated a metrics class in base_scraper.py to summarize statistics related to scraping performance, which is now used across different scrapers including prowlarr, torrentio, and zilean.
* Enhance Prowlarr individual indexer searching logic with indexer healthcheck management & comprehensive metrics summary
Added detailed health checks for indexers and included their statuses in logging. Enhanced background search to use circuit breakers and handle indexers in chunks, improving reliability and fault tolerance.
* Handle custom id items with imdb id based on title and year matching for moving to imdb title
* handle exception onf fetching prowlarr indexer torrent page fetching
* Refactor Circuit Breaker and add state management methods
Reorganized Circuit Breaker class for better clarity and maintainability. Added methods to manage states (`is_closed`, `reset`, `record_failure`, `record_success`) and refactored `call` method to utilize these state checks. Enhanced logging and status reporting with `get_status` method.
* Simplify series metadata & episode data retrieval logic
Streamlined the `get_series_meta` function by removing complex filtering conditions and reducing the aggregation pipeline.
* Refactor CircuitBreaker for web scraping and enhanced recovery
Removed git revision from the version string in the API response. Added a new field for branding description in settings and integrated it in the home template. Introduced VERSION argument and environment variable in the Dockerfile.
* Refactor ZileanScraper to use parallel requests for searching and filtering streams with new endpoints
* Fix DLHD scraping & enable DLHD without MediaFlow
* Switch to httpx for async HTTP requests
* Implement caching for PikPak token to reduce login error
* Refactor torrent cleanup logic
* Add inactivity monitor extension to close idle spiders
Introduced `InactivityMonitor` to automatically close spiders that remain inactive for a specified time. This extension checks activity at regular intervals and uses configurable settings for check intervals and inactivity timeouts. If no items are scraped within the timeout period, the spider is closed to free resources.
* Handle TypeError in dynamic sorting of streams
* Refactor torrent info scraper to support pre-processing.
Introduced a pre-processing function mapping to handle specific indexer requirements before parsing the HTML. Added a custom pre-processing function for "TheRARBG" to handle URL adjustments, improving modularity and readability in the `get_torrent_info` function.
* Refine error logging and fix hash key in spider
* Fix Prowlarr not stop on max process limit
* #315: Integrate ScrapeOps logging into all scrapers & spiders
Added ScrapeOps logging to Zilean, Torrentio, Prowlarr, and Prowlarr Feed scrapers to enhance request tracking and error handling. Configured ScrapeOps API key in settings and updated Pipfile/Pipfile.lock with scrapeops-python-requests and scrapeops-scrapy dependencies.
* Refactor scraper cache status handler
* verify torrent before parsing on prowlarr & prioritize magnet on badass_torrents
* Add support for provide locally hosted mediaflow proxy public address and reduce the time leg on private ip address checking
* handle RD exception cases
* do not setup scrapeops when api key is none
* Enhanced dynamic sorting of torrent streams
Revised the dynamic_sort_key function to handle different key types more efficiently with match-case. Simplified error handling and improved logging to capture sorting data in the case of exceptions.
* add missing last update date for metadata
* update domain for nowmesports
* Refactor stream parsing and handling for DRM and MediaFlow
* Add encryption support for mediaflow url and refactor URL validation
* Enhance LiveTV spider to support MPD and handle new player API logic
* Update TV stream validation and add DRM fields in UI
* Add mediaflow_proxy_required.mp4 exception video
* Add FlareSolverr integration to scrapy spiders
* [WIP] TamilBulb spider to scraper
* Update StreamedSpider to new API and adjust schedulers
mhdzumair
Munif Tanjim
ragmehos