Reworked the caching logic to handle a missing cache key for "events" type gracefully. Integrated a new DLHD schedule service for improved event metadata and stream handling, leveraging fuzzy matching and IPTV-org data retrieval for enhanced TV channel mappings. Updated scraper configuration, enhanced genre handling, and added support for dynamic event descriptions.
Refactored sports metadata validation by removing unnecessary fields like 'year' and making 'poster', 'background', and 'logo' optional with fallback values. Improved the handling of catalogs, series detection, and created conditional title adjustments. Updated UI and JSON artifacts to support individual WWE and UFC categories.
Previously, non-English catalogs were being processed for TorrentGalaxy sources, leading to incorrect catalog entries. This update ensures non-English catalogs from TorrentGalaxy are ignored, preserving data integrity. Other sources remain unaffected.
Revised TMDB data fetching with improved retry logic and error handling. Refactored metadata management for movies and series, simplifying pipelines and introducing extensibility. Introduced `meta_fetcher` for centralized metadata fetching, replacing scattered IMDb and TMDB logic.
Updated the torrent parsing logic to add timestamp conversion, improve file data extraction, and refine metadata validation. Adjusted season and episode parsing for better accuracy in file metadata.
Update Crud functions for new database structure & utlize 10x performance in the catalog data fetchers with new db structure and mongodb indexes. Added support for multi season pack torrents, Updated metadata handling to use a centralized fetching mechanism and improved filters for matching existing entries. Refactored episode processing logic to account for season and episode numbers more flexibly. Enhanced querying pipelines for better performance in catalog and metadata retrieval.
Added `parsed_data` support for more robust torrent metadata extraction and improved season/episode assignment logic. Refined IMDB integration to handle null ratings and streamline title-year searches. Database index and query filters updated for better flexibility and accuracy.
Integrate `dramatiq-abort` with Redis backend for task abortion and enhance the `CloseSpider` extension to abort jobs with no recent activity. These changes improve task management and error handling for idle spiders in the scraping process. Updated dependencies and project settings accordingly.
Introduced `MoviesTVTgxSpider` for parsing TGx movie/TV torrents and added relevant pipelines for processing. Enhanced scheduler, constants, and configurations to support the new functionality. Made updates to improve torrent detail parsing and handle dynamic content loading using Playwright.
Improve clarity and consistency by refactoring metadata creation, episode organization, and calls related to IMDb searches. Centralize and reuse the `organize_episodes` functionality, removing redundant code and improving maintainability. Adjust pipelines for better handling of Redis connections and metadata attributes.
Removed unnecessary size conversion logic to simplify code and reduce dependencies. Enhanced catalog processing with validation against predefined data and prevented duplication. Adjusted logging levels for better error visibility in TMDB integration.
Introduced TMDB API support to fetch and update movie and series metadata, replacing parts of IMDb-based functionality. Enhanced the sports parser pipeline to retrieve TMDB metadata for events, ensuring consistency in handling posters, backgrounds, and descriptions.
Moved Redis client setup to a new module `redis_database` and updated all references across the codebase. Also added a new configuration setting for redis_max_connections for better resource management.
Removed unnecessary link fields in models and streamlined creation and organization of torrent streams in store pipelines. Updated the organize_episodes function to fetch streams directly from the database and async functionality for saving episodes back to their respective streams.
* Refactor ZileanScraper to use parallel requests for searching and filtering streams with new endpoints
* Fix DLHD scraping & enable DLHD without MediaFlow
* Switch to httpx for async HTTP requests
* Implement caching for PikPak token to reduce login error
* Refactor torrent cleanup logic
* Add inactivity monitor extension to close idle spiders
Introduced `InactivityMonitor` to automatically close spiders that remain inactive for a specified time. This extension checks activity at regular intervals and uses configurable settings for check intervals and inactivity timeouts. If no items are scraped within the timeout period, the spider is closed to free resources.
* Handle TypeError in dynamic sorting of streams
* Refactor torrent info scraper to support pre-processing.
Introduced a pre-processing function mapping to handle specific indexer requirements before parsing the HTML. Added a custom pre-processing function for "TheRARBG" to handle URL adjustments, improving modularity and readability in the `get_torrent_info` function.
* Refine error logging and fix hash key in spider
* Fix Prowlarr not stop on max process limit
* #315: Integrate ScrapeOps logging into all scrapers & spiders
Added ScrapeOps logging to Zilean, Torrentio, Prowlarr, and Prowlarr Feed scrapers to enhance request tracking and error handling. Configured ScrapeOps API key in settings and updated Pipfile/Pipfile.lock with scrapeops-python-requests and scrapeops-scrapy dependencies.
* Refactor scraper cache status handler
* verify torrent before parsing on prowlarr & prioritize magnet on badass_torrents
* Add support for provide locally hosted mediaflow proxy public address and reduce the time leg on private ip address checking
* handle RD exception cases
* do not setup scrapeops when api key is none
* Enhanced dynamic sorting of torrent streams
Revised the dynamic_sort_key function to handle different key types more efficiently with match-case. Simplified error handling and improved logging to capture sorting data in the case of exceptions.
* add missing last update date for metadata
* update domain for nowmesports
* Refactor stream parsing and handling for DRM and MediaFlow
* Add encryption support for mediaflow url and refactor URL validation
* Enhance LiveTV spider to support MPD and handle new player API logic
* Update TV stream validation and add DRM fields in UI
* Add mediaflow_proxy_required.mp4 exception video
* Add FlareSolverr integration to scrapy spiders
* [WIP] TamilBulb spider to scraper
* Update StreamedSpider to new API and adjust schedulers
* Add Support for download streams via web browser & support disabling the metadata for imdb title
* Add env config to enable or disable download via browser feature
Mohamed Zumair