Replaced flat `catalogs` structure with detailed `catalog_stats` to support per-catalog metadata (e.g., stream counts, last added date). Updated query pipelines, database schema, and index definitions accordingly. Enhanced stream update/delete logic for more granular catalog-level tracking.
Introduce the `is_custom` boolean field to `MediaFusionMetaData` with a default value of `False`. Enhance unique index behavior by adding a partial filter for custom IDs and adjust other indexes for better query support. Include support for setting `is_custom` during metadata creation in CRUD operations.
Update Crud functions for new database structure & utlize 10x performance in the catalog data fetchers with new db structure and mongodb indexes. Added support for multi season pack torrents, Updated metadata handling to use a centralized fetching mechanism and improved filters for matching existing entries. Refactored episode processing logic to account for season and episode numbers more flexibly. Enhanced querying pipelines for better performance in catalog and metadata retrieval.
- Updated streaming providers to allow adding torrents by file for webseed and private torrents.
- Implemented functionality to store the content of torrent files.
Added `parsed_data` support for more robust torrent metadata extraction and improved season/episode assignment logic. Refined IMDB integration to handle null ratings and streamline title-year searches. Database index and query filters updated for better flexibility and accuracy.
Removed unnecessary link fields in models and streamlined creation and organization of torrent streams in store pipelines. Updated the organize_episodes function to fetch streams directly from the database and async functionality for saving episodes back to their respective streams.
* Refactor ZileanScraper to use parallel requests for searching and filtering streams with new endpoints
* Fix DLHD scraping & enable DLHD without MediaFlow
* Switch to httpx for async HTTP requests
* Implement caching for PikPak token to reduce login error
* Refactor torrent cleanup logic
* Add inactivity monitor extension to close idle spiders
Introduced `InactivityMonitor` to automatically close spiders that remain inactive for a specified time. This extension checks activity at regular intervals and uses configurable settings for check intervals and inactivity timeouts. If no items are scraped within the timeout period, the spider is closed to free resources.
* Handle TypeError in dynamic sorting of streams
* Refactor torrent info scraper to support pre-processing.
Introduced a pre-processing function mapping to handle specific indexer requirements before parsing the HTML. Added a custom pre-processing function for "TheRARBG" to handle URL adjustments, improving modularity and readability in the `get_torrent_info` function.
* Refine error logging and fix hash key in spider
* Fix Prowlarr not stop on max process limit
* #315: Integrate ScrapeOps logging into all scrapers & spiders
Added ScrapeOps logging to Zilean, Torrentio, Prowlarr, and Prowlarr Feed scrapers to enhance request tracking and error handling. Configured ScrapeOps API key in settings and updated Pipfile/Pipfile.lock with scrapeops-python-requests and scrapeops-scrapy dependencies.
* Refactor scraper cache status handler
* verify torrent before parsing on prowlarr & prioritize magnet on badass_torrents
* Add support for provide locally hosted mediaflow proxy public address and reduce the time leg on private ip address checking
* handle RD exception cases
* do not setup scrapeops when api key is none
* Enhanced dynamic sorting of torrent streams
Revised the dynamic_sort_key function to handle different key types more efficiently with match-case. Simplified error handling and improved logging to capture sorting data in the case of exceptions.
* add missing last update date for metadata
* update domain for nowmesports
* Refactor stream parsing and handling for DRM and MediaFlow
* Add encryption support for mediaflow url and refactor URL validation
* Enhance LiveTV spider to support MPD and handle new player API logic
* Update TV stream validation and add DRM fields in UI
* Add mediaflow_proxy_required.mp4 exception video
* Add FlareSolverr integration to scrapy spiders
* [WIP] TamilBulb spider to scraper
* Update StreamedSpider to new API and adjust schedulers
* feat: Sort & Filter stream languages, quality based on user preference
* feat: Use field aliases & compress after encrypt user data to reduce the config user data size
* Refactor catalog parsing logic and add CatalogParsePipeline
* Update language handling in stream creation
* update packages
* Update README
* refactor parse torrent streams
* use lru cache for common function
* handle item checking
* address code review
* Update PTT & parse complete language
* cleanup
mhdzumair
Rag