Commit Graph

172 Commits

Author SHA1 Message Date
mhdzumair 4d96ae86cc Refactor middleware error handling and update middleware order to print access logs
Consolidate error handling logic across middleware components to improve maintainability. Adjust the order of middleware addition in the app for improved logging and user data access.
2024-11-02 22:59:20 +05:30
Mohamed Zumair 467f378207 Add Configuration validations & loading widget for responsiveness (#342) 2024-11-02 19:56:02 +05:30
Mohamed Zumair 481891e324 Update version handling and add branding description (#341)
Removed git revision from the version string in the API response. Added a new field for branding description in settings and integrated it in the home template. Introduced VERSION argument and environment variable in the Dockerfile.
2024-11-02 07:15:16 +05:30
Mohamed Zumair 05fbafe8e6 Add configuration mode & Refactor stream filtering and catalog configuration (#340)
* Add configuration mode switch for user customization

Introduce a 'Configuration Mode' switch to toggle between 'Newbie' and 'Pro' modes, allowing users to select between simplified and advanced configurations. Updated CSS and JS to handle the display of specific settings and sections based on the selected mode, enhancing user experience and customization options.

* Update default language filters and sorting priority.

* Refactor stream filtering and catalog configuration.

Removed redundant catalog selection logic and refactored stream filtering to streamline the code. Updated configuration scripts and UI to align with new functionality, leading to cleaner and more maintainable code.
2024-11-01 23:00:41 +05:30
Mohamed Zumair b19ac4442b Improve health check functionality (#338)
Enhanced the /health endpoint to perform an HTTP HEAD request to Google, returning the status code and response time. This change provides a more accurate health status and logs errors if the check fails.
2024-11-01 16:18:39 +05:30
Mohamed Zumair 9ebf4b8092 Generate manifest.json using Jinja template for cleanest & 2.5x faster (#329) 2024-10-19 22:44:34 +05:30
Mohamed Zumair ecaf7e5cf1 Added filter for only show cached stream, Redis lock for prevent concurrent live scrape, bug fixes & improvements (#325)
* cleanup lifespan vars

* Show API process time in logs

* store the cache validation of full URL for live sports event

* Add support for filter only show cached streams

* Update default configs

* properly handle prowlarr exception on MaxProcessLimitReached

* Add redis lock mechanism to prevent concurrent live scrapers from multiple instances of stremio addon installation users
2024-10-15 18:43:03 +05:30
Mohamed Zumair 52d8956034 Add ScrapeOps monitoring, Add zilean filtered endpoint, Fix Prowlarr scraping with download link & Fix PikPak login error etc (#319)
* Refactor ZileanScraper to use parallel requests for searching and filtering streams with new endpoints

* Fix DLHD scraping & enable DLHD without MediaFlow

* Switch to httpx for async HTTP requests

* Implement caching for PikPak token to reduce login error

* Refactor torrent cleanup logic

* Add inactivity monitor extension to close idle spiders

Introduced `InactivityMonitor` to automatically close spiders that remain inactive for a specified time. This extension checks activity at regular intervals and uses configurable settings for check intervals and inactivity timeouts. If no items are scraped within the timeout period, the spider is closed to free resources.

* Handle TypeError in dynamic sorting of streams

* Refactor torrent info scraper to support pre-processing.

Introduced a pre-processing function mapping to handle specific indexer requirements before parsing the HTML. Added a custom pre-processing function for "TheRARBG" to handle URL adjustments, improving modularity and readability in the `get_torrent_info` function.

* Refine error logging and fix hash key in spider

* Fix Prowlarr not stop on max process limit

* #315: Integrate ScrapeOps logging into all scrapers & spiders

Added ScrapeOps logging to Zilean, Torrentio, Prowlarr, and Prowlarr Feed scrapers to enhance request tracking and error handling. Configured ScrapeOps API key in settings and updated Pipfile/Pipfile.lock with scrapeops-python-requests and scrapeops-scrapy dependencies.

* Refactor scraper cache status handler

* verify torrent before parsing on prowlarr & prioritize magnet on badass_torrents

* Add support for provide locally hosted mediaflow proxy public address and reduce the time leg on private ip address checking

* handle RD exception cases

* do not setup scrapeops when api key is none

* Enhanced dynamic sorting of torrent streams

Revised the dynamic_sort_key function to handle different key types more efficiently with match-case. Simplified error handling and improved logging to capture sorting data in the case of exceptions.

* add missing last update date for metadata

* update domain for nowmesports
2024-10-14 05:58:23 +05:30
Mohamed Zumair 7103d03428 #312: Fix IMDb data scraping (#313)
* #312: Fix IMDb data scraping

* Handle NoneType in IMDb scraper and improve type annotations
2024-10-06 22:49:20 +05:30
Mohamed Zumair 64773baecb Add torrent blocking feature for DMCA (#302) 2024-09-28 09:44:53 +05:30
Mohamed Zumair 0e7a5126f5 Add centralized config management for spiders & Remove deprecated spiders and update related configurations (#299)
* Add centralized config management for spiders

* Remove deprecated spiders and update related configurations
2024-09-27 21:50:29 +05:30
Mohamed Zumair 77042b2054 Fallback imdb metadata scraping via cinemeta & Scrapy Ratelimit middleware & metadata empty cache bugfixes (#290)
* Add ratelimit retry middleware for scrapy & bugfixes

* #289: bugfix metadata & catalog empty cache error

* Add support for fallback imdb metadata scraping through cinemeta API
2024-09-18 10:13:53 +05:30
Mohamed Zumair c8f660b7ee Add Support for download streams via web browser & support disabling the metadata for imdb title (#287)
* Add Support for download streams via web browser & support disabling the metadata for imdb title

* Add env config to enable or disable download via browser feature
2024-09-18 06:54:37 +05:30
mhdzumair 3119d809eb validate secret str character limit and correctly capture the results 2024-09-16 12:50:51 +05:30
mhdzumair f5e2b30dfe bugfixes & adding missing libs 2024-09-16 11:01:01 +05:30
Mohamed Zumair cf8ed59012 Added Prowlarr feed scraping & Improve Advanced scraping capability for prowlarr, zilean, torrentio & more bugfixes & improvements (#286)
* Update Python version

* Refactor Redis client

* WIP: Refactor scrapers

* Added Prowlarr feed scraping & Improve Advanced scraping capability for prowlarr, zilean, torrentio & more bugfixes & improvements

* Improve batch_process_with_circuit_breaker implementation to yield an individual task as soon as completed
2024-09-16 08:26:57 +05:30
Mohamed Zumair d169b28aec Add support for RPDB posters & Support fallback posters via MediaFusion posters (#284) 2024-09-12 12:17:33 +05:30
Mohamed Zumair 76980a92dd Added new MediaFusion configuration setup for Kodi Add-On with QR code & Configure URL setup (#282)
* Added new MediaFusion configuration setup for Kodi Add-On with QR code & Configure URL setup

* remove test codes & correctly generate qr code with existing secret
2024-09-12 01:59:28 +05:30
Mohamed Zumair 649de9e503 Update Python version & refactror Redis client usage and streaming provider route endpoints (#277)
* Update Python version

* Refactor Redis client

* Refactor streaming routes endpoint functions
2024-09-05 06:46:07 +05:30
Mohamed Zumair e502d0f10e Add support for MediaFlow Proxy for Debrid stream & Live stream (#271)
* Add support for MediaFlow Proxy for Debrid stream & Live stream

* Add emoji for mediaflow proxy stream and add error logging
2024-08-29 10:02:01 +05:30
Mohamed Zumair 3ccd1f8a55 Sort & Filter stream languages, quality based on user preference & Reduce userdata size (#250)
* feat: Sort & Filter stream languages, quality based on user preference

* feat: Use field aliases & compress after encrypt user data to reduce the config user data size

* Refactor catalog parsing logic and add CatalogParsePipeline

* Update language handling in stream creation

* update packages

* Update README

* refactor parse torrent streams

* use lru cache for common function

* handle item checking

* address code review

* Update PTT & parse complete language

* cleanup
2024-08-26 22:38:27 +05:30
Mohamed Zumair 130b4c06a5 Add support for parental guides nudity filter and certification filter in user config (#244)
* Add support for parental guides nudity filter and certification filter in user config

* refactor validator according to review
2024-07-21 09:18:19 +05:30
mhdzumair 3ceb4c2849 exception handling in get_poster function 2024-07-20 06:56:11 +05:30
mhdzumair b2e6692582 only scrap the first page for tgx daily scheduler params 2024-07-08 21:40:32 +05:30
mhdzumair 38fd344024 Added support for filter movie & series by genre 2024-07-08 00:57:55 +05:30
mhdzumair ad57412761 Added support for Fighting (WWE & UFC) scrapy for TGX 2024-07-08 00:13:58 +05:30
mhdzumair 8e32e2ccaf bugfix variable typo 2024-06-29 05:44:32 +05:30
mhdzumair ba60d58aad Add missing arab-torrents scheduler 2024-06-28 12:00:49 +05:30
mhdzumair f563d49a30 Rebrand scripts of mhdtv to nowmetv 2024-06-09 23:17:12 +05:30
mhdzumair 5488504de5 Add metrics dashboard UI for viewing torrents info, scrapy schedule info & etc 2024-06-07 23:14:08 +05:30
mhdzumair 4b926b0535 Add support namespace in Live TV streams to store specific to elfhosted tenants namespace 2024-06-03 13:17:07 +05:30
mhdzumair 24d645a730 feat: Refactor TV stream validation and update logic for schedule function 2024-06-03 07:05:11 +05:30
mhdzumair 08218bb5c2 feat: Add support for scrape seeders count from trackers & daily scheduler job to scrape 2024-06-03 00:54:06 +05:30
mhdzumair 57e2993df3 chore: Import IMDb data scraper 2024-05-27 06:03:14 +05:30
mhdzumair ade259cd57 PikPak: Fix auth errror & Update streaming provider username field to email 2024-05-26 07:21:54 +05:30
mhdzumair a35d6a938c prowlarr: Refactor update torrent stream data & add support for title matching with AKA titles 2024-05-25 22:45:40 +05:30
mhdzumair f3cb84a041 remove contribution streams by default with an option to enable through catalog 2024-05-20 22:55:27 +05:30
mhdzumair 9b22370753 fix: Preserve user sorting options in configure 2024-05-19 23:28:11 +05:30
mhdzumair d91d067b83 feat: Add support for PikPak task autoretry & Improve torrent name finder & Exc handling, Refactors 2024-05-19 23:00:07 +05:30
mhdzumair 455093f79d feat: Add support for user import data & scraper 2024-05-19 16:52:40 +05:30
mhdzumair 27679ce6e1 feat: Add support for quickly trigger IMDb data update within stremio 2024-05-15 11:28:25 +05:30
mhdzumair cf2ee1f5fe Refactor poster URLs to use separate host for images 2024-05-12 23:55:42 +05:30
mhdzumair 45bf4e01d4 Add support for forward user's IP to Alldebrid & refactoring streaming providers 2024-05-12 19:02:40 +05:30
mhdzumair d8a074aca7 Add MotoGP TGX scheduler and update config settings 2024-05-07 14:04:59 +05:30
mhdzumair 9768a19a0e Add support for MotoGP & refactor 2024-05-07 13:58:43 +05:30
mhdzumair c5ad0a89f3 Add support for parent guide nudity & certificate filters & various improvments to crud functions 2024-05-06 00:16:46 +05:30
mhdzumair 02814c5fda create imdb scraping data functions & schedule job for periodic scraping 2024-05-05 23:59:30 +05:30
mhdzumair 14fe44d71c Make logo and addon name can be configurable through env 2024-05-01 14:01:52 +05:30
mhdzumair f73b1e436f add Prometheus metrics & API metrics endpoint to get stats 2024-05-01 12:39:49 +05:30
mhdzumair 7ff883678b Add DaddyLiveHD scraper and scheduler 2024-04-28 22:47:57 +05:30