6 Commits

Author	SHA1	Message	Date
mhdzumair	863247018d	Add dramatiq-abort support and extend scrapy CloseSpider to abort the dramatiq worker on stale spiders ... Integrate `dramatiq-abort` with Redis backend for task abortion and enhance the `CloseSpider` extension to abort jobs with no recent activity. These changes improve task management and error handling for idle spiders in the scraping process. Updated dependencies and project settings accordingly.	2024-12-15 23:01:24 +05:30
mhdzumair	f73b1e436f	add Prometheus metrics & API metrics endpoint to get stats	2024-05-01 12:39:49 +05:30
mhdzumair	3779efbd6b	Implement TaskManager for dramatiq worker to maintain minimum iterval before triggering same scraper task & worker improvments	2024-04-27 17:08:55 +05:30
mhdzumair	5721e48b42	Add MaxTasksPerChild middleware for Dramatiq broker to resolve memory leak	2024-04-24 09:28:44 +05:30
mhdzumair	5fadfac0a9	Add dramatiq library and enhance prowlarr search functionality	2024-02-03 19:57:35 +05:30
mhdzumair	4414141c8c	change file structure	2022-09-10 20:15:05 +05:30