430 Commits

Author	SHA1	Message	Date
Hongbo Wu	ef6c6349de	add remove_on_complete and remove_on_fail to the unique job	2024-02-07 18:03:16 +08:00
Hongbo Wu	468f835f9e	Merge pull request #3502 from omnivore-app/fix/bullmq-job-retry-backoff ... increase the seed valud of backoff delay to 2 seconds	2024-02-06 15:09:40 +08:00
Hongbo Wu	a939180dc9	fix: dedupe update-db job by using library_item_id as job_id	2024-02-06 14:10:53 +08:00
Hongbo Wu	53f63b90d7	increase the seed valud of backoff delay to 2 seconds	2024-02-05 16:26:05 +08:00
Jackson Harper	0a38cd6422	More attempts now that lock timeouts are lower	2024-02-03 19:08:13 +08:00
Hongbo Wu	1742ffda24	change priority to 5	2024-02-02 16:18:41 +08:00
Hongbo Wu	25e374f6ff	fix: replace webhook endpoint with a bullmq job	2024-02-02 16:18:40 +08:00
Hongbo Wu	26b7103f0b	improve batch update	2024-02-02 13:13:18 +08:00
Hongbo Wu	1bccf33320	add job_id and parent for tracking	2024-02-02 10:50:07 +08:00
Hongbo Wu	abaf726044	revert the change of priority	2024-02-01 13:22:06 +08:00
Hongbo Wu	19fe60d27a	change the update-db job priority to 5 and use exponential backoff strategy for retrying	2024-02-01 13:14:01 +08:00
Hongbo Wu	c3276fdaf1	Merge pull request #3466 from omnivore-app/fix/rewrite-trigger-functions-in-db ... create a job for db trigger	2024-01-31 19:47:45 +08:00
Hongbo Wu	1b8111b907	wait until all jobs are done in test	2024-01-31 19:10:57 +08:00
Hongbo Wu	5646413b59	add a job for updating highlight annotations in library_item	2024-01-31 14:15:33 +08:00
Hongbo Wu	5ed35bbeff	create a job for update labels in library item	2024-01-31 13:32:37 +08:00
Hongbo Wu	ddddf8234e	create a job for db trigger	2024-01-30 23:53:05 +08:00
Hongbo Wu	36d2c21466	remove thumbnail job on complete or failure	2024-01-30 18:24:48 +08:00
Hongbo Wu	2eedba1522	fix: assign the rule job with priority 1 and do not retry	2024-01-30 18:14:34 +08:00
Hongbo Wu	809c572ca7	reduce the size of the log entry by truncating any string values to 500 characters	2024-01-29 14:32:45 +08:00
Hongbo Wu	d5d581fc54	remove redundant logs	2024-01-29 14:32:43 +08:00
Hongbo Wu	30fef30384	resolve conflicts	2024-01-26 20:32:56 +08:00
Hongbo Wu	e03895c8b7	enqueue trigger rule job once page is created or updated	2024-01-26 17:46:17 +08:00
Hongbo Wu	7556c6da5f	move trigger rule into a job	2024-01-26 17:45:40 +08:00
Hongbo Wu	52419c39da	resolve conflicts	2024-01-26 13:01:53 +08:00
Hongbo Wu	7e89235806	update content-fetch	2024-01-26 13:01:53 +08:00
Hongbo Wu	4c765378e3	do not retry if find_thumbnail job fails	2024-01-26 13:01:53 +08:00
Hongbo Wu	e54c691c09	create a job for finding thumbnail	2024-01-26 13:01:51 +08:00
Hongbo Wu	fd7c2ffb49	fix code blocks not formatted correctly in articles from wechat official accounts	2024-01-25 16:30:59 +08:00
Hongbo Wu	dae90aaffa	fix code blocks not formatted correctly	2024-01-25 16:30:59 +08:00
Hongbo Wu	5e239d2568	run readability in save-page instead of puppeteer	2024-01-25 16:30:59 +08:00
Hongbo Wu	68a4fb298d	add refreshedAt to the gql schema	2024-01-24 12:03:26 +08:00
Hongbo Wu	8f5d4ae8a0	fix: catch rejected promise if it is in a try-catch block	2024-01-23 18:45:46 +08:00
Jackson Harper	56a8ef23b6	Remove authorized function from imported helpers ... This creates some issues with init order when using the queue processor.	2024-01-21 13:22:17 +08:00
Jackson Harper	db3d7dca36	Add missing file	2024-01-21 07:43:50 +08:00
Jackson Harper	2961f69f02	Add a refresh context to make debugging rss jobs easier	2024-01-20 09:47:46 +08:00
Jackson Harper	84375b7c36	Add validation before refreshing feed URLs	2024-01-19 16:25:05 +08:00
Jackson Harper	fb28d7b3d1	Enqueue new feeds at front	2024-01-18 14:34:07 +08:00
Jackson Harper	9e6b749604	Update error message	2024-01-17 16:21:42 +08:00
Jackson Harper	e1e2f86219	Remove duplicated error	2024-01-17 16:20:42 +08:00
Jackson Harper	7a925f3fdc	Clean up redis code, remove connection close in job	2024-01-17 11:14:59 +08:00
Jackson Harper	50b48a4225	Separate worker and client code for redis connections	2024-01-16 23:21:48 +08:00
Jackson Harper	7c8304ac15	Fix client imports	2024-01-16 22:08:30 +08:00
Jackson Harper	e050864a91	Add more debugging	2024-01-16 21:30:17 +08:00
Jackson Harper	62fdb866a8	Add some extra logging	2024-01-16 21:30:17 +08:00
Jackson Harper	5986bd0a68	Change around redis initialization so we can pull in secrets from secret manager async on startup	2024-01-16 21:30:15 +08:00
Jackson Harper	fc25d6d615	Enqueue rss-refresh-feed jobs in bull	2024-01-16 21:25:31 +08:00
Hongbo Wu	150a456c35	replace redis client library with ioredis	2024-01-16 15:42:50 +08:00
Hongbo Wu	816019e6a3	extend ttl of the saved item cache to 26 hours	2024-01-16 11:17:21 +08:00
Hongbo Wu	ccd4f2d3f0	catch redis client error and continue the process	2024-01-05 15:20:42 +08:00
Hongbo Wu	bcd05cc7fc	set the url in redis when an item is inserted into db	2024-01-04 18:10:14 +08:00