Commit Graph

127 Commits

Author SHA1 Message Date
927394e07c fix: save url operation is delayed 2024-02-21 17:54:08 +08:00
819135f118 Label the content-fetch and web images 2024-02-21 14:16:21 +08:00
856a7bc5ef Add health checks to content-fetch 2024-02-12 14:42:51 +08:00
86f4553dd1 fix: duplicate key value violates unique constraint "library_item_pkey" 2024-02-07 20:41:34 +08:00
9a342f273c Increase delay on save page retry 2024-02-06 12:41:11 +08:00
9e6b5d2dcf Backoff for save page 2024-02-03 15:35:08 +08:00
fd80724d79 fallback to guid if link of rss item is not available 2024-01-26 22:09:53 +08:00
52419c39da resolve conflicts 2024-01-26 13:01:53 +08:00
e4e07c8acf cache fetched content for 24 hours 2024-01-26 13:01:53 +08:00
7e89235806 update content-fetch 2024-01-26 13:01:53 +08:00
5e239d2568 run readability in save-page instead of puppeteer 2024-01-25 16:30:59 +08:00
f4a2edf8f4 show detailed logs 2024-01-23 16:58:08 +08:00
fd0bb37d3a make title optional for cache 2024-01-23 16:54:35 +08:00
1332cda356 give highest priority to save-page job for adding items by links 2024-01-23 16:48:57 +08:00
eac93e6cb6 Reduce logging 2024-01-22 12:02:35 +08:00
6ab737432b More logging 2024-01-21 15:31:16 +08:00
52050728ba Add more debugging logs 2024-01-20 10:29:07 +08:00
6b89f4b809 graceful shutdown redis 2024-01-19 18:06:12 +08:00
356ad38678 reduce logs 2024-01-19 18:01:09 +08:00
4340286b89 add retry on failure in jobs 2024-01-19 17:38:25 +08:00
8acdea5002 set cache if not exists 2024-01-19 17:14:49 +08:00
51d5166321 get cached content from redis when saving pages 2024-01-19 17:06:37 +08:00
699d38609b cache fetch result in redis 2024-01-19 16:58:24 +08:00
7c25a628d9 fix pem format 2024-01-19 11:44:16 +08:00
2e8f173879 create queue on start up 2024-01-19 11:26:53 +08:00
a6d7623cea do not close redis connection 2024-01-19 10:39:10 +08:00
846d1e3381 fix: build content-fetch in dockerfile 2024-01-19 10:13:05 +08:00
d2c17f7539 graceful shutdown redis 2024-01-18 21:52:33 +08:00
3e291d1019 fix tests 2024-01-18 21:13:32 +08:00
d4f49dceb6 remove job id 2024-01-18 18:48:46 +08:00
a6f0e2f2d9 add save page job processor 2024-01-18 18:48:46 +08:00
43df5d30a8 prioritized jobs 2024-01-18 18:48:46 +08:00
0015a946b2 create batch save page jobs after fetching content 2024-01-18 18:48:46 +08:00
5136874372 remove debug logs 2024-01-18 18:48:46 +08:00
d9feb740cb convert content-fetch to typescript 2024-01-18 18:48:46 +08:00
cd3402b98a rewrite puppeteer in typescript 2024-01-18 18:48:46 +08:00
51e586ed3d separate content-fetch in puppeteer packages from saving page content 2024-01-18 18:48:46 +08:00
41805d13db Bump @sentry/serverless from 6.19.3 to 7.77.0
Bumps [@sentry/serverless](https://github.com/getsentry/sentry-javascript) from 6.19.3 to 7.77.0.
- [Release notes](https://github.com/getsentry/sentry-javascript/releases)
- [Changelog](https://github.com/getsentry/sentry-javascript/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-javascript/compare/6.19.3...7.77.0)

---
updated-dependencies:
- dependency-name: "@sentry/serverless"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-07 03:35:00 +00:00
8ddfa0a389 Fixes for new docker images 2023-10-26 11:19:38 +08:00
939a00b893 Update yarn file 2023-10-25 16:58:11 +08:00
16784728cf Use debian for base content-fetch container 2023-10-25 09:44:15 +08:00
c4773dc904 Landing page improvements and various supporting improvements 2023-10-24 09:43:39 +01:00
4b171c0657 docs: fix typos in packages/content-fetch/README.md 2023-10-18 20:35:35 +05:45
a97fcd1e88 do not use single process in chromium 2023-08-02 15:58:32 +08:00
63cbb3011e upgrade puppeteer and update chromium args 2023-08-02 15:33:15 +08:00
fa1ff9ba17 Upgrade node to 18.16 2023-07-24 15:55:11 +08:00
6cd6994aff Fix docker image 2022-12-28 15:28:27 +08:00
a5f5e6fbdb Fix docker build 2022-12-28 11:51:11 +08:00
e866541ae1 Fix puppeteer launch in head mode 2022-11-17 11:28:46 +08:00
e75e49a7b4 Remove logging dependecies in puppeteer-parse 2022-11-17 11:28:26 +08:00