|
|
244fb4ccb5
|
fix: removing node with background image
|
2023-06-26 16:40:14 +08:00 |
|
|
|
43dcb8d5d7
|
fix: invalid url in puppeteer
|
2023-06-26 15:08:46 +08:00 |
|
|
|
d2e476a3ea
|
add debug logs
|
2023-06-14 12:53:43 +08:00 |
|
|
|
9fbe32b219
|
mark import failed on the last failed retry
|
2023-06-13 17:30:26 +08:00 |
|
|
|
4a438f45f5
|
add timeout and content length to the axios requests
|
2023-06-12 21:31:35 +08:00 |
|
|
|
a4449e6f93
|
better handling error in puppeteer
|
2023-06-12 21:05:54 +08:00 |
|
|
|
345c56c57a
|
fix: catch importer status update error
|
2023-06-01 16:11:35 +08:00 |
|
|
|
813a90e3c9
|
feat: send importer status update
|
2023-06-01 16:11:35 +08:00 |
|
|
|
67014e68fd
|
feat: send email notification once the import has finished
|
2023-06-01 16:11:35 +08:00 |
|
|
|
19facec4e2
|
feat: add importer metrics collector
|
2023-06-01 16:11:35 +08:00 |
|
|
|
d69e732579
|
fix: dedupe url if failed to save
|
2023-06-01 16:11:35 +08:00 |
|
|
|
8ddde61285
|
fix: increase twitter-handler timeout to 60 seconds
|
2023-06-01 16:11:35 +08:00 |
|
|
|
df7e1fd269
|
fix: catch scrapingbee api error
|
2023-06-01 16:11:35 +08:00 |
|
|
|
014dc773e3
|
Fix labels not saved correctly by saveUrl
|
2023-04-12 17:04:59 +08:00 |
|
|
|
5c46903fbe
|
Add labels and state to puppeteer-parse
|
2023-04-12 16:50:11 +08:00 |
|
|
|
9c96a4eab2
|
fix puppeteer throwing error when create_time is not defined
|
2023-03-15 13:15:21 +08:00 |
|
|
|
aae6759bcb
|
return published date if the class name is omnivore-published-date which we added when we scraped the article
|
2023-03-13 12:08:01 +08:00 |
|
|
|
1002e0baeb
|
improve regex for url parse
|
2023-03-07 13:51:46 +08:00 |
|
|
|
e39a7cc16b
|
improve urlParse for puppeteer-parse
|
2023-03-07 12:13:00 +08:00 |
|
|
|
69b87078a5
|
Fix not timeout if scroll more than 5 seconds
|
2023-02-13 22:10:56 +08:00 |
|
|
|
d816ee9563
|
Allow image and media to be loaded in puppeteer
|
2023-02-13 16:50:39 +08:00 |
|
|
|
145a8b9730
|
Fix last two tweets not saved in a long thread by waiting for document loaded after scrolling
|
2023-01-12 22:03:29 +08:00 |
|
|
|
7bfd36e308
|
Fix readability not run in puppeteer-parse
|
2023-01-12 18:33:47 +08:00 |
|
|
|
611a50e967
|
Merge pull request #1559 from omnivore-app/feat/importer
New upload file importer
|
2023-01-04 15:32:36 +08:00 |
|
|
|
ab16447c14
|
Fix not saving all the tweets in a thread
|
2023-01-03 21:26:34 +08:00 |
|
|
|
dc444a471d
|
Dont allow puppeteer to fail if empty tag is found
|
2023-01-03 12:42:06 +08:00 |
|
|
|
8dc4c48f87
|
Add createImageProxyUrl callback to the readability in puppeteer-parse service
|
2022-12-28 16:51:23 +08:00 |
|
|
|
a5f5e6fbdb
|
Fix docker build
|
2022-12-28 11:51:11 +08:00 |
|
|
|
7c39db207b
|
Replace createArticle with savePage in puppeteer-parse service
|
2022-12-28 10:15:05 +08:00 |
|
|
|
6a49689d1e
|
Add readability in puppeteer-parse
|
2022-12-27 21:22:49 +08:00 |
|
|
|
44e4e080dd
|
increase navigation timeout to 30 seconds
|
2022-11-17 11:28:46 +08:00 |
|
|
|
c18217e697
|
Fix logRecord being shared by requests
|
2022-11-17 11:28:46 +08:00 |
|
|
|
d9237723c5
|
Share puppeteer browser instance between content handlers
|
2022-11-17 11:28:46 +08:00 |
|
|
|
e866541ae1
|
Fix puppeteer launch in head mode
|
2022-11-17 11:28:46 +08:00 |
|
|
|
e75e49a7b4
|
Remove logging dependecies in puppeteer-parse
|
2022-11-17 11:28:26 +08:00 |
|
|
|
b3e73cd84b
|
Add puppeteer-extra plugins
|
2022-11-17 11:27:51 +08:00 |
|
|
|
8bdbd86e97
|
Make headless = false in local testing
|
2022-11-17 11:27:51 +08:00 |
|
|
|
f39e65c159
|
Fix rebase error
|
2022-11-16 10:16:09 +08:00 |
|
|
|
c8575a203b
|
Fix tests
|
2022-11-16 10:16:09 +08:00 |
|
|
|
177c69499b
|
Replace logger.log with logger.info
|
2022-11-16 10:16:07 +08:00 |
|
|
|
b09fb4c037
|
Fix docker build
|
2022-11-16 10:15:49 +08:00 |
|
|
|
b18af10e75
|
Import puppeteer-parse in content-fetch
|
2022-11-16 10:15:49 +08:00 |
|
|
|
00fed8a0fb
|
Remove content-fetch-gcf and create a Dockerfile for the cloud function
|
2022-11-16 10:15:49 +08:00 |
|
|
|
cb858484c6
|
Make puppeteer-parse a module
|
2022-11-16 10:15:49 +08:00 |
|
|
|
2bad70fda9
|
Catch puppeteer navigation timeout error
|
2022-10-25 16:59:48 +08:00 |
|
|
|
d6e465d482
|
Add Dockerfile for pdfHandler
|
2022-10-04 15:28:12 +08:00 |
|
|
|
53d6afe25f
|
Fix tests
|
2022-10-04 10:47:58 +08:00 |
|
|
|
9cae703666
|
Fix Dockerfile
|
2022-10-04 10:20:13 +08:00 |
|
|
|
78bce2d1e7
|
Add Dockerfile for cloud run
|
2022-10-04 09:51:42 +08:00 |
|
|
|
726df26c15
|
Temporarily copy content-handler to cloud functions root dir when doing cloud build
|
2022-10-03 15:39:05 +08:00 |
|
