Benature/omnivore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4,272 Commits 1 Branch 0 Tags
b3e73cd84be0cc5eb2447bd3cfab20e7a11861c0
Commit Graph

72 Commits

Author SHA1 Message Date
Hongbo Wu
b3e73cd84b Add puppeteer-extra plugins 2022-11-17 11:27:51 +08:00
Hongbo Wu
8bdbd86e97 Make headless = false in local testing 2022-11-17 11:27:51 +08:00
Hongbo Wu
f39e65c159 Fix rebase error 2022-11-16 10:16:09 +08:00
Hongbo Wu
c8575a203b Fix tests 2022-11-16 10:16:09 +08:00
Hongbo Wu
177c69499b Replace logger.log with logger.info 2022-11-16 10:16:07 +08:00
Hongbo Wu
b09fb4c037 Fix docker build 2022-11-16 10:15:49 +08:00
Hongbo Wu
b18af10e75 Import puppeteer-parse in content-fetch 2022-11-16 10:15:49 +08:00
Hongbo Wu
00fed8a0fb Remove content-fetch-gcf and create a Dockerfile for the cloud function 2022-11-16 10:15:49 +08:00
Hongbo Wu
cb858484c6 Make puppeteer-parse a module 2022-11-16 10:15:49 +08:00
Hongbo Wu
2bad70fda9 Catch puppeteer navigation timeout error 2022-10-25 16:59:48 +08:00
Hongbo Wu
d6e465d482 Add Dockerfile for pdfHandler 2022-10-04 15:28:12 +08:00
Hongbo Wu
53d6afe25f Fix tests 2022-10-04 10:47:58 +08:00
Hongbo Wu
9cae703666 Fix Dockerfile 2022-10-04 10:20:13 +08:00
Hongbo Wu
78bce2d1e7 Add Dockerfile for cloud run 2022-10-04 09:51:42 +08:00
Hongbo Wu
726df26c15 Temporarily copy content-handler to cloud functions root dir when doing cloud build 2022-10-03 15:39:05 +08:00
Hongbo Wu
cae2715a52 Remove content-handler 2022-10-03 11:15:09 +08:00
Hongbo Wu
a9607adfd3 Import content-handler as local dependency 2022-10-03 11:11:24 +08:00
Hongbo Wu
99956539a0 Handle newsletter in content-handlers 2022-09-30 12:51:22 +08:00
Hongbo Wu
206d795c54 Import content-handler in puppeteer 2022-09-30 12:51:22 +08:00
Hongbo Wu
cb609d893e Escape HTML entities in puppeteer-parse 2022-09-23 16:40:32 +08:00
dependabot[bot]
d12f3642e6 Bump puppeteer-core from 15.3.2 to 16.1.0 
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 15.3.2 to 16.1.0.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v15.3.2...v16.1.0)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-08-11 02:15:16 +00:00
dependabot[bot]
b86d9a08b9 Bump @google-cloud/logging-winston from 4.2.2 to 5.1.1 
Bumps [@google-cloud/logging-winston](https://github.com/googleapis/nodejs-logging-winston) from 4.2.2 to 5.1.1.
- [Release notes](https://github.com/googleapis/nodejs-logging-winston/releases)
- [Changelog](https://github.com/googleapis/nodejs-logging-winston/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/nodejs-logging-winston/compare/v4.2.2...v5.1.1)

---
updated-dependencies:
- dependency-name: "@google-cloud/logging-winston"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-18 21:59:44 +00:00
Jackson Harper
f17ee64676 Use ScrapingBee for some hosts 2022-07-16 14:09:45 -07:00
Jackson Harper
2660262c69 Use puppeteer-core 2022-07-15 11:43:55 -07:00
Jackson Harper
33f35092ba Add puppeteer-core package 2022-07-15 11:19:44 -07:00
Jackson Harper
38b2b0c271 Remove puppeteer-core package in GCF 2022-07-15 11:01:15 -07:00
Jackson Harper
2447bd658e Use chrome-aws-lambda in GCF 2022-07-15 10:58:58 -07:00
Hongbo Wu
d404cd7c4c fix comment 2022-07-15 21:41:06 +08:00
Hongbo Wu
1f1698ea81 sync changes to content-fetch-gcf 2022-07-15 15:11:41 +08:00
dependabot[bot]
bb7ea78e8f Bump puppeteer-core from 13.7.0 to 15.3.2 
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 13.7.0 to 15.3.2.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v13.7.0...v15.3.2)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-11 21:35:51 +00:00
dependabot[bot]
e91f25e58c Bump axios from 0.26.0 to 0.27.2 
Bumps [axios](https://github.com/axios/axios) from 0.26.0 to 0.27.2.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.27.2/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.26.0...v0.27.2)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-27 21:31:14 +00:00
Hongbo Wu
0cc7e84a82 Fix content not getting parsed by linkedom properly without <html> tag by replacing innerHtml with outerHtml 2022-05-18 15:52:16 +08:00
Hongbo Wu
8f0447ed3f Stop blocking images and css file 2022-05-18 15:50:52 +08:00
Hongbo Wu
629aa54c58 Fix youtube handler 2022-05-18 11:28:33 +08:00
Hongbo Wu
ca662964e6 Fix not getting youtube video id from url 2022-05-17 21:51:03 +08:00
Jackson Harper
0e31a40331 Use chrome-aws-lambda in the puppeteer GCF 2022-05-13 16:48:51 -07:00
Jackson Harper
7bfb8cfee4 Merge pull request #597 from omnivore-app/remove-chrome-aws-lambda 
Optimize puppeteer and remove chrome-aws-lambda dependencies
 2022-05-13 16:12:24 -07:00
Hongbo Wu
6f09a4b31a Fix missing variable name in medium handler 2022-05-13 17:47:21 +08:00
Hongbo Wu
f5003c1370 Stop blocking script 2022-05-13 12:17:19 +08:00
Hongbo Wu
37e55add98 Stop blocking stylesheet and media 2022-05-13 12:09:05 +08:00
Hongbo Wu
ad99f933e5 Fix tests cont 2022-05-12 17:53:28 +08:00
Hongbo Wu
60bbbb6cf3 Block requests to 'font', 'image', 'stylesheet', 'script', 'media' in puppeteer 2022-05-12 17:10:38 +08:00
Hongbo Wu
b766e17189 Remove jsdom in content-fetch 2022-05-12 16:48:59 +08:00
Hongbo Wu
9606cd6b28 Remove chrome-aws-lambda dependencies 2022-05-12 16:32:22 +08:00
Hongbo Wu
0984dca183 Remove adblocker and block resources by url and also block mathJax script 2022-05-11 22:04:47 +08:00
Hongbo Wu
0b11c31317 Add linkedom dependency in packages/api 2022-05-10 18:31:25 +08:00
Hongbo Wu
7dd460352e Merge pull request #566 from omnivore-app/optimize-puppeteer 
Optimize saving performance
 2022-05-10 09:43:33 +08:00
Jackson Harper
1f0abbaa30 Add tests 2022-05-09 11:11:13 -07:00
Jackson Harper
31eb1a0f61 Fix parsing video IDs from youtube URLs 2022-05-09 11:03:17 -07:00
Hongbo Wu
1dffbfc881 Add puppeteer-code dependencies 2022-05-09 13:45:45 +08:00