Benature/omnivore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2,957 Commits 1 Branch 0 Tags
4f19d9f5ec05f77ecad9124b7865b9a807fb717c
Commit Graph

51 Commits

Author SHA1 Message Date
dependabot[bot]
b86d9a08b9 Bump @google-cloud/logging-winston from 4.2.2 to 5.1.1 
Bumps [@google-cloud/logging-winston](https://github.com/googleapis/nodejs-logging-winston) from 4.2.2 to 5.1.1.
- [Release notes](https://github.com/googleapis/nodejs-logging-winston/releases)
- [Changelog](https://github.com/googleapis/nodejs-logging-winston/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/nodejs-logging-winston/compare/v4.2.2...v5.1.1)

---
updated-dependencies:
- dependency-name: "@google-cloud/logging-winston"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-18 21:59:44 +00:00
Jackson Harper
f17ee64676 Use ScrapingBee for some hosts 2022-07-16 14:09:45 -07:00
Jackson Harper
2660262c69 Use puppeteer-core 2022-07-15 11:43:55 -07:00
Jackson Harper
33f35092ba Add puppeteer-core package 2022-07-15 11:19:44 -07:00
Jackson Harper
38b2b0c271 Remove puppeteer-core package in GCF 2022-07-15 11:01:15 -07:00
Jackson Harper
2447bd658e Use chrome-aws-lambda in GCF 2022-07-15 10:58:58 -07:00
Hongbo Wu
d404cd7c4c fix comment 2022-07-15 21:41:06 +08:00
Hongbo Wu
1f1698ea81 sync changes to content-fetch-gcf 2022-07-15 15:11:41 +08:00
dependabot[bot]
bb7ea78e8f Bump puppeteer-core from 13.7.0 to 15.3.2 
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 13.7.0 to 15.3.2.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v13.7.0...v15.3.2)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-11 21:35:51 +00:00
dependabot[bot]
e91f25e58c Bump axios from 0.26.0 to 0.27.2 
Bumps [axios](https://github.com/axios/axios) from 0.26.0 to 0.27.2.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.27.2/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.26.0...v0.27.2)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-27 21:31:14 +00:00
Hongbo Wu
0cc7e84a82 Fix content not getting parsed by linkedom properly without <html> tag by replacing innerHtml with outerHtml 2022-05-18 15:52:16 +08:00
Hongbo Wu
8f0447ed3f Stop blocking images and css file 2022-05-18 15:50:52 +08:00
Hongbo Wu
629aa54c58 Fix youtube handler 2022-05-18 11:28:33 +08:00
Hongbo Wu
ca662964e6 Fix not getting youtube video id from url 2022-05-17 21:51:03 +08:00
Jackson Harper
0e31a40331 Use chrome-aws-lambda in the puppeteer GCF 2022-05-13 16:48:51 -07:00
Jackson Harper
7bfb8cfee4 Merge pull request #597 from omnivore-app/remove-chrome-aws-lambda 
Optimize puppeteer and remove chrome-aws-lambda dependencies
 2022-05-13 16:12:24 -07:00
Hongbo Wu
6f09a4b31a Fix missing variable name in medium handler 2022-05-13 17:47:21 +08:00
Hongbo Wu
f5003c1370 Stop blocking script 2022-05-13 12:17:19 +08:00
Hongbo Wu
37e55add98 Stop blocking stylesheet and media 2022-05-13 12:09:05 +08:00
Hongbo Wu
ad99f933e5 Fix tests cont 2022-05-12 17:53:28 +08:00
Hongbo Wu
60bbbb6cf3 Block requests to 'font', 'image', 'stylesheet', 'script', 'media' in puppeteer 2022-05-12 17:10:38 +08:00
Hongbo Wu
b766e17189 Remove jsdom in content-fetch 2022-05-12 16:48:59 +08:00
Hongbo Wu
9606cd6b28 Remove chrome-aws-lambda dependencies 2022-05-12 16:32:22 +08:00
Hongbo Wu
0984dca183 Remove adblocker and block resources by url and also block mathJax script 2022-05-11 22:04:47 +08:00
Hongbo Wu
0b11c31317 Add linkedom dependency in packages/api 2022-05-10 18:31:25 +08:00
Hongbo Wu
7dd460352e Merge pull request #566 from omnivore-app/optimize-puppeteer 
Optimize saving performance
 2022-05-10 09:43:33 +08:00
Jackson Harper
1f0abbaa30 Add tests 2022-05-09 11:11:13 -07:00
Jackson Harper
31eb1a0f61 Fix parsing video IDs from youtube URLs 2022-05-09 11:03:17 -07:00
Hongbo Wu
1dffbfc881 Add puppeteer-code dependencies 2022-05-09 13:45:45 +08:00
Hongbo Wu
4c7f6d0281 Update comments 2022-05-09 13:45:45 +08:00
Hongbo Wu
4571f1f51c Add metrics 2022-05-09 13:45:45 +08:00
Hongbo Wu
21799b7b6d Add puppeteer-stealth and puppeteer-ad-block plugin and a user-data-dir to reduce processing time 2022-05-09 13:45:45 +08:00
Hongbo Wu
35e3e347d6 Remove image text 2022-05-05 19:25:09 +08:00
Hongbo Wu
c4ea185e17 Add comment 2022-05-05 12:16:53 +08:00
Hongbo Wu
6f29f18743 Parse image and save it in a <img> element 2022-05-05 12:13:08 +08:00
Jackson Harper
08e14c7577 Fix parsing of video IDs out of shared YouTube URLs 2022-05-04 10:45:16 -07:00
Hongbo Wu
b679451548 Fix parsing articles from www.derstandard.at (#459) 
* Fix parsing articles from www.derstandard.at

* slim cookies down
 2022-04-22 10:53:28 +08:00
Jackson Harper
a4198e34dd Merge pull request #335 from omnivore-app/dependabot/npm_and_yarn/chrome-aws-lambda-10.1.0 
Bump chrome-aws-lambda from 7.0.0 to 10.1.0
 2022-04-04 15:36:06 -07:00
dependabot[bot]
3b4254a711 Bump luxon from 1.28.0 to 2.3.1 
Bumps [luxon](https://github.com/moment/luxon) from 1.28.0 to 2.3.1.
- [Release notes](https://github.com/moment/luxon/releases)
- [Changelog](https://github.com/moment/luxon/blob/master/CHANGELOG.md)
- [Commits](https://github.com/moment/luxon/compare/1.28.0...2.3.1)

---
updated-dependencies:
- dependency-name: luxon
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-04 17:55:59 +00:00
dependabot[bot]
f356ebd781 Bump chrome-aws-lambda from 7.0.0 to 10.1.0 
Bumps [chrome-aws-lambda](https://github.com/alixaxel/chrome-aws-lambda) from 7.0.0 to 10.1.0.
- [Release notes](https://github.com/alixaxel/chrome-aws-lambda/releases)
- [Commits](https://github.com/alixaxel/chrome-aws-lambda/compare/v7.0.0...v10.1.0)

---
updated-dependencies:
- dependency-name: chrome-aws-lambda
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-28 21:27:06 +00:00
Jackson Harper
5036924b21 Merge pull request #280 from omnivore-app/dependabot/npm_and_yarn/jsdom-19.0.0 
Bump jsdom from 16.7.0 to 19.0.0
 2022-03-25 09:08:55 -07:00
dependabot[bot]
6ac10c4780 Bump @google-cloud/functions-framework from 1.9.0 to 3.0.0 
Bumps [@google-cloud/functions-framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) from 1.9.0 to 3.0.0.
- [Release notes](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/releases)
- [Changelog](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/compare/v1.9.0...v3.0.0)

---
updated-dependencies:
- dependency-name: "@google-cloud/functions-framework"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-23 18:51:39 +00:00
dependabot[bot]
3ec6f22199 Bump jsdom from 16.7.0 to 19.0.0 
Bumps [jsdom](https://github.com/jsdom/jsdom) from 16.7.0 to 19.0.0.
- [Release notes](https://github.com/jsdom/jsdom/releases)
- [Changelog](https://github.com/jsdom/jsdom/blob/master/Changelog.md)
- [Commits](https://github.com/jsdom/jsdom/compare/16.7.0...19.0.0)

---
updated-dependencies:
- dependency-name: jsdom
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-23 18:51:08 +00:00
Jackson Harper
8ed16e0e8a Bump winston in the puppeteer service 2022-02-23 09:56:45 -08:00
dependabot[bot]
0f843809e3 Bump axios from 0.21.4 to 0.26.0 
Bumps [axios](https://github.com/axios/axios) from 0.21.4 to 0.26.0.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/master/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.21.4...v0.26.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-14 21:35:58 +00:00
Jackson Harper
46b526961a Dockerize the puppeteer-parse service and add to docker-compose 2022-02-12 13:14:00 -08:00
Jackson Harper
ceb6794781 Add better getting started instructions for running puppeteer 2022-02-11 16:12:30 -08:00
Jackson Harper
42836b6b38 Simplify startup of the puppeteer service 
- Run on port 9090 so we don't conflict with other services
- Route the docker-compose requests to the host network
- Dont require preview bucket information on startup
 2022-02-11 14:44:32 -08:00
Jackson Harper
8163b5a4de Upgrade axios 2022-02-11 13:51:43 -08:00
Jackson Harper
ef2b53e7f7 Upgrade Cloud Storage and the apollo-express libraries 2022-02-11 13:21:03 -08:00