Benature/omnivore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10,862 Commits 1 Branch 0 Tags
66a90d72772b0d2d90764569b9615d02108ae5dd
Commit Graph

47 Commits

Author SHA1 Message Date
Hongbo Wu
a924c8448b capture content-fetch success and error events 2024-05-13 14:55:48 +08:00
Hongbo Wu
d23bccf459 upgrade puppeteer-core to prevent ProtocolTimeout and adding more debug logs 2024-05-13 14:28:26 +08:00
Hongbo Wu
a03eee5ef7 fix dependecies 2024-01-18 18:48:46 +08:00
Hongbo Wu
d9feb740cb convert content-fetch to typescript 2024-01-18 18:48:46 +08:00
Hongbo Wu
cd3402b98a rewrite puppeteer in typescript 2024-01-18 18:48:46 +08:00
Hongbo Wu
51e586ed3d separate content-fetch in puppeteer packages from saving page content 2024-01-18 18:48:46 +08:00
Luke Channings
c4773dc904 Landing page improvements and various supporting improvements 2023-10-24 09:43:39 +01:00
Hongbo Wu
792cf0b207 upgrade google cloud sdk 2023-08-11 13:55:29 +08:00
Hongbo Wu
63cbb3011e upgrade puppeteer and update chromium args 2023-08-02 15:33:15 +08:00
Hongbo Wu
8dc4c48f87 Add createImageProxyUrl callback to the readability in puppeteer-parse service 2022-12-28 16:51:23 +08:00
Hongbo Wu
6a49689d1e Add readability in puppeteer-parse 2022-12-27 21:22:49 +08:00
Hongbo Wu
e75e49a7b4 Remove logging dependecies in puppeteer-parse 2022-11-17 11:28:26 +08:00
Hongbo Wu
b3e73cd84b Add puppeteer-extra plugins 2022-11-17 11:27:51 +08:00
Hongbo Wu
c8575a203b Fix tests 2022-11-16 10:16:09 +08:00
Hongbo Wu
b18af10e75 Import puppeteer-parse in content-fetch 2022-11-16 10:15:49 +08:00
Hongbo Wu
00fed8a0fb Remove content-fetch-gcf and create a Dockerfile for the cloud function 2022-11-16 10:15:49 +08:00
Hongbo Wu
cb858484c6 Make puppeteer-parse a module 2022-11-16 10:15:49 +08:00
Hongbo Wu
53d6afe25f Fix tests 2022-10-04 10:47:58 +08:00
Hongbo Wu
78bce2d1e7 Add Dockerfile for cloud run 2022-10-04 09:51:42 +08:00
Hongbo Wu
726df26c15 Temporarily copy content-handler to cloud functions root dir when doing cloud build 2022-10-03 15:39:05 +08:00
Hongbo Wu
a9607adfd3 Import content-handler as local dependency 2022-10-03 11:11:24 +08:00
Hongbo Wu
cb609d893e Escape HTML entities in puppeteer-parse 2022-09-23 16:40:32 +08:00
dependabot[bot]
d12f3642e6 Bump puppeteer-core from 15.3.2 to 16.1.0 
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 15.3.2 to 16.1.0.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v15.3.2...v16.1.0)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-08-11 02:15:16 +00:00
dependabot[bot]
b86d9a08b9 Bump @google-cloud/logging-winston from 4.2.2 to 5.1.1 
Bumps [@google-cloud/logging-winston](https://github.com/googleapis/nodejs-logging-winston) from 4.2.2 to 5.1.1.
- [Release notes](https://github.com/googleapis/nodejs-logging-winston/releases)
- [Changelog](https://github.com/googleapis/nodejs-logging-winston/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/nodejs-logging-winston/compare/v4.2.2...v5.1.1)

---
updated-dependencies:
- dependency-name: "@google-cloud/logging-winston"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-18 21:59:44 +00:00
Jackson Harper
33f35092ba Add puppeteer-core package 2022-07-15 11:19:44 -07:00
Jackson Harper
38b2b0c271 Remove puppeteer-core package in GCF 2022-07-15 11:01:15 -07:00
Hongbo Wu
1f1698ea81 sync changes to content-fetch-gcf 2022-07-15 15:11:41 +08:00
dependabot[bot]
bb7ea78e8f Bump puppeteer-core from 13.7.0 to 15.3.2 
Bumps [puppeteer-core](https://github.com/puppeteer/puppeteer) from 13.7.0 to 15.3.2.
- [Release notes](https://github.com/puppeteer/puppeteer/releases)
- [Changelog](https://github.com/puppeteer/puppeteer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/puppeteer/puppeteer/compare/v13.7.0...v15.3.2)

---
updated-dependencies:
- dependency-name: puppeteer-core
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-11 21:35:51 +00:00
dependabot[bot]
e91f25e58c Bump axios from 0.26.0 to 0.27.2 
Bumps [axios](https://github.com/axios/axios) from 0.26.0 to 0.27.2.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.27.2/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.26.0...v0.27.2)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-27 21:31:14 +00:00
Jackson Harper
0e31a40331 Use chrome-aws-lambda in the puppeteer GCF 2022-05-13 16:48:51 -07:00
Hongbo Wu
b766e17189 Remove jsdom in content-fetch 2022-05-12 16:48:59 +08:00
Hongbo Wu
9606cd6b28 Remove chrome-aws-lambda dependencies 2022-05-12 16:32:22 +08:00
Hongbo Wu
0984dca183 Remove adblocker and block resources by url and also block mathJax script 2022-05-11 22:04:47 +08:00
Hongbo Wu
1dffbfc881 Add puppeteer-code dependencies 2022-05-09 13:45:45 +08:00
Hongbo Wu
21799b7b6d Add puppeteer-stealth and puppeteer-ad-block plugin and a user-data-dir to reduce processing time 2022-05-09 13:45:45 +08:00
Jackson Harper
a4198e34dd Merge pull request #335 from omnivore-app/dependabot/npm_and_yarn/chrome-aws-lambda-10.1.0 
Bump chrome-aws-lambda from 7.0.0 to 10.1.0
 2022-04-04 15:36:06 -07:00
dependabot[bot]
3b4254a711 Bump luxon from 1.28.0 to 2.3.1 
Bumps [luxon](https://github.com/moment/luxon) from 1.28.0 to 2.3.1.
- [Release notes](https://github.com/moment/luxon/releases)
- [Changelog](https://github.com/moment/luxon/blob/master/CHANGELOG.md)
- [Commits](https://github.com/moment/luxon/compare/1.28.0...2.3.1)

---
updated-dependencies:
- dependency-name: luxon
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-04 17:55:59 +00:00
dependabot[bot]
f356ebd781 Bump chrome-aws-lambda from 7.0.0 to 10.1.0 
Bumps [chrome-aws-lambda](https://github.com/alixaxel/chrome-aws-lambda) from 7.0.0 to 10.1.0.
- [Release notes](https://github.com/alixaxel/chrome-aws-lambda/releases)
- [Commits](https://github.com/alixaxel/chrome-aws-lambda/compare/v7.0.0...v10.1.0)

---
updated-dependencies:
- dependency-name: chrome-aws-lambda
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-28 21:27:06 +00:00
Jackson Harper
5036924b21 Merge pull request #280 from omnivore-app/dependabot/npm_and_yarn/jsdom-19.0.0 
Bump jsdom from 16.7.0 to 19.0.0
 2022-03-25 09:08:55 -07:00
dependabot[bot]
6ac10c4780 Bump @google-cloud/functions-framework from 1.9.0 to 3.0.0 
Bumps [@google-cloud/functions-framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) from 1.9.0 to 3.0.0.
- [Release notes](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/releases)
- [Changelog](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/compare/v1.9.0...v3.0.0)

---
updated-dependencies:
- dependency-name: "@google-cloud/functions-framework"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-23 18:51:39 +00:00
dependabot[bot]
3ec6f22199 Bump jsdom from 16.7.0 to 19.0.0 
Bumps [jsdom](https://github.com/jsdom/jsdom) from 16.7.0 to 19.0.0.
- [Release notes](https://github.com/jsdom/jsdom/releases)
- [Changelog](https://github.com/jsdom/jsdom/blob/master/Changelog.md)
- [Commits](https://github.com/jsdom/jsdom/compare/16.7.0...19.0.0)

---
updated-dependencies:
- dependency-name: jsdom
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-23 18:51:08 +00:00
Jackson Harper
8ed16e0e8a Bump winston in the puppeteer service 2022-02-23 09:56:45 -08:00
dependabot[bot]
0f843809e3 Bump axios from 0.21.4 to 0.26.0 
Bumps [axios](https://github.com/axios/axios) from 0.21.4 to 0.26.0.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/master/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.21.4...v0.26.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-14 21:35:58 +00:00
Jackson Harper
42836b6b38 Simplify startup of the puppeteer service 
- Run on port 9090 so we don't conflict with other services
- Route the docker-compose requests to the host network
- Dont require preview bucket information on startup
 2022-02-11 14:44:32 -08:00
Jackson Harper
8163b5a4de Upgrade axios 2022-02-11 13:51:43 -08:00
Jackson Harper
ef2b53e7f7 Upgrade Cloud Storage and the apollo-express libraries 2022-02-11 13:21:03 -08:00
Jackson Harper
84f32935f5 Open source omnivore 2022-02-11 09:24:33 -08:00