Benature/omnivore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5,700 Commits 1 Branch 0 Tags
9c96a4eab202869b166be510dc2162b57fe9fe04
Commit Graph

52 Commits

Author SHA1 Message Date
Jackson Harper
46e371bc26 Remove more debug 2023-01-24 20:36:34 +08:00
Jackson Harper
1d81398bbd Improve logging on errors in the importer tool 2023-01-24 20:03:53 +08:00
Jackson Harper
cba676c9de Build fixes in import-handler 2023-01-24 14:55:00 +08:00
Jackson Harper
b7abb5fa91 Use glob to find _matter_history file in archives 
It looks like these have another directory within the archive, so
we have to search for the _matter_history file to find the full
archive content.
 2023-01-24 14:14:04 +08:00
Jackson Harper
d2e7f8e58c Add .dockerignore for the import handler 2023-01-13 09:19:15 +08:00
Jackson Harper
6845696f23 Handle exceptions from JSON.parse, in this case just return ok so the request isnt retried 2023-01-12 16:10:25 +08:00
Jackson Harper
6f588e2170 Create read stream after all early return checks 2023-01-12 15:48:54 +08:00
Jackson Harper
444d4a69eb i guess this was my first rodeo 2023-01-12 15:37:50 +08:00
Jackson Harper
c240f1d2c8 Make SavePage API call if we have content provided 2023-01-12 10:50:06 +08:00
Jackson Harper
7263e91d6b Enable ts-node files to pull in readability.d.ts 2023-01-11 23:59:27 +08:00
Jackson Harper
ee8bad424b Fix typo 2023-01-11 23:19:59 +08:00
Jackson Harper
fa866f1d29 Make function more clear 2023-01-11 23:08:30 +08:00
Jackson Harper
ee50b237c6 Fix order of args 2023-01-11 22:56:10 +08:00
Jackson Harper
0bc87deba0 Change order of ctx arguments 2023-01-11 22:55:04 +08:00
Jackson Harper
04375d6856 Add some extra debugging 2023-01-11 22:42:37 +08:00
Jackson Harper
6623d5a43a Handle pubsub event parsing 2023-01-11 20:59:41 +08:00
Jackson Harper
62db9da121 Parse pubsub message 2023-01-11 20:07:54 +08:00
Jackson Harper
154109bb9d Log headers 2023-01-11 19:46:12 +08:00
Jackson Harper
50e1dbda5a Add debug for Eventarc 2023-01-11 18:36:37 +08:00
Jackson Harper
c75a558bd4 Fix entrypoint in Dockerfile 2023-01-11 18:24:55 +08:00
Jackson Harper
dea4ffae33 Fixes for dockerizing the import-handler service 2023-01-11 18:10:22 +08:00
Jackson Harper
b6f9a8bab5 Add uuid package to import-handler 2023-01-11 17:06:04 +08:00
Jackson Harper
e31a806fcb Start http server to run in Cloud Run 2023-01-11 16:57:00 +08:00
Jackson Harper
6aabd0311f Fix scripts in import-handler package 2023-01-11 16:30:33 +08:00
Jackson Harper
9e83f0f11c Packaging for the import handler 2023-01-11 16:22:31 +08:00
Jackson Harper
903d14ead3 Add Dockerfile 2023-01-11 15:50:10 +08:00
Jackson Harper
e392af4800 Update matter import handler to use archives instead of just history files 2023-01-11 15:35:05 +08:00
Jackson Harper
592c7150a7 Handle case where import has a single entry 2023-01-06 12:49:29 +08:00
Jackson Harper
88b6fcfb5d Use cookie auth instead of header as GCP uses the Authorization header 2023-01-05 20:17:43 +08:00
Jackson Harper
0580f7a876 Fix the email endpoint for the import-handler 2023-01-05 17:58:26 +08:00
Jackson Harper
8c89a192fa Use 8.5.1 until we figure out issues with 9.0.0 2023-01-05 15:12:56 +08:00
Jackson Harper
687d126bd3 Fix the task URLs, send an email if no URLs are parsed for import 2023-01-05 11:49:16 +08:00
Jackson Harper
750fdd6b45 Remove unused packages 2023-01-05 11:48:54 +08:00
Jackson Harper
f177a0d1db Remove unused type 2023-01-05 11:14:57 +08:00
Jackson Harper
37dc0336fa Set Authorization to authToken 2023-01-05 11:13:58 +08:00
Jackson Harper
dd9d00ec8d Linting fixes, wait on email send 2023-01-04 18:30:28 +08:00
Jackson Harper
d89a307aca Send email to user after a bulk import completes 2023-01-04 18:15:26 +08:00
Jackson Harper
540581146f Linting fixes 2023-01-04 15:09:04 +08:00
Jackson Harper
fe993f1f41 Use Cloud Task instead of pubsub for intra service communication 2023-01-04 14:48:03 +08:00
Jackson Harper
97981f9a7b Updates to handle new import file format 2023-01-03 20:55:53 +08:00
Jackson Harper
9213d222c4 Add the _matter_history importer 2023-01-03 18:13:59 +08:00
Hongbo Wu
c8575a203b Fix tests 2022-11-16 10:16:09 +08:00
dependabot[bot]
d1b013d826 Bump @google-cloud/functions-framework from 3.1.0 to 3.1.2 
Bumps [@google-cloud/functions-framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) from 3.1.0 to 3.1.2.
- [Release notes](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/releases)
- [Changelog](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/compare/v3.1.0...v3.1.2)

---
updated-dependencies:
- dependency-name: "@google-cloud/functions-framework"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-25 21:32:05 +00:00
dependabot[bot]
e91f25e58c Bump axios from 0.26.0 to 0.27.2 
Bumps [axios](https://github.com/axios/axios) from 0.26.0 to 0.27.2.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v0.27.2/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.26.0...v0.27.2)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-27 21:31:14 +00:00
dependabot[bot]
452a776aef Bump @google-cloud/functions-framework from 3.0.0 to 3.1.0 
Bumps [@google-cloud/functions-framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) from 3.0.0 to 3.1.0.
- [Release notes](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/releases)
- [Changelog](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/compare/v3.0.0...v3.1.0)

---
updated-dependencies:
- dependency-name: "@google-cloud/functions-framework"
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-04-04 22:36:00 +00:00
dependabot[bot]
6ac10c4780 Bump @google-cloud/functions-framework from 1.9.0 to 3.0.0 
Bumps [@google-cloud/functions-framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) from 1.9.0 to 3.0.0.
- [Release notes](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/releases)
- [Changelog](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/compare/v1.9.0...v3.0.0)

---
updated-dependencies:
- dependency-name: "@google-cloud/functions-framework"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-23 18:51:39 +00:00
dependabot[bot]
ce23281a84 Bump concurrently from 6.2.1 to 7.0.0 
Bumps [concurrently](https://github.com/open-cli-tools/concurrently) from 6.2.1 to 7.0.0.
- [Release notes](https://github.com/open-cli-tools/concurrently/releases)
- [Commits](https://github.com/open-cli-tools/concurrently/compare/v6.2.1...v7.0.0)

---
updated-dependencies:
- dependency-name: concurrently
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-03-07 21:21:07 +00:00
Jackson Harper
f30c6f21c3 Update segment and nodemon to bump update-notifier 2022-03-01 12:20:13 -08:00
dependabot[bot]
0f843809e3 Bump axios from 0.21.4 to 0.26.0 
Bumps [axios](https://github.com/axios/axios) from 0.21.4 to 0.26.0.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/master/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v0.21.4...v0.26.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-14 21:35:58 +00:00
Jackson Harper
8163b5a4de Upgrade axios 2022-02-11 13:51:43 -08:00