Benature/omnivore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,170 Commits 1 Branch 0 Tags
c63ecc94fbce88596ac0f3fe04df869bccd7e99e
Commit Graph

59 Commits

Author SHA1 Message Date
Hongbo Wu
c03d59c3c9 Add test case for failures 2023-04-17 17:20:14 +08:00
Hongbo Wu
a1e782e39d Fix tests 2023-04-17 17:14:40 +08:00
Hongbo Wu
11538347af Fix incorrect successful and failed counter of imported urls in the importer 2023-04-17 10:24:46 +08:00
Hongbo Wu
87d3f5b3d8 Add labels and state to savePage API 2023-04-12 16:50:11 +08:00
Hongbo Wu
fcf08fc364 Add state and labels to savePageInput 2023-04-12 16:50:11 +08:00
Hongbo Wu
52399b3702 Add state and labels to the cloud task 2023-04-12 16:50:11 +08:00
Hongbo Wu
4b578bbebb Process labels and state in the csv file 2023-04-12 16:50:11 +08:00
Jackson Harper
46e371bc26 Remove more debug 2023-01-24 20:36:34 +08:00
Jackson Harper
1d81398bbd Improve logging on errors in the importer tool 2023-01-24 20:03:53 +08:00
Jackson Harper
cba676c9de Build fixes in import-handler 2023-01-24 14:55:00 +08:00
Jackson Harper
b7abb5fa91 Use glob to find _matter_history file in archives 
It looks like these have another directory within the archive, so
we have to search for the _matter_history file to find the full
archive content.
 2023-01-24 14:14:04 +08:00
Jackson Harper
d2e7f8e58c Add .dockerignore for the import handler 2023-01-13 09:19:15 +08:00
Jackson Harper
6845696f23 Handle exceptions from JSON.parse, in this case just return ok so the request isnt retried 2023-01-12 16:10:25 +08:00
Jackson Harper
6f588e2170 Create read stream after all early return checks 2023-01-12 15:48:54 +08:00
Jackson Harper
444d4a69eb i guess this was my first rodeo 2023-01-12 15:37:50 +08:00
Jackson Harper
c240f1d2c8 Make SavePage API call if we have content provided 2023-01-12 10:50:06 +08:00
Jackson Harper
7263e91d6b Enable ts-node files to pull in readability.d.ts 2023-01-11 23:59:27 +08:00
Jackson Harper
ee8bad424b Fix typo 2023-01-11 23:19:59 +08:00
Jackson Harper
fa866f1d29 Make function more clear 2023-01-11 23:08:30 +08:00
Jackson Harper
ee50b237c6 Fix order of args 2023-01-11 22:56:10 +08:00
Jackson Harper
0bc87deba0 Change order of ctx arguments 2023-01-11 22:55:04 +08:00
Jackson Harper
04375d6856 Add some extra debugging 2023-01-11 22:42:37 +08:00
Jackson Harper
6623d5a43a Handle pubsub event parsing 2023-01-11 20:59:41 +08:00
Jackson Harper
62db9da121 Parse pubsub message 2023-01-11 20:07:54 +08:00
Jackson Harper
154109bb9d Log headers 2023-01-11 19:46:12 +08:00
Jackson Harper
50e1dbda5a Add debug for Eventarc 2023-01-11 18:36:37 +08:00
Jackson Harper
c75a558bd4 Fix entrypoint in Dockerfile 2023-01-11 18:24:55 +08:00
Jackson Harper
dea4ffae33 Fixes for dockerizing the import-handler service 2023-01-11 18:10:22 +08:00
Jackson Harper
b6f9a8bab5 Add uuid package to import-handler 2023-01-11 17:06:04 +08:00
Jackson Harper
e31a806fcb Start http server to run in Cloud Run 2023-01-11 16:57:00 +08:00
Jackson Harper
6aabd0311f Fix scripts in import-handler package 2023-01-11 16:30:33 +08:00
Jackson Harper
9e83f0f11c Packaging for the import handler 2023-01-11 16:22:31 +08:00
Jackson Harper
903d14ead3 Add Dockerfile 2023-01-11 15:50:10 +08:00
Jackson Harper
e392af4800 Update matter import handler to use archives instead of just history files 2023-01-11 15:35:05 +08:00
Jackson Harper
592c7150a7 Handle case where import has a single entry 2023-01-06 12:49:29 +08:00
Jackson Harper
88b6fcfb5d Use cookie auth instead of header as GCP uses the Authorization header 2023-01-05 20:17:43 +08:00
Jackson Harper
0580f7a876 Fix the email endpoint for the import-handler 2023-01-05 17:58:26 +08:00
Jackson Harper
8c89a192fa Use 8.5.1 until we figure out issues with 9.0.0 2023-01-05 15:12:56 +08:00
Jackson Harper
687d126bd3 Fix the task URLs, send an email if no URLs are parsed for import 2023-01-05 11:49:16 +08:00
Jackson Harper
750fdd6b45 Remove unused packages 2023-01-05 11:48:54 +08:00
Jackson Harper
f177a0d1db Remove unused type 2023-01-05 11:14:57 +08:00
Jackson Harper
37dc0336fa Set Authorization to authToken 2023-01-05 11:13:58 +08:00
Jackson Harper
dd9d00ec8d Linting fixes, wait on email send 2023-01-04 18:30:28 +08:00
Jackson Harper
d89a307aca Send email to user after a bulk import completes 2023-01-04 18:15:26 +08:00
Jackson Harper
540581146f Linting fixes 2023-01-04 15:09:04 +08:00
Jackson Harper
fe993f1f41 Use Cloud Task instead of pubsub for intra service communication 2023-01-04 14:48:03 +08:00
Jackson Harper
97981f9a7b Updates to handle new import file format 2023-01-03 20:55:53 +08:00
Jackson Harper
9213d222c4 Add the _matter_history importer 2023-01-03 18:13:59 +08:00
Hongbo Wu
c8575a203b Fix tests 2022-11-16 10:16:09 +08:00
dependabot[bot]
d1b013d826 Bump @google-cloud/functions-framework from 3.1.0 to 3.1.2 
Bumps [@google-cloud/functions-framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) from 3.1.0 to 3.1.2.
- [Release notes](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/releases)
- [Changelog](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/compare/v3.1.0...v3.1.2)

---
updated-dependencies:
- dependency-name: "@google-cloud/functions-framework"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-25 21:32:05 +00:00