113 Commits

Author SHA1 Message Date
4e582fb55d Improving Self-Hosting and Removing 3rd Party dependencies. (#4513)
* fix: Library Header layout shift

* Bump Github Actions versions.

* Self-Hosting Changes

* Fix Minio Environment Variable

* Just make pdfs successful, due to lack of PDFHandler

* Fix issue where flag was set wrong

* Added an NGINX Example file

* Add some documentation for self-hosting via Docker Compose

* Make some adjustments to Puppeteer due to failing sites.

* adjust timings

* Add start of Mail Service

* Fix Docker Files

* More email service stuff

* Add Guide to use Zapier for Email-Importing.

* Ensure that if no env is provided it uses the old email settings

* Add some instructions for self-hosted email

* Add SNS Endpoints for Mail Watcher

* Add steps and functionality for using SES and SNS for email

* Uncomment a few jobs.

* Added option for Firefox for parser. Was having issues with Chromium on Docker.

* Add missing space.

Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>

* Fix some wording on the Guide

* update browser extension to handle self-hosted instances

* add slight documentation to options page

* Fix MV

* Do raw handlers for Medium

* Fix images in Medium

* Update self-hosting/GUIDE.md

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update Guide with other variables

* Add The Verge to JS-less handlers

* Update regex and image-proxy

* Update self-hosting/nginx/nginx.conf

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update regex and image-proxy

* Update self-hosting/docker-compose/docker-compose.yml

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Fix Minio for Export

* Merge to main

* Update GUIDE with newer NGINX

* Update nginx config to include api/save route

* Enable Native PDF View for PDFS

* Enable Native PDF View for PDFS

* feat:lover packages test

* feat:working build

* feat:alpine build

* docs:api dockerfile docs

* Write a PDF.js wrapper to replace pspdfkit

* Revert changes for replication, set settings to have default mode

* build folder got removed due to gitignore on pdf

* Add Box shadow to pdf pages

* Add Toggle for Progress in PDFS, enabled native viewer toggle

* Update node version to LTS

* Update node version to LTS

* Fix Linting issues

* Fix Linting issues

* Make env variable nullable

* Add touchend listener for mobile

* Make changes to PDF for mobile

* fix(android): change serverUrl to selfhosted first

* feat:2 stage alpine content fetch

* feat:separated start script

* fix:changed to node 22

* Add back youtube functionality and add guide

* trigger build

* Fix cache issue on YouTube

* Allow empty AWS_S3_ENDPOINT

* Allow empty AWS_S3_ENDPOINT

* Add GCHR for all images

* Add GCHR For self hosting.

* Add GCHR For self hosting.

* Test prebuilt.

* Test prebuilt

* Test prebuilt...

* Fix web image

* Remove Web Image (For now)

* Move docker-compose to images

* Move docker-compose files to correct locations

* Remove the need for ARGS

* Update packages, and Typescript versions

* Fix

* Fix issues with build on Web

* Correct push

* Fix Linting issues

* Fix Trace import

* Add missing types

* Fix Tasks

* Add information into guide about self-build

* Fix issues with PDF Viewer

---------

Co-authored-by: keumky2 <keumky2@woowahan.com>
Co-authored-by: William Theaker <wtheaker@nvidia.com>
Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>
Co-authored-by: David Adams <david@dadams2.com>
Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>
Co-authored-by: m1xxos <66390094+m1xxos@users.noreply.github.com>
Co-authored-by: Adil <mr.adil777@gmail.com>
2025-01-27 13:33:16 +01:00
70d6e9dcae reduce logs 2024-08-29 18:18:17 +08:00
0366c426bc process up to 2 jobs concurrently 2024-08-21 17:58:58 +08:00
03ff18af81 remove unused dependencies 2024-08-21 14:09:49 +08:00
0fbc6d0a87 update importer to use content-fetch queue 2024-08-21 12:24:35 +08:00
49556453e9 fix importer dependency 2024-08-13 17:38:31 +08:00
3fe22e0f8e Merge pull request #4215 from omnivore-app/feature/exporter
create a cloud run service for the exporter
2024-08-13 17:20:44 +08:00
110f537e72 sleep for 1 second to avoid rate limiting 2024-07-22 18:36:11 +08:00
95657e3d3e queue email sending jobs in exporter and importer 2024-07-21 11:11:29 +08:00
01bce43781 create a cloud run service for the exporter 2024-07-19 17:12:40 +08:00
56a97e5711 Bump @fast-csv/parse from 4.3.6 to 5.0.0
Bumps [@fast-csv/parse](https://github.com/C2FO/fast-csv/tree/HEAD/packages/parse) from 4.3.6 to 5.0.0.
- [Release notes](https://github.com/C2FO/fast-csv/releases)
- [Changelog](https://github.com/C2FO/fast-csv/blob/main/packages/parse/CHANGELOG.md)
- [Commits](https://github.com/C2FO/fast-csv/commits/v5.0.0/packages/parse)

---
updated-dependencies:
- dependency-name: "@fast-csv/parse"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-08 21:53:29 +00:00
649463b785 only disconnect redis message queue client if it is different from cache client 2024-07-04 16:09:32 +08:00
0a1b7345e9 add timeout 2024-07-04 12:56:28 +08:00
bbc7b5e600 use @omnivore/utils in import-handler 2024-07-03 22:20:27 +08:00
88a7e8d85b fix tests 2024-04-04 12:17:15 +08:00
c356183acd Remove typescript and eslint changes 2024-03-12 11:01:44 +08:00
7d291f084b Merge all changes from main, update theming of Discover 2024-03-07 17:39:57 +01:00
628cfc64d1 fix: replace \n with new line in redis cert 2024-01-22 11:09:15 +08:00
8a91a3c629 fix importer tests 2024-01-18 18:48:46 +08:00
150a456c35 replace redis client library with ioredis 2024-01-16 15:42:50 +08:00
c0373646cb Fix Linting... 2024-01-09 11:22:32 +01:00
32ae38fcba upgrade sentry to 7.77.0 in packages/api 2023-11-08 12:23:36 +08:00
dd1cb188e4 fix tests 2023-10-30 14:41:53 +08:00
47f67c237d change source to pocket if item is imported from pocket 2023-10-30 14:41:52 +08:00
c4773dc904 Landing page improvements and various supporting improvements 2023-10-24 09:43:39 +01:00
9ee87e2e52 convert type of date in the csv to int before parsing date 2023-08-15 18:44:30 +08:00
e3654e8ae2 fix test 2023-08-14 16:38:24 +08:00
d25f19207b add savedAt and publishedAt to the csv file 2023-08-14 13:49:06 +08:00
ba2d34c1af validate csv file in the importer 2023-08-14 13:38:36 +08:00
792cf0b207 upgrade google cloud sdk 2023-08-11 13:55:29 +08:00
fa1ff9ba17 Upgrade node to 18.16 2023-07-24 15:55:11 +08:00
fc7a649566 limit import to 20k urls 2023-07-17 18:09:13 +08:00
af7350bd31 fix: importer status update email not sent
* create omnivore-authorization header to contain the token
2023-06-28 11:00:40 +08:00
8ad937ed84 allow more labels format in the csv 2023-06-14 13:33:20 +08:00
42305ee0b4 use a separate queue for sending emails 2023-06-14 13:17:59 +08:00
dddd763ba2 fix graphql error by setting clientRequestId empty 2023-06-13 21:15:34 +08:00
27ff2856e2 remove client request id in import task which caused the pdf upload failure 2023-06-13 18:21:36 +08:00
4a438f45f5 add timeout and content length to the axios requests 2023-06-12 21:31:35 +08:00
63bddf13a1 set default metrics to zero 2023-06-05 18:16:41 +08:00
4cad18f51c fix taskId not being set in the cloud task payload 2023-06-05 17:52:36 +08:00
992a78e173 add dockerfile 2023-06-01 16:29:30 +08:00
8a89a47c16 fix: testcases 2023-06-01 16:11:35 +08:00
1073f88d69 fix: add start:collector command 2023-06-01 16:11:35 +08:00
813a90e3c9 feat: send importer status update 2023-06-01 16:11:35 +08:00
fba8e0424d feat: add importer metrics collector 2023-06-01 16:11:35 +08:00
5f465112bf feat: update matter importer metrics 2023-06-01 16:11:35 +08:00
67014e68fd feat: send email notification once the import has finished 2023-06-01 16:11:35 +08:00
19facec4e2 feat: add importer metrics collector 2023-06-01 16:11:35 +08:00
c64476b44a Add test for label parsing 2023-05-24 18:17:29 +08:00
9218e16415 Add test file 2023-05-24 18:10:30 +08:00