Commit Graph

193 Commits

Author SHA1 Message Date
4e582fb55d Improving Self-Hosting and Removing 3rd Party dependencies. (#4513)
* fix: Library Header layout shift

* Bump Github Actions versions.

* Self-Hosting Changes

* Fix Minio Environment Variable

* Just make pdfs successful, due to lack of PDFHandler

* Fix issue where flag was set wrong

* Added an NGINX Example file

* Add some documentation for self-hosting via Docker Compose

* Make some adjustments to Puppeteer due to failing sites.

* adjust timings

* Add start of Mail Service

* Fix Docker Files

* More email service stuff

* Add Guide to use Zapier for Email-Importing.

* Ensure that if no env is provided it uses the old email settings

* Add some instructions for self-hosted email

* Add SNS Endpoints for Mail Watcher

* Add steps and functionality for using SES and SNS for email

* Uncomment a few jobs.

* Added option for Firefox for parser. Was having issues with Chromium on Docker.

* Add missing space.

Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>

* Fix some wording on the Guide

* update browser extension to handle self-hosted instances

* add slight documentation to options page

* Fix MV

* Do raw handlers for Medium

* Fix images in Medium

* Update self-hosting/GUIDE.md

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update Guide with other variables

* Add The Verge to JS-less handlers

* Update regex and image-proxy

* Update self-hosting/nginx/nginx.conf

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Update regex and image-proxy

* Update self-hosting/docker-compose/docker-compose.yml

Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>

* Fix Minio for Export

* Merge to main

* Update GUIDE with newer NGINX

* Update nginx config to include api/save route

* Enable Native PDF View for PDFS

* Enable Native PDF View for PDFS

* feat:lover packages test

* feat:working build

* feat:alpine build

* docs:api dockerfile docs

* Write a PDF.js wrapper to replace pspdfkit

* Revert changes for replication, set settings to have default mode

* build folder got removed due to gitignore on pdf

* Add Box shadow to pdf pages

* Add Toggle for Progress in PDFS, enabled native viewer toggle

* Update node version to LTS

* Update node version to LTS

* Fix Linting issues

* Fix Linting issues

* Make env variable nullable

* Add touchend listener for mobile

* Make changes to PDF for mobile

* fix(android): change serverUrl to selfhosted first

* feat:2 stage alpine content fetch

* feat:separated start script

* fix:changed to node 22

* Add back youtube functionality and add guide

* trigger build

* Fix cache issue on YouTube

* Allow empty AWS_S3_ENDPOINT

* Allow empty AWS_S3_ENDPOINT

* Add GCHR for all images

* Add GCHR For self hosting.

* Add GCHR For self hosting.

* Test prebuilt.

* Test prebuilt

* Test prebuilt...

* Fix web image

* Remove Web Image (For now)

* Move docker-compose to images

* Move docker-compose files to correct locations

* Remove the need for ARGS

* Update packages, and Typescript versions

* Fix

* Fix issues with build on Web

* Correct push

* Fix Linting issues

* Fix Trace import

* Add missing types

* Fix Tasks

* Add information into guide about self-build

* Fix issues with PDF Viewer

---------

Co-authored-by: keumky2 <keumky2@woowahan.com>
Co-authored-by: William Theaker <wtheaker@nvidia.com>
Co-authored-by: Russ Taylor <729694+russtaylor@users.noreply.github.com>
Co-authored-by: David Adams <david@dadams2.com>
Co-authored-by: Mike Baker <1426795+mbaker3@users.noreply.github.com>
Co-authored-by: m1xxos <66390094+m1xxos@users.noreply.github.com>
Co-authored-by: Adil <mr.adil777@gmail.com>
2025-01-27 13:33:16 +01:00
70d6e9dcae reduce logs 2024-08-29 18:18:17 +08:00
4674321531 reduce blocking domain to 1 hour 2024-08-18 12:37:10 +08:00
e6ebac5e13 fix: special handler for youtube shorts 2024-08-18 11:52:21 +08:00
f2b3a66b72 remove metadata and cover image from content 2024-07-18 11:18:51 +08:00
02280b576b fix: wechat official account articles published date not captured because its format is changed 2024-07-18 10:33:06 +08:00
bbc7b5e600 use @omnivore/utils in import-handler 2024-07-03 22:20:27 +08:00
e0ad660f8f use first paragraph more than 50 characters as exceprt 2024-06-05 08:57:10 +08:00
1791075c2d Merge pull request #3941 from omnivore-app/feat/tik-tok-embeds
Add support for embedding TikTok videos
2024-05-13 16:28:31 +08:00
d23bccf459 upgrade puppeteer-core to prevent ProtocolTimeout and adding more debug logs 2024-05-13 14:28:26 +08:00
7c1178b6e0 Clean up type errors 2024-05-13 14:06:24 +08:00
4d0f1bec88 Add support for embedding TikTok videos 2024-05-13 13:30:54 +08:00
0ac5299c32 do not pass browser instance to content-handler 2024-05-13 13:10:02 +08:00
a091b2c64b fix not saving tweets of domain x.com 2024-05-09 14:56:32 +08:00
310a38882c feat: embed tweets if user saves tweets 2024-05-08 10:11:37 +08:00
88a7e8d85b fix tests 2024-04-04 12:17:15 +08:00
3ee6787e39 Improve transcript generation 2024-03-14 12:48:56 +08:00
f7225b298a Rebase 2024-03-14 12:48:56 +08:00
29cb15de32 use nitter handler for twitter new domain x.com 2024-01-26 21:50:21 +08:00
e96458192d fix redis client cert in nitter handler 2024-01-23 19:30:38 +08:00
d9feb740cb convert content-fetch to typescript 2024-01-18 18:48:46 +08:00
cd3402b98a rewrite puppeteer in typescript 2024-01-18 18:48:46 +08:00
150a456c35 replace redis client library with ioredis 2024-01-16 15:42:50 +08:00
c4773dc904 Landing page improvements and various supporting improvements 2023-10-24 09:43:39 +01:00
38a1290b2a Use scrapingbee for fool.ca 2023-08-30 12:13:52 +08:00
ad6ce8077b speically handle zhihu.com 2023-08-16 11:54:48 +08:00
053625276b Merge pull request #2608 from Podginator/feat/Ars-technica-handler
Ars Technica Multipage handling
2023-08-07 10:43:39 +08:00
0392f6f009 Linting Fixes 2023-08-04 22:36:15 +02:00
49c1976ac9 Add Ars Technica handler for multipage articles 2023-08-04 20:00:03 +02:00
cf5f46026a Add Paywalled Wired Handler 2023-08-02 19:53:20 +02:00
3146b3bcc5 remove redundant logs 2023-07-27 17:55:17 +08:00
a3fa695957 add comment 2023-07-04 20:47:51 +08:00
a818fa721c use asia/shanghai timezone to format published date in wechat articles 2023-07-04 20:46:39 +08:00
43cb72db97 comment 2023-07-04 19:02:47 +08:00
e06f833e21 fix: missing published date in some wechat articles
* some wechat articles do not have `create_time` in the dom javascript so use the `publish_time` node in the dom content
2023-07-04 18:46:35 +08:00
607e22ce94 fix: tweet with no text failed to save 2023-06-26 21:46:52 +08:00
937be20928 fix: tweet format 2023-06-26 21:11:04 +08:00
15c964481a fix: tweet not saved
* compare username ignoring case
2023-06-26 18:07:47 +08:00
978a21d160 fix typo 2023-06-23 16:59:34 +08:00
a5ddc9aced lowercase username before comparing 2023-06-23 16:56:03 +08:00
41f0c5b3a3 skip saving unavailable and earlier replies 2023-06-23 16:26:07 +08:00
4b896dd56a update instances 2023-06-23 14:30:13 +08:00
918585dfb6 get the redis connection info from env 2023-06-23 14:08:20 +08:00
e2b66be75d fix dependecies 2023-06-23 13:33:38 +08:00
716fdf9a61 expire key after 1 day 2023-06-23 13:19:18 +08:00
fc0257b480 sort by descending order 2023-06-23 12:52:27 +08:00
b746046727 update default score 2023-06-23 12:34:31 +08:00
9ef40b3ed9 deduct latency from the score of the member in sorted set 2023-06-23 12:27:34 +08:00
ae504a970b save instances in redis as a sorted set 2023-06-23 12:17:41 +08:00
4607754f58 reduce timeout to 20s 2023-06-23 11:36:30 +08:00