Commit Graph

13 Commits

Author SHA1 Message Date
5f11726bd5 set lock_timeout while running migrations 2024-06-19 11:52:22 +08:00
612151c151 remove elastic from docker-compose 2023-10-05 14:24:04 +08:00
71826148fd Add type to the highlight in the elasticsearch and three types are HIGHLIGHTS, REDACTION and NOTE 2023-03-22 10:34:40 +08:00
2ef5d101bb Create index if not exists in migration 2022-06-17 22:07:41 +08:00
b9c2c1186b Fix batch_size=500 to avoid exceeding max memory limit and requests_per_second=250 to throttle update requests
Fix batch_size=500 to avoid exceeding max memory limit and requests_per_second=250 to throttle update requests
2022-05-05 21:10:49 +08:00
7183ca2e89 Add request timeout = 30m in elastic client 2022-05-05 13:28:01 +08:00
911afd77e5 Reduce requests per second and scroll_size 2022-05-05 12:40:44 +08:00
96fbd84bbb Extend timeout 2022-05-05 09:02:56 +08:00
cd9c6a40d9 Tune the update speed 2022-05-04 22:44:28 +08:00
8f60a9f905 Add default state to migration script 2022-05-04 21:54:28 +08:00
46aac8ac0a Match the env var name (#512) 2022-04-29 14:10:56 +08:00
2b70d480d2 Remove article saving request (#493)
* Add state and taskName in elastic page mappings

* Add state and taskName in elastic page interface

* Create page with PROCESSING state before scrapping

* Update createArticleRequest API

* Fix tests

* Add default state for pages

* Update createArticle API

* Update save page

* Update save file

* Update saving item description

* Show unable to parse content for failed page

* Fix date parsing

* Search for not failed pages

* Fix tests

* Add test for saveUrl

* Update get article saving request api

* Update get article test

* Add test for articleSavingRequest API

* Add test for failure

* Return new page id if clientRequestId empty

* Update clientRequestId in savePage

* Update clientRequestId in saveFile

* Replace article with slug in articleSavingRequest

* Add slug in articleSavingRequest response

* Depreciate article

* Use slug in web

* Remove article and highlight fragments

* Query article.slug on Prod

* Show unable to parse description for failed page

* Fix a bug having duplicate pages when saving the same url multiple times

* Add state in response

* Rename variables in removeArticle API

* Rename state

* Add state in response in web

* Make state an enum

* Open temporary page by link id

* Use an empty reader view as the background for loading pages

* Progressively load the article page as content is loaded

* Add includePending flag in getArticles API

* Set includePending = true in web

* Add elastic update mappings in migration script

* Add elastic mappings in docker image

* Move index_settings.json to migrate package

* Remove elastic index creation in api

* Move elastic migrations to a separate directory

* Remove index_settings from api docker image

Co-authored-by: Jackson Harper <jacksonh@gmail.com>
2022-04-29 13:41:06 +08:00
84f32935f5 Open source omnivore 2022-02-11 09:24:33 -08:00