Commit Graph

139 Commits

Author SHA1 Message Date
d05f091876 Add debug logging for GQL errors 2022-04-12 12:23:55 -07:00
6c5e88cf12 Merge pull request #405 from omnivore-app/feature/label-max-length
Set label name max length = 64 char
2022-04-12 11:14:20 -07:00
ae0d1dd2ee Feature/search highlights backend (#395)
* add highlight mappings

* return highlight in resolvers

* temporarily skip highlight tests

* add test for getting highlights

* update merge highlight

* separate elastic methods

* roll back merge highlight test

* add highlight to elastic script

* update delete highlight in elastic

* migrate highlight data from postgres to elastic

* rescue not found exception when page is not found in the migration script

* exclude highlights in searching pages results

* search pages with highlights only with has:highlight query

* add search endpoint to search pages or highlights

* reduce code smell in search api

* fix rebase error

* fix tests

* add test for search highlight

* add test for new search endpoint

* add labels to search results

* update schema

* update search query

* fix update/share highlights

* fix rebase error

* fix tests

* add highlight model in elastic

* add savedAt and publishedAt date range in search query

* add sort by updated and recently read

* fix tests

* close db connection when tests are done

* test github action

* revert github action test

* fix rebase error

* add docker-compose for api-test

* remove unused env

* remove highlights with no page attached to

* allow get_articles resolver to search for query so we can merge it without web changes
2022-04-12 12:31:08 +08:00
2ebdaba780 add generate api key api and test (#392)
* add generate api key api and test

* test if user can make api call with the api key
2022-04-12 12:11:45 +08:00
bb665a704b make label name <= 64 char length 2022-04-12 12:08:02 +08:00
b4420839bd fix: fail to search by excluding labels with capital letter (#400)
* fix: fail to search by excluding labels with capital letter

* make all the excluded labels lowcased

* Revert "make all the excluded labels lowcased"

This reverts commit 866bed40801af522cb7f07aa3119ed813f7c34b6.
2022-04-11 14:05:21 +08:00
48ca13818f Prettier fixes 2022-04-06 10:45:01 -07:00
57f865c9a2 Update to typeorm 0.3 2022-04-06 10:37:13 -07:00
1cdb7753c3 Update typeorm usage 2022-04-06 09:56:32 -07:00
06360a5993 Merge branch 'main' into OMN-190 2022-04-06 09:24:49 -07:00
d1269f815e identify a newsletter by checking both list-url and unsubscribe-url in headers and email sender address 2022-04-06 20:59:14 +08:00
7569e988bf upgrade typeorm to 3.0 (#359)
* upgrade typeorm to 3.0

* use new datasource object in typeorm 3

* fix tests

* fix tests

* migrate before creating connection

* fail the test if migration failed
2022-04-06 10:32:41 +08:00
9845a49040 Remove semicolon 2022-04-04 08:17:46 -07:00
c8ec7d636e Use require to import address parser function 2022-04-04 08:11:26 -07:00
92925b6808 Prettier fixes 2022-04-03 10:21:16 -07:00
da28998130 Pull ul list out of newsletter blurb
next/react doesnt want child elements of the paragraphs

Improve formatting

Improve wording

Use buttons in the subscribe directly blocks

Simplify docs on setting up forwarding rules

Add extra padding on bottom of help docs

Remove unused style

Add emails help page

Improve formatting

Prefetch page content on iOS

Reduce the reader overly length now that items are precached

Add invalidation when highlights are added to items

fix missing index_settings.json file in api dockerfile for creating elastic index (#363)

Handle full email address objects in the to param from sendgrid

These come in a format like:

"jacksonh-dfdf@inbox.omnivore.app" <jacksonh-dfdf@inbox-demo.omnivore.app>

New IDs for short highlights because they dont cascade delete now

Testing CI issues

Simplify test

CI test

Use promises for async tests

Temporarily remove test to debug CI

Re-enable

re-enable test, return error

Specify a userId when looking up saved email pages

create a unique url for newsletters without a URL

Use 500ms on page test timeouts

Increase timeout

Dont use deep equal to match newsletter label

Run just the labels API

Run against just the newsletter emails

Run without the page tests

Fix

Set the allow uncaught flag

Remove highlight tests

Remove newsletters tests

more resolver tests

Remove newsetter tests

Comment out resolver tests

Use nock for external requests in tests

Specify puppeteer url for tests

Comment out more tests

uncomment tests

re-enable

re-enable email test

Re-disable

Re-enable one pdf attachment test

Re-disable pdf attachment test

Use promises on setTimeout tests

rm label tests

mv label tests into a context

Comment out pdf tests

Comment out pdf tests

Async test

Async wrappers

Delay when creating test pages

More debugging

Unique short ids

Remove potentially problematic test

Fetch page before returning for test

handler in before block

more debugging

More debugging

Move errors checks into contexts

Use a context when saving newsletters to force index refresh

Prettier fix

Fix newsletter label check, remove setTimeout

Re-enable test

timeout on pdf router handler

Fix method call

comment out PDF test

Unique fake username

Comment out PDF test

Debugging signed urls

Re-enable

New email

pdf test

PDF tests

Comment out pdf test

Add nock stubs for email URLs

Use full address for PDF test

Remove debug

Use full email addresses
2022-04-02 16:56:24 -07:00
1978a92a1b Effected review corrections 2022-03-31 13:03:50 +00:00
a8faaf835c Remove debug sort on labels 2022-03-30 14:13:26 -07:00
2c15516d96 Order labels by createdAt so they maintain a consistent order 2022-03-30 13:28:37 -07:00
11db7be473 [Omn-190] - [Settings View] - Labels 2022-03-30 12:24:29 +00:00
6d405432af add site_name and site_icon to page model and return in resolver (#341)
* add site_name and site_icon to page model and return in resolver

* fix tests
2022-03-30 10:43:10 +08:00
2bada0b9df Merge pull request #293 from omnivore-app/feature/priority-by-rate-limit
Use lower priority queue for large imports
2022-03-28 13:19:35 -07:00
d9af397b9c Merge pull request #331 from omnivore-app/fix/update-page-after-deletion
fix document_missing_exception being thrown to sentry when updating page in elastic
2022-03-28 10:30:31 -07:00
fd18affaec fix document_missing_exception being thrown to sentry when updating page in elastic 2022-03-28 21:16:33 +08:00
b6c04f3508 fix saving report item by linking to elastic page 2022-03-28 20:38:54 +08:00
1cdbfaca26 Merge pull request #294 from omnivore-app/feature/add-created-or-updated_at-to-newsletter_email
add created_at and updated_at to newsletter emails and sort by create…
2022-03-25 09:10:54 -07:00
27735f7310 add page id in updated page json file in gcs (#319) 2022-03-25 13:50:43 +08:00
e7203bebb5 add a function_resolver to return originalArticleUrl from article.url (#317)
* add a function_resolver to return originalArticleUrl from article.url

* add a test
2022-03-25 11:25:30 +08:00
6597cc37b6 add created_at and updated_at to newsletter emails and sort by created_at desc 2022-03-22 18:43:47 +08:00
acec3b1acb Use lower priority queue for large imports
* add a count query in elastic
  * if priority is not set, check the rate limit of user's libary
    - if 5 articles added in the last minute: use low queue
    - default: use normal queue
2022-03-22 18:08:08 +08:00
1893e36375 Fix uploading endpoint returns 504 (#289)
* add timeout = 60s for uploading to private bucket

* add debug logs

* send response to client to close connection

* add tests for upload

* add PUBSUB_VERIFICATION_TOKEN in .env.test
2022-03-22 14:20:40 +08:00
3e9063c145 return empty content instead of null in search result (#291) 2022-03-22 13:34:44 +08:00
bee3b3c6fa fix article_id type mismatch (uuid => varchar) when saving highlights in postgres by setting article_id to be undefined (#285) 2022-03-22 11:48:01 +08:00
4ca7c89622 format date string as yyyy-mm-dd (#288) 2022-03-22 11:46:39 +08:00
c15efe6aad Fix sentry log size error (#287)
* reduce sentry log size error by partially logging page created in elastic

* remove some debugging logs

* remove more debugging logs
2022-03-22 11:45:58 +08:00
f9bce1932c Print the logging level on startup 2022-03-21 09:31:05 -07:00
77050905a5 retry max three times if update labels is conflicted 2022-03-18 10:39:11 +08:00
9d8d2e40d7 reduce time to save reading progress in elastic update 2022-03-18 10:29:01 +08:00
ac567f7e71 retry max three times if update is conflicted 2022-03-18 10:17:20 +08:00
7910565dbf Merge pull request #251 from omnivore-app/fix/jsonld-encoding
Use html decoding when getting values from fetched oembed
2022-03-17 16:53:21 -07:00
563131ea23 Fix/newsletter label not add (#256)
* default sort by savedAt to avoid recent read page to appear on top of the list

* revert lint on generated code

* fix elastic update script syntax error
2022-03-17 13:22:38 +08:00
a2ce98229e default sort by savedAt to avoid recent read page to appear on top of… (#254)
* default sort by savedAt to avoid recent read page to appear on top of the list

* revert lint on generated code
2022-03-17 10:54:00 +08:00
af037a2837 make readingProgress required in the elastic page data (#253)
* make readingProgress required in the elastic page data

* delete readingProgress from function_resolvers because we have stored them in elastic
2022-03-17 10:06:21 +08:00
ff1200f3a1 Use html decoding when getting values from fetched oembed
If we fetch oembed data from an external source, instead of
handling it in readabilityjs we need to html decode it.
2022-03-16 15:29:42 -07:00
89f3719ba3 Fix linting 2022-03-16 14:23:39 -07:00
0314d8cf17 Add some extra logging 2022-03-16 13:53:55 -07:00
dc27c0c7a4 Return 0 if looking up page reading progress fails 2022-03-16 13:33:36 -07:00
27143cbedf Remove function handlers for reading progress
These shouldn't be needed anymore because the values are stored
in elastic.
2022-03-16 13:11:36 -07:00
07dcd5da26 add label to page only if not exists 2022-03-16 23:00:37 +08:00
1c4dcd7b00 automatically update updatedAt when page is updated in elastic 2022-03-16 22:28:39 +08:00