|
|
eb0f0b6c4f
|
improve data migration speed
|
2023-10-05 14:32:05 +08:00 |
|
|
|
86478c13d2
|
fix sql
|
2023-10-05 14:32:05 +08:00 |
|
|
|
42e01613d4
|
inner join user table when inserting highlight to make sure user exists
|
2023-10-05 14:32:05 +08:00 |
|
|
|
4d846c520b
|
create a new sanitized tuple because tuple is immutable
|
2023-10-05 14:31:07 +08:00 |
|
|
|
475e32e859
|
Catch error from converting string to datetime
|
2023-10-05 14:31:07 +08:00 |
|
|
|
71c203cdb2
|
Update migration script to check the unique index while inserting data into library_item table
|
2023-10-05 14:31:07 +08:00 |
|
|
|
e950662ced
|
Replace the unique constraint on library_item table with a unique index of userId and md5 hashed original url
|
2023-10-05 14:31:07 +08:00 |
|
|
|
6604c42bf4
|
remove null byte from content
|
2023-10-05 14:31:07 +08:00 |
|
|
|
093440e1ab
|
replace surrogates with placeholder
|
2023-10-05 14:31:07 +08:00 |
|
|
|
aa9c97fcbc
|
encode with utf-16 if failed
|
2023-10-05 14:31:07 +08:00 |
|
|
|
f9ce8f6b3e
|
encode surrogatepass to avoid error when inserting into postgres
|
2023-10-05 14:31:07 +08:00 |
|
|
|
65115b3ba7
|
fix a typo
|
2023-10-05 14:31:07 +08:00 |
|
|
|
4c30a4f561
|
use enumerate to get index from for loop
|
2023-10-05 14:31:07 +08:00 |
|
|
|
e68f7b42ba
|
print original id for debugging
|
2023-10-05 14:31:07 +08:00 |
|
|
|
71e3c716be
|
excute insert query one by one if batch insert failed and if string is too long for tsvector, disable the update_library_item_tsv trigger temporarily and insert the failed one again
|
2023-10-05 14:31:07 +08:00 |
|
|
|
663ccdae41
|
try to insert library item one by one if batch insert failed
|
2023-10-05 14:31:07 +08:00 |
|
|
|
323be5376a
|
truncate long description to 1MB
|
2023-10-05 14:31:07 +08:00 |
|
|
|
85febfa8b6
|
fix a typo
|
2023-10-05 14:31:07 +08:00 |
|
|
|
c7e56fbce6
|
add PG_TIMEOUT to env var
|
2023-10-05 14:31:07 +08:00 |
|
|
|
6d6d34e94e
|
convert ES_TIMEOUT to int
|
2023-10-05 14:31:07 +08:00 |
|
|
|
e9527bb5ad
|
add es_timeout to env var
|
2023-10-05 14:31:07 +08:00 |
|
|
|
f46006e293
|
add end_time to the migrate script
|
2023-10-05 14:31:07 +08:00 |
|
|
|
4d1d9a8c1b
|
skip item if content length is greater than 1MB
|
2023-10-05 14:31:07 +08:00 |
|
|
|
0ff9fdef04
|
add elastic scroll time in the env var
|
2023-10-05 14:31:07 +08:00 |
|
|
|
9e5196406e
|
revert rls on old tables
|
2023-10-05 14:31:07 +08:00 |
|
|
|
d5c247a98d
|
make cool-down time an env var
|
2023-10-05 14:31:07 +08:00 |
|
|
|
330e50fba5
|
increase scroll to 2m and cool down for 5 seconds
|
2023-10-05 14:31:07 +08:00 |
|
|
|
7d4948a43b
|
reduce scroll to 1m
|
2023-10-05 14:31:07 +08:00 |
|
|
|
8f1664b058
|
increase timeout
|
2023-10-05 14:31:07 +08:00 |
|
|
|
a83f7921f4
|
reduce shortId length to 14
|
2023-10-05 14:31:07 +08:00 |
|
|
|
a947310d30
|
reduce shortId length to 14
|
2023-10-05 14:31:07 +08:00 |
|
|
|
ecbabacc78
|
set reading progress to 0 if null
|
2023-10-05 14:31:07 +08:00 |
|
|
|
a445bc55d4
|
add default item_type
|
2023-10-05 14:31:07 +08:00 |
|
|
|
b4512d2d1c
|
convert ES_SCAN_SIZE to int
|
2023-10-05 14:31:07 +08:00 |
|
|
|
b20ad4a210
|
reorder migrate sql
|
2023-10-05 14:31:07 +08:00 |
|
|
|
ac77580cca
|
fix label names update not reflected
|
2023-10-05 14:31:07 +08:00 |
|
|
|
15f9017291
|
fix in:archive search
|
2023-10-05 14:31:07 +08:00 |
|
|
|
dc98719b85
|
reduce log
|
2023-10-05 14:31:07 +08:00 |
|
|
|
0c5ba53c26
|
fix recommendations
|
2023-10-05 14:31:07 +08:00 |
|
|
|
9cf5d2bc4e
|
change permission of the migration script
|
2023-10-05 14:31:06 +08:00 |
|
|
|
4ff5484d8e
|
change item_type to text
|
2023-10-05 14:31:06 +08:00 |
|
|
|
101825df46
|
add assertion of migrated data in python script
|
2023-10-05 14:31:06 +08:00 |
|
|
|
b2fbbfb073
|
add deleting existing highlights from postgres before migration
|
2023-10-05 14:31:06 +08:00 |
|
|
|
41e8ee3aca
|
finish migration script
|
2023-10-05 14:31:06 +08:00 |
|
|
|
68e381deb2
|
revert note changes
|
2023-10-05 14:31:06 +08:00 |
|
|
|
fdd771bc06
|
add migration script
|
2023-10-05 14:31:06 +08:00 |
|
|
|
23b6997d7e
|
resolve db migration script version
|
2023-10-05 14:30:10 +08:00 |
|
|
|
024d558cfe
|
update labels rls
|
2023-10-05 14:30:10 +08:00 |
|
|
|
4b95b732cf
|
fix highlight tests
|
2023-10-05 14:30:10 +08:00 |
|
|
|
b036149fe2
|
fix feature tests
|
2023-10-05 14:30:10 +08:00 |
|
