da28998130
next/react doesnt want child elements of the paragraphs Improve formatting Improve wording Use buttons in the subscribe directly blocks Simplify docs on setting up forwarding rules Add extra padding on bottom of help docs Remove unused style Add emails help page Improve formatting Prefetch page content on iOS Reduce the reader overly length now that items are precached Add invalidation when highlights are added to items fix missing index_settings.json file in api dockerfile for creating elastic index (#363) Handle full email address objects in the to param from sendgrid These come in a format like: "jacksonh-dfdf@inbox.omnivore.app" <jacksonh-dfdf@inbox-demo.omnivore.app> New IDs for short highlights because they dont cascade delete now Testing CI issues Simplify test CI test Use promises for async tests Temporarily remove test to debug CI Re-enable re-enable test, return error Specify a userId when looking up saved email pages create a unique url for newsletters without a URL Use 500ms on page test timeouts Increase timeout Dont use deep equal to match newsletter label Run just the labels API Run against just the newsletter emails Run without the page tests Fix Set the allow uncaught flag Remove highlight tests Remove newsletters tests more resolver tests Remove newsetter tests Comment out resolver tests Use nock for external requests in tests Specify puppeteer url for tests Comment out more tests uncomment tests re-enable re-enable email test Re-disable Re-enable one pdf attachment test Re-disable pdf attachment test Use promises on setTimeout tests rm label tests mv label tests into a context Comment out pdf tests Comment out pdf tests Async test Async wrappers Delay when creating test pages More debugging Unique short ids Remove potentially problematic test Fetch page before returning for test handler in before block more debugging More debugging Move errors checks into contexts Use a context when saving newsletters to force index refresh Prettier fix Fix newsletter label check, remove setTimeout Re-enable test timeout on pdf router handler Fix method call comment out PDF test Unique fake username Comment out PDF test Debugging signed urls Re-enable New email pdf test PDF tests Comment out pdf test Add nock stubs for email URLs Use full address for PDF test Remove debug Use full email addresses
Readability.js
A standalone version of the readability library used for Firefox Reader View.
Usage on the web
To parse a document, you must create a new Readability object from a DOM document object, and then call parse(). Here's an example:
var article = new Readability(document).parse();
This article object will contain the following properties:
title: article titlecontent: HTML string of processed article contenttextContent: text content of the article (all HTML removed)length: length of an article, in charactersexcerpt: article description, or short excerpt from the contentbyline: author metadatadir: content direction
If you're using Readability on the web, you will likely be able to use a document reference
from elsewhere (e.g. fetched via XMLHttpRequest, in a same-origin <iframe> you have access to, etc.).
Optional
Readability's parse() works by modifying the DOM. This removes some elements in the web page.
You could avoid this by passing the clone of the document object while creating a Readability object.
var documentClone = document.cloneNode(true);
var article = new Readability(documentClone).parse();
Usage from Node.js
Readability is available on npm:
npm install @omnivore/readability
In Node.js, you won't generally have a DOM document object. To obtain one, you can use external
libraries like jsdom. While this repository contains a parser of
its own (JSDOMParser), that is restricted to reading XML-compatible markup and therefore we do
not recommend it for general use.
If you're using jsdom to create a DOM object, you should ensure that the page doesn't run (page)
scripts (avoid fetching remote resources etc.) as well as passing it the page's URI as the url
property of the options object you pass the JSDOM constructor.
Example:
var { Readability } = require('@omnivore/readability');
var JSDOM = require('jsdom').JSDOM;
var doc = new JSDOM("<body>Here's a bunch of text</body>", {
url: "https://www.example.com/the-page-i-got-the-source-from"
});
let reader = new Readability(doc.window.document);
let article = reader.parse();
What's Readability-readerable?
It's a quick-and-dirty way of figuring out if it's plausible that the contents of a given document are suitable for processing with Readability. It is likely to produce both false positives and false negatives. The reason it exists is to avoid bogging down a time-sensitive process (like loading and showing the user a webpage) with the complex logic in the core of Readability. Improvements to its logic (while not deteriorating its performance) are very welcome.
Security
If you're going to use Readability with untrusted input (whether in HTML or DOM form), we strongly recommend you use a sanitizer library like DOMPurify to avoid script injection when you use the output of Readability. We would also recommend using CSP to add further defense-in-depth restrictions to what you allow the resulting content to do. The Firefox integration of reader mode uses both of these techniques itself. Sanitizing unsafe content out of the input is explicitly not something we aim to do as part of Readability itself - there are other good sanitizer libraries out there, use them!
Contributing
For outstanding issues, see the issue list in this repo, as well as this bug list.
Any changes to Readability.js itself should be reviewed by an appropriate Firefox/toolkit peer, such as @gijsk, since these changes will be merged to mozilla-central and shipped in Firefox.
To test local changes to Readability.js, you can use the automated tests. There's a node script to help you create new ones.
Tests
Please run eslint as a first check that your changes are valid JS and adhere to our style guidelines.
To run the test suite:
$ mocha test/test-*.js
To run a specific test page by its name:
$ mocha test/test-*.js -g 001
To run the test suite in TDD mode:
$ mocha test/test-*.js -w
Combo time:
$ mocha test/test-*.js -w -g 001
Benchmarks
Benchmarks for all test pages:
$ npm run perf
Reference benchmark:
$ npm run perf-reference
License
Copyright (c) 2010 Arc90 Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.