8000 lense/docs/web-clipper.md at v0.10.0 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
171 lines (130 loc) · 9.8 KB

web-clipper.md

File metadata and controls

171 lines (130 loc) · 9.8 KB

Web clipper

Status: built (Chrome, Edge, Firefox and Safari). Every browser's build comes from the same code; each gets a target in extensions/browser/targets/.

The Lens web clipper is a browser extension that turns what you're reading into Markdown and sends it to your Lens, where it's imported like pasted text and runs through the namespace's pipeline. It works like MarkDownload: you choose what to clip, look over the Markdown, and send it, copy it or download it as a .md file.

What it clips

  • The page: the article, through Mozilla's Readability (menus, ads and footers left out; turn that off in the options to keep the whole page), with its metadata: title, author, site, publish date, lead image, keywords and language, from the page's meta tags, Open Graph and JSON-LD.
  • The selection: what you've selected, keeping the list, table, quote or code block it's in.
  • A part you pick: hover to outline an element, ↑ to widen to its parent, ↓ to narrow again, click or Enter to take it, Esc to stop.
  • A link or an image from the right-click menu: a link is sent for Lens to capture itself (a PDF as it is, a page printed by Chromium on the full image); an image is sent as a file.

Relative links and images become absolute, lazy-loaded images are loaded, code blocks keep their language, and tables, strikethrough and task lists become GitHub-style Markdown.

Setting it up

  1. In Lens, open a namespace's admin page and make an import webhook (see the API, Import webhooks). Copy its token: it's shown once.
  2. In the clipper's options, enter your Lens address and the token and press Add and check. The browser asks to let the clipper talk to that address; the clipper then checks the token without importing anything.

Add one per namespace you clip into, and pick between them in the popup. The tokens stay on this device; every other setting syncs through your browser account. Plain http:// addresses are only accepted on this computer or the local network.

Ways to clip

Toolbar button, or Alt+Shift+L the popup: clips the selection if there is one, else the page, for you to look over and send
Right-click, or Alt+Shift+P / S clip the page / the selection and send it straight away (a ✓ on the button when it's in)
Right-click Pick part of the page, or Alt+Shift+K the picker, then send

With Right-click and shortcut clips open the popup on, those open the popup to look over first instead.

The same extension has the Lens assistant in a side panel: Ask Lens in the popup, or Ask Lens… on the right-click menu. See The assistant in your browser.

Templates

What's sent is the template with the clip's variables filled in. The default puts the metadata in YAML front matter and the content under the title:

---
title: {title|yaml}
source: {url}
author: {byline|yaml}
site: {siteName|yaml}
published: {published:YYYY-MM-DD}
clipped: {date:YYYY-MM-DDTHH:mm:ssZ}
image: {image}
tags: {tags|list}
---

# {title}

{content}

A front matter line whose value comes out empty is left out. Selections and picked parts can have a template of their own; the title has one too ({title} by default).

Variables: title, url, domain, origin, siteName, byline, excerpt, published, modified, date (now), keywords, tags (yours, from the options), image, lang, clipType (page, selection or element) and content.

  • Dates take a format after a colon: {date:YYYY-MM-DD HH:mm} (YYYY YY MM M DD D HH H mm ss Z).
  • Lists take a separator after a colon: {keywords:, }.
  • Filters follow a bar and chain: {title|kebab}, {title|lower|trim}. There are lower, upper, trim, kebab, snake, camel, pascal, title, filename, yaml, list, hashtags, json and first.
  • {{ and }} are literal braces.

Markdown and clipping options

Heading style, bullet marker, fenced or indented code and its fence, emphasis and strong markers, inline or reference links, links kept or reduced to their text, images kept, reduced to their alt text or dropped, and GitHub tables on or off.

Also send the clip's images as files pushes up to 19 of the content's images along with the Markdown; each becomes a resource in Lens. The browser asks once to let the clipper read images from any site. Also ask Lens to capture the page itself sends the address too, for a PDF snapshot next to the Markdown (needs Chromium on your Lens).

Building it

cd extensions/browser
npm ci
npm test            # templates, metadata, Markdown, sending (jsdom)
npm run build       # dist/<target>/ for every target; --target chrome for one

From the repository root, pnpm exec nx run-many -t test build -p clipper does the same (installing first when the lockfile changed), and CI runs it in Other projects when a change affects the clipper (Monorepo and CI).

Load dist/chrome with Load unpacked on chrome://extensions (Developer mode on). The Browser extension workflow builds every target on each pull request that changes the clipper and keeps lens-clipper-<target>.zip with the run; a release gets them attached.

A target is extensions/browser/targets/<name>.json, merged over src/manifest.json (a null removes a key), so a browser that needs something different (Firefox's browser_specific_settings and background scripts, say) adds only that. The code reaches the extension API as globalThis.browser ?? globalThis.chrome.

Firefox

Firefox 128 or later (desktop). targets/firefox.json runs the background as a script instead of a service worker and gives the add-on its id, lens-clipper@adeelahmad.github.io; the assistant opens in Firefox's sidebar (sidebar_action) instead of a side panel. To try a build, open about:debugging, choose This Firefox, then Load Temporary Add-on… and pick dist/firefox/manifest.json; it stays until Firefox restarts.

The Browser extension (Firefox) workflow runs Mozilla's linter, web-ext lint (the checks addons.mozilla.org runs), on every pull request that changes the clipper, and keeps lens-clipper-firefox.zip (ready to upload to addons.mozilla.org) with the run. On a release, if the AMO_JWT_ISSUER and AMO_JWT_SECRET secrets hold an addons.mozilla.org API key, Mozilla signs it as an unlisted add-on and the release gets lens-clipper-firefox.xpi, which any Firefox installs by opening it. That copy is built from dist/firefox-selfhosted, so it updates itself from the newest release (see below).

Releases and updates

Every Lens release builds the clipper for each browser and attaches lens-clipper-<browser>.zip to it. How a copy keeps itself up to date depends on where it came from. Each publishing step runs only when its repository secrets are set, and is skipped otherwise:

Browser Updates through Secrets
Chrome the Chrome Web Store: each release is uploaded and published CWS_EXTENSION_ID, CWS_CLIENT_ID, CWS_CLIENT_SECRET, CWS_REFRESH_TOKEN
Edge Microsoft Edge Add-ons: each release is uploaded and submitted EDGE_PRODUCT_ID, EDGE_CLIENT_ID, EDGE_API_KEY
Chrome, Edge (self-hosted) lens-clipper-<browser>.crx, which checks lens-clipper-<browser>-updates.xml on the newest release CRX_PRIVATE_KEY
Firefox the Mozilla-signed lens-clipper-firefox.xpi, which checks lens-clipper-firefox-updates.json on the newest release AMO_JWT_ISSUER, AMO_JWT_SECRET
Safari the App Store, from the Safari build APPLE_*, see Safari
  • Chrome and Edge only install a .crx from outside their stores through policy (ExtensionInstallForcelist with the update XML's address) or on Linux. For everyone else the stores are the way to get updates.
  • CRX_PRIVATE_KEY is a PEM RSA key (openssl genrsa 2048), and it fixes the extension's ID: keep it, because a new key makes a different extension.
  • The self-hosted builds are dist/<browser>-selfhosted. They are built when EXT_UPDATE_BASE is set, which a release sets to its releases/latest/download address, so every installed copy finds the newest version.

Safari

Safari runs the same clipper, carried inside a small macOS app (Lens Clipper.app). Each release has lens-clipper-safari-app.zip attached; to build it yourself on a Mac with Xcode, run extensions/browser/safari/build.sh.

To install the zip: unzip it, move Lens Clipper.app to Applications and open it once. In Safari, turn on Settings > Advanced > Show features for web developers and Develop > Allow Unsigned Extensions, then enable Lens Clipper in Settings > Extensions. The zip is unsigned, so Safari forgets that allowance each time it restarts. Safari asks before the clipper reads a site; allow your Lens server and the pages you clip.

Updates: Safari extensions update only through the App Store. When the repository has the App Store Connect secrets, the Safari clipper workflow signs each release and uploads it there:

Secret What it is
APPLE_TEAM_ID Your Apple Developer team id
APPLE_API_KEY_ID, APPLE_API_ISSUER An App Store Connect API key with the Admin role (Users and Access > Integrations), and its issuer id
APPLE_API_KEY_P8 The contents of that key's AuthKey_….p8 file

Create the app's record in App Store Connect once, with the bundle id dev.lens.clipper. Each upload then shows up in TestFlight, which keeps testers up to date; submit a build for review to release it on the Mac App Store, where Safari updates it for everyone. Without the secrets the upload is skipped and only the zip is attached.

0