8000 feat(playlists): keep moments as clips and play them back to back by adeelahmad · Pull Request #685 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
10BC8 Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,14 +199,16 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
entity and topic, written by you or the assistant, in a tree where notes are renamed, moved, copied and pinned (pins
are your own) and each note shows the notes it's inside and the ones inside it, with today's note one click away,
templates to start from, a search by their words, a download as Markdown, and Undo and Recently deleted instead of a
confirm. Export an asset or a collection as an evidence PDF with everything Lens found about it, or the parts you
confirm. Keep a moment of any recording, video or document as a clip with a title and a note, in one click from the
transcript or a highlight, and play clips from any resources back to back in a playlist, reordered by dragging and
taken out with Undo. Export an asset or a collection as an evidence PDF with everything Lens found about it, or the parts you
pick. Publish a namespace's resources, episodes, timeline activities or text as a private RSS feed or a podcast
feed that any feed reader or podcast app follows (Feeds, from the Podcasts page), and let a routine fill it on a schedule with
what's new, or make an episode about the week's new resources. Follow a saved collection in your podcast app: its
page gives your own private address for it, listing its assets with audio or video that you can read.
([Podcasts](docs/podcasts.md),
[specs from transcripts](docs/notes.md#specs-from-transcripts), [notes](docs/notes.md),
[evidence PDFs](docs/api.md#evidence-pdfs), [published feeds](docs/feeds.md);
[playlists](docs/playlists.md), [evidence PDFs](docs/api.md#evidence-pdfs), [published feeds](docs/feeds.md);
[checks](docs/requirements.md#learn-from-it))
- **Automate it.** Pipelines and workflows on a canvas (with Convert steps that make a video's soundtrack an audio
asset and a document's pages image assets, and Read steps that pick their own converter, such as anytopdf; what their model steps write about
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/clips.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
- **Clips and playlists.** Keep a moment of any recording, video or document as a clip, with a title and a note: select
words in a transcript and choose Add to playlist, or add a highlight from its tab. One click puts it at the end of
the playlist you used last (a new one called Clips when you have none), with Undo. The new Playlists page lists yours
and the ones others shared; a playlist's page plays its clips back to back on each resource's own audio or video,
reorders them by dragging or with Up and Down, and takes one out with Undo. A shared playlist shows each reader only
the clips of resources they can read. Agents use `/api/v1/playlists` and the MCP tools `list_playlists` and
`add_to_playlist`. New tables only: nothing to do when upgrading ([Playlists](docs/playlists.md)).
41 changes: 41 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1845,6 +1845,47 @@ assets with audio or video that you can read) and when an app last read it, made
`POST /{cid}/feed/token` gives it a new address and `DELETE /{cid}/feed` stops it
([A collection's podcast feed](feeds.md#a-collections-podcast-feed)).

## playlists

Clips of moments of any resources, kept in order to play back to back.

```
GET /api/v1/playlists
POST /api/v1/playlists
GET /api/v1/playlists/{pid}
PATCH /api/v1/playlists/{pid}
DELETE /api/v1/playlists/{pid}
POST /api/v1/playlists/{pid}/items
PATCH /api/v1/playlists/{pid}/items/{iid}
DELETE /api/v1/playlists/{pid}/items/{iid}
PUT /api/v1/playlists/{pid}/order
```

A playlist is its maker's: a `name` unique among theirs (up to 120 characters), a `description`, and `shared`. A
private one is theirs alone; a shared one is read by everyone signed in, and each reader gets only the clips of
resources they have a role on (in the resource's namespace, or on its collection), whoever made it. Only its maker
changes it. Up to 200 each. Sharing and unsharing are audited (`playlist.share`, `playlist.unshare`).

A clip is a moment of a resource you can read: `POST /{pid}/items` with the `recording` and `t0` and `t1` (ms from
the start; a document's pages have a reading-pace clock; without `t1`, or with one not after `t0`, it lasts 30
seconds, and it stops at the end of the resource), a `title` (up to 200 characters), a `note` (up to 2,000) and the
`quote` picked in the text, at `position` (from 0; default the end). With `highlight` it clips that
[highlight](#highlights): its resource, passage and label, unless given; the clip stays when the highlight goes.
Clips aren't highlights: anyone who can read a resource keeps a clip of it, and only for themselves, where an editor
makes a highlight for everyone. Up to 500 in a playlist. `PATCH` changes a clip's title, note (empty clears them) or
times; `DELETE` takes it out, and Undo adds it again with its `position`. `PUT /{pid}/order` with `items`, clip ids
in their new order, puts those first, as named, and the clips not named after them, as they were.

`GET /playlists` lists yours, the latest used first (the first is where a clip goes when none is named; adding,
changing, removing or reordering clips uses a playlist), then the ones others shared, each with `count` and
`duration_ms` of the clips you can read in it. `GET /{pid}` is everything a player needs: the clips in order, each
with its resource's `recording_title`, `namespace` and `media` kind, its signed `audio` link (the resource's audio or
video, as the recording player gets it; `null` when it has none, so play its text), its times and `link`, its page in
the web app at the clip's start. `hidden` counts the clips of your own playlist whose resources you can no longer
read. Clips go when their resource is deleted; private playlists go when their maker's account is removed, shared ones
stay. Agents use them through [MCP](mcp.md) (`list_playlists`, `add_to_playlist`); the web app's side is in
[Playlists](playlists.md).

## views

```
Expand Down
7 changes: 5 additions & 2 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,9 @@ has the same tools, but sees only that namespace, whatever else the person who s
that take a namespace use this one when none is named, `list_namespaces` and `list_files` show only it, and the tools
about the server itself (`file_storage`, `monitoring`, `services`, `mcp_servers`, `sandbox_gateway`, `install_settings`,
`mesh` and the ones that change them) aren't there, even for admins. Nor are the person's own conversations (`list_chats`,
`read_chat`, `rename_chat`, `export_chat`) and projects (`list_projects`, `read_project`, `create_project`,
`update_project`, `schedule_project`), which belong to no namespace. An admin counts as an owner of the namespace there.
`read_chat`, `rename_chat`, `export_chat`), projects (`list_projects`, `read_project`, `create_project`,
`update_project`, `schedule_project`) and playlists (`list_playlists`, `add_to_playlist`), which belong to no namespace.
An admin counts as an owner of the namespace there.

It is off until an owner of the namespace turns it on: **Admin → Namespaces →** the namespace **→ Connect apps**, which
also copies its address, or `PATCH /api/v1/namespaces/<namespace>/integrations` with `{"mcp": true}`, or an agent's
Expand Down Expand Up @@ -97,6 +98,8 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `find_notes` | [notes](notes.md) by words in their title, summary or text, or by where they're filed (PARA), with the words around the match and links |
| `read_note` | a note's Markdown with its links and backlinks, by id or as the page of a thing (`recording:12`) |
| `write_note` | write a note or a thing's page, marked as written by an assistant. Needs a write-scope token and editor access |
| `list_playlists` | the person's [playlists](api.md#playlists), the latest used first, then the ones others shared, each with how many clips they can read in it and its link; with `playlist_id`, its clips in order, each with its resource, times, title, note and a link to that moment |
| `add_to_playlist` | keep a moment of a resource (or a highlight) as a clip with a title and a note, at the end of a playlist: by id, by name (made when there's none of that name) or the one used last (a new one called Clips when there's none). Needs a write-scope token |
| `list_chats` | the person's conversations with Lens's [assistant](assistant.md#chat), newest first, by words in their title, each with its link |
| `read_chat` | one of them: what it draws on, and the version showing (or the one ending at a `message`), each message with its versions, the tools an answer used, its citations with links and its record; `tree` adds every message's parent and role |
| `rename_chat` | name one of them, or have Lens [title](assistant.md#titles) it from how it starts (no `title`). Needs a write-scope token |
Expand Down
48 changes: 48 additions & 0 deletions docs/playlists.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Playlists

Keep a moment of a recording, a video or a document as a clip, with a title and a note, and gather clips from any
resources into a playlist that plays them back to back.

## Keeping a clip

Select words in a transcript and choose **Add to playlist**, or choose the **Add to playlist** button beside a
highlight in its tab. One click: the clip goes at the end of the playlist you used last (a new one called Clips when
you have none), titled with the words you picked or the highlight's label, and the message that says where it went
has **Undo** and a link to the playlist. A clip of a selection runs from the first word to the last; a clip of a
single moment lasts 30 seconds, or to the end of the resource.

Anyone who can read a resource keeps clips of it, for themselves. That's how a clip differs from a
[highlight](api.md#highlights), which an editor makes for everyone who reads the resource. A clip of a highlight
copies its passage and label and stays when the highlight goes.

## Playing a playlist

**Playlists** in the menu (there once you have one; the toast links there before that) lists yours, the latest used first, then the ones others shared. Type a name and press Enter
to make one. A playlist's page plays its clips back to back: **Play all**, or the play button of any clip, then
Previous and Next. Each clip plays from its start to its end on its resource's own audio or video (the same signed link
the recording player uses) and then the next one starts; clips of text without audio are skipped, and **Open here**
opens the resource at the moment playing.

Drag a clip to reorder it, or use its Up and Down buttons; the order is saved at once. **Remove from playlist** takes a
clip out with Undo, which puts it back in its place. Rename a playlist by editing its name, share it with the
**Shared** switch, and delete it with Undo, which makes it again with its clips.

## Who sees what

A playlist is yours and private until you share it. A shared playlist is read by everyone signed in, and each person
gets only the clips of resources they have a role on, whoever made the playlist: a clip of a resource they can't read
isn't shown, nor are its title and note. Only you change your playlists. Clips go when their resource is deleted;
your private playlists go when your account is removed, and shared ones stay.

## Agents

Agents use playlists through the [API](api.md#playlists) and [MCP](mcp.md): `list_playlists` lists them (and one's
clips, with links to their moments), and `add_to_playlist` keeps a moment or a highlight as a clip, in a playlist
named by id or name, or by default the one used last.

## Refine later

- Play clips of text (documents, transcripts without audio) by reading them aloud instead of skipping them.
- Share a playlist with chosen people or a namespace, not only everyone signed in, and give it a public link.
- Export a playlist as one audio or video file, or as a podcast feed.
- Let the assistant in chat list playlists and add clips, as MCP does.
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,7 @@ request, to another check that proves the same thing; loosening a requirement or
| A saved collection is a private podcast feed of its assets with audio or video: one address per person who may read it, listing only what they can read, from the collection's page in one click with Copy, rotated or stopped like a namespace's feed; through the API and MCP | [test_feeds.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_feeds.py) `test_a_collection_feed_lists_its_assets_with_media`, `test_a_collection_feed_token_rotates_and_stops`, `test_agents_get_a_collection_feed`; [collection-feed.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/collection-feed.test.ts) |
| A routine fills a feed on a schedule, or makes a podcast episode about what's new: graph workflow nodes pick the resources or timeline activities since the last run, add them to a feed, or make an episode about them; Try says what they would do | [test_feed_nodes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_feed_nodes.py) `test_a_routine_publishes_new_resources_to_a_feed`, `test_a_routine_publishes_the_timeline_and_text`, `test_a_routine_makes_an_episode_and_the_feed_lists_it_once_made`, `test_try_says_what_it_would_do_and_bad_settings_are_refused` |
| Feeds in the web app: a Feeds page, reached from the Podcasts page and under Explore once there is a feed; New feed with the title focused and Enter making it in the current namespace; a feed's page copies the address in one click, gets a new one, edits what readers show, and takes items out or deletes the feed with Undo instead of a confirm; the publishing nodes on the canvas, checked before Publish | [feeds-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/feeds-ui.test.tsx) `the Feeds page lists each feed with its kind, namespace, items and whether an app reads it`, `New feed opens on the title, defaults to the current namespace, and Enter makes it and opens its page`, `a feed's page copies the address, gets a new one, and takes an item out with Undo`, `deleting a feed is undone by making it again with its items, at a new address`, `editors change what readers show, and Enter in a field saves`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `a Select node into Publish to feed or Podcast episode is a whole graph workflow`, `says what to fix: a feed to choose, the limits, and a graph that publishes nothing`, `names the selection and the feed on the canvas`; [nav-groups.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/nav-groups.test.tsx) `feeds are under Explore, after Podcasts, for everyone`, `Feeds is in the nav only once there is a feed, so a phone's nav stays as short as it was` |
| A moment of any resource kept as a clip (a title, a note) in one click from the transcript or a highlight, in the playlist used last; playlists play clips back to back, reorder and remove with Undo, and each reader gets only the clips of resources they can read; through the API and MCP | [test_playlists.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_playlists.py) `test_playlist_crud_and_clips`, `test_order_reorder_remove_and_undo`, `test_last_used_is_the_default`, `test_readers_get_only_clips_of_resources_they_can_read`, `test_clips_of_highlights`, `test_clips_go_with_their_resource_and_private_playlists_with_their_maker`, `test_agents_list_playlists_and_add_moments`; [playlists-model.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/playlists-model.test.ts); [playlist-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/playlist-page.test.tsx) `reorders with Up and Down and saves the new order`, `removes a clip at once, and Undo puts it back in its place`; [recording-comments.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-comments.test.tsx) `keeps a highlight as a clip in the latest playlist in one click, with Undo`, `makes a first playlist called Clips when there is none`; [nav-groups.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/nav-groups.test.tsx) `Playlists is in the nav only once there is one, under Explore after Feeds` |
| An asset or a collection exported as an evidence PDF, whole or the parts you pick | [test_evidence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_evidence.py) `test_a_collection_as_one_evidence_pdf`, `test_everything_lens_found_goes_in`, `test_picking_the_parts`; [evidence-parts.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/evidence-parts.test.tsx) |

## Automate it
Expand Down
Loading
Loading
0