8000 feat(import): credit WebVTT voice spans as each line's speaker by adeelahmad · Pull Request #689 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
Google accounts connect with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google
Drive can sign in through rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs,
Office files, photos, JSON and zip or tar archives (photos and videos dated by when they were taken, from the file
itself, and photos placed by their GPS), and, with an Apache Tika server (an opt-in extra), documents nothing else
itself, and photos placed by their GPS), and transcripts made elsewhere (SRT, WebVTT, text, Word, PDF), each line
credited to the speaker the file names; and, with an Apache Tika server (an opt-in extra), documents nothing else
here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking
the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off
by default), taking a video's own captions as its transcript when people wrote some (off by default too). Say what a
Expand Down
5 changes: 5 additions & 0 deletions changelog.d/vtt-voices.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **WebVTT voices name the speaker.** A transcript imported from a WebVTT file now credits each line to the speaker its
voice span names, `<v Name>` or `<v.loud Name>` with classes, closed by `</v>` or not, and those speakers join the
namespace's people like any other named speaker. A cue with two voices becomes two lines that share its time; the tags
are left out of the text, and a cue without a voice is read as before. A YouTube caption cue that starts with a line
of one space is no longer lost ([Importing text and transcripts](docs/processing.md#importing-transcripts)).
5 changes: 4 additions & 1 deletion docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,10 @@ and faces (video only), analyze and report. The asset takes the file's fingerpri
same file find it, unless another asset in the namespace has it already.

Speakers are recognised from `Name: text`, `[12:30] Name: text`, `Name (12:30): text`, Otter/Zoom/Teams exports (a
`Name 12:30` line, then what they said), `speaker|emotion|text` lines, and the speakers in SRT/VTT and JSON. Anything
`Name 12:30` line, then what they said), `speaker|emotion|text` lines, and the speakers in SRT/VTT and JSON. In
WebVTT, a cue's voice span names its speaker: `<v Name>`, or `<v.loud Name>` with classes after the `v`, closed by
`</v>` or not. The tags are left out of the text, and a cue with two voices becomes two lines that share its time by
how many words each says; a cue without a voice takes its speaker from a `Name: text` line, or has none. Anything
else becomes paragraphs split into segments of about 40 words with estimated times. Markdown and MDX are reduced to text
first: front matter or the first heading becomes the title; imports, exports, JSX and code blocks are dropped. A PDF
imported as a transcript needs a text layer; import a scan as a document instead, and its pages are read by OCR. Named speakers are matched to the namespace's people by name; generic labels (SPEAKER_00, S1, CH0)
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Import webhooks hand back the PDFs of documents a light server can't convert | [test_import_hooks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_import_hooks.py) `test_a_hook_hands_back_the_pdfs_of_documents_this_server_cant_convert` |
| Connect with Google; Dropbox, OneDrive and Google Drive sign in through rclone on the server | [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_connecting_the_account_in_the_browser`; [test_drive_google.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_drive_google.py) `test_drive_connects_with_google_for_rclone`; [test_rclone_signin.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rclone_signin.py) `test_drive_signs_in_through_rclone_with_no_google_app`, `test_a_browser_on_the_server_reaches_rclone_itself`, `test_onedrive_finds_the_accounts_own_drive` |
| Uploads of audio, PDFs, Office files, photos, JSON and archives | [test_uploads.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_uploads.py) `test_uploading_audio_in_chunks`, `test_json_uploaded_is_read_as_records`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_a_pdf_becomes_a_document_with_its_pages_and_text`, `test_scans_and_images_are_read_by_ocr`; [test_conversion.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_conversion.py) `test_office_and_text_files_become_documents`; [test_anytopdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_anytopdf.py) `test_an_uploaded_zip_keeps_its_files`; [test_archives.py](https://github.com/adeelahmad 8000 /lense/blob/main/fastapi_backend/tests/api/test_archives.py) `test_reading_an_archive`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `import-file`, `import-paste` |
| Transcripts made elsewhere (text, Markdown, Word, PDF, SRT, WebVTT, JSON) imported with each line credited to the speaker the file names, WebVTT voice spans (`<v Name>`, `<v.loud Name>`) included | [test_importing.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_importing.py) `test_formats_and_speakers_by_name`, `test_webvtt_voice_spans_name_each_line` |
| Media browsers can't play (an AVI, HEVC, AMR) gets a copy they can, for the player only, off by default; files they play get none | [test_playable.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_playable.py) `test_a_camera_clip_gets_a_copy_every_browser_plays`, `test_a_file_browsers_play_gets_no_copy` |
| Photos and videos dated by when they were taken, from the file itself (a photo's EXIF date and GPS fix as `place`, a video's creation time) when no name or source dates them; a date already set is kept and none is invented | [test_file_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_file_dates.py) `test_a_photo_is_dated_and_placed_by_its_exif`, `test_a_date_already_set_is_kept`, `test_a_video_is_dated_by_its_creation_time`, `test_a_scanned_video_is_dated_by_its_creation_time` |
| An uploaded video, end to end | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
Expand Down
44 changes: 40 additions & 4 deletions fastapi_backend/app/domain/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -733,6 +733,8 @@ def _generic(s):

PIPE = re.compile(r"^([^|]{1,40})\|([^|]{0,24})\|(.+)$")
SPK = re.compile(r"^(?:\[?((?:\d{1,2}:)?\d{1,2}:\d{2})\]?\s+)?([A-Z][\w .'’-]{0,30}?)\s*:\s+(\S.*)$")
# a WebVTT voice span's start tag: <v Name>, or <v.loud Name> with classes after the v
VOICE = re.compile(r"<v(?:\.[^\s>]*)?(?:\s+([^>]*))?>")
VTT_TIME = re.compile(r"((?:\d+:)?\d{1,2}:\d{2}[.,]\d{1,3})\s*-->\s*((?:\d+:)?\d{1,2}:\d{2}[.,]\d{1,3})")


Expand All @@ -743,20 +745,54 @@ def _clock(s):

def _cues(raw):
segs = []
for block in re.split(r"\n\s*\n", raw.replace("\r", "")):
raw = re.sub(r"(-->[^\n]*\n)[ \t]+\n", r"\1", raw.replace("\r", "")) # YouTube's cues start with a line of one space
for block in re.split(r"\n\s*\n", raw):
lines = [l for l in block.strip().split("\n") if l.strip()]
for i, l in enumerate(lines):
m = VTT_TIME.search(l)
if m:
t0, t1 = _clock(m.group(1)), _clock(m.group(2))
voices = _voices("\n".join(lines[i + 1 :]))
if voices:
segs += _voiced(t0, t1, voices)
break
text = " ".join(lines[i + 1 :]).strip()
sm = SPK.match(text) or re.match(r"^<v\s+([^>]+)>(.*)$", text)
spk, text = (sm.group(sm.lastindex - 1).strip(), sm.group(sm.lastindex).strip()) if sm else (None, text)
sm = SPK.match(text)
spk, text = (sm.group(2).strip(), sm.group(3).strip()) if sm else (None, text)
if text:
segs.append({"t0": _clock(m.group(1)), "t1": _clock(m.group(2)), "text": re.sub(r"<[^>]+>", "", text), "speaker": spk})
segs.append({"t0": t0, "t1": t1, "text": re.sub(r"<[^>]+>", "", text), "speaker": spk})
break
return segs


def _voices(body):
"""A cue's WebVTT voice spans (<v Name>, <v.loud Name>, closed by </v> or the next one) as [(speaker, text)], one
per change of voice; None when it names no voice."""
parts = VOICE.split(body)
if len(parts) == 1:
return None
runs = []
for name, text in [(None, parts[0]), *zip(parts[1::2], parts[2::2])]:
text = " ".join(re.sub(r"<[^>]+>", "", text).split())
spk = (html.unescape(name).strip() or None) if name else None
if not text:
continue
if runs and runs[-1][0] == spk:
runs[-1] = (spk, runs[-1][1] + " " + text)
else:
runs.append((spk, text))
return runs or None


def _voiced(t0, t1, runs):
"""One line per voice in a cue, sharing its time by how many words each says."""
n, w, out = sum(len(x.split()) for _, x in runs), 0, []
for spk, text in runs:
start, w = t0 + (t1 - t0) * w // n, w + len(text.split())
out.append({"t0": start, "t1": t0 + (t1 - t0) * w // n, "text": text, "speaker": spk})
return out


HEADER = re.compile(r"^([A-Z][\w .'’-]{0,40}?)\s*[\(\[]?((?:\d{1,2}:)?\d{1,2}:\d{2})(?:\.\d+)?[\)\]]?\s*:?\s*$")
PAREN = re.compile(r"^([A-Z][\w .'’-]{0,40}?)\s*[\(\[]((?:\d{1,2}:)?\d{1,2}:\d{2})[\)\]]\s*:\s*(\S.*)$")

Expand Down
69 changes: 69 additions & 0 deletions fastapi_backend/tests/domain/test_importing.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,75 @@ def test_paste_and_chunk_stitching(db, cfg):
assert naive * 0.6 < stitched < naive * 0.9


YOUTUBE_VTT = """WEBVTT
Kind: captions
Language: en

00:00:00.160 --> 00:00:02.869 align:start position:0%
\x20
welcome<00:00:00.480><c> back</c><00:00:00.880><c> to</c><00:00:01.040><c> the</c><00:00:01.280><c> channel</c>

00:00:02.869 --> 00:00:05.000 align:start position:0%
today<00:00:03.200><c> we're</c><00:00:03.440><c> looking</c><00:00:03.760><c> at</c><00:00:04.000><c> tides</c>
"""

MEETING_VTT = """WEBVTT

1b2c4d5e-2a7f-4b1c-9d3e-0f6a8b7c5d4e/12-0
00:00:03.120 --> 00:00:06.480
<v Priya Natarajan>Can everyone hear me okay?</v>

1b2c4d5e-2a7f-4b1c-9d3e-0f6a8b7c5d4e/13-0
00:00:06.900 --> 00:00:09.300
<v Tom O'Brien>Yes, loud and clear.</v>

1b2c4d5e-2a7f-4b1c-9d3e-0f6a8b7c5d4e/14-0
00:00:09.300 --> 00:00:12.000
<v Priya Natarajan>Great, let's start with the
budget review.</v>
"""

CLASSED_VTT = """WEBVTT

00:00:00.000 --> 00:00:02.000
<v.loud Esme>It's a blue apple tree!

00:00:02.000 --> 00:00:04.000
<v Mary>No way!
<v.first.loud Esme>Oh <i>yes</i>!

00:00:04.000 --> 00:00:05.000
Nobody named here.
"""


def test_webvtt_voice_spans_name_each_line(db, cfg):
yt = ingest.read_text_transcript(YOUTUBE_VTT, "auto")["segments"] # no voices: as before, the inline timings dropped
assert [(s["speaker"], s["text"], s["t0"], s["t1"]) for s in yt] == [
(None, "welcome back to the channel", 160, 2869),
(None, "today we're looking at tides", 2869, 5000),
]
meeting = ingest.read_text_transcript(MEETING_VTT, "vtt")["segments"] # Teams and Zoom: <v Name>...</v>
assert [(s["speaker"], s["text"]) for s in meeting] == [
("Priya Natarajan", "Can everyone hear me okay?"),
("Tom O'Brien", "Yes, loud and clear."),
("Priya Natarajan", "Great, let's start with the budget review."),
]
classed = ingest.read_text_transcript(CLASSED_VTT, "vtt")["segments"] # classes after the v; two voices in one cue
assert [(s["speaker"], s["text"], s["t0"], s["t1"]) for s in classed] == [
("Esme", "It's a blue apple tree!", 0, 2000),
("Mary", "No way!", 2000, 3000),
("Esme", "Oh yes!", 3000, 4000),
(None, "Nobody named here.", 4000, 5000),
]
srt = "1\n00:00:01,000 --> 00:00:02,500\nAlice: <i>still</i> read from the line\n" # a cue without a voice: unchanged
assert [(s["speaker"], s["text"]) for s in ingest.read_text_transcript(srt, "srt")["segments"]] == [
("Alice", "still read from the line")
]
ingest.import_text(db, cfg, "pods", MEETING_VTT, title="Standup") # the voices become the namespace's people
assert speaker_names(db, "pods") == {"Priya Natarajan", "Tom O'Brien"}


def test_an_engine_that_isnt_installed_falls_back_to_one_that_is(monkeypatch):
"""faster-whisper is there (EXTRAS=whisper), SenseVoice (the default engine) not yet: imports are transcribed anyway."""
made = []
Expand Down
Loading
0