Topic</strong></th>	Singer</strong></th>	Position</strong></th>	Brief Supporting Quote</strong></th></tr></thead>
NATURAL VS. TECHNICAL</strong></td>	Rise Stevens</td>	Less technical thinking is better</td>	“When I did not know about the larynx I was singing better… Some people think so much about all of these technical things that they really mess themselves up”</td></tr>
</td>	Luciano Pavarotti</td>	Years of intense technical work required</td>	“It took me six years of study… never change ideas… the first five or six months it is very depressing”</td></tr>
</td>	John Alexander</td>	Singing is natural like speaking</td>	“Singing is simply sustained speech… I feel almost all vocal problems can be solved by shaping the singing technique to conform to the speaking technique”</td></tr>
MENTAL FOCUS</strong></td>	Birgit Nilsson</td>	Think low for high notes</td>	“The higher the note goes, the lower the support. The support is as low as possible”</td></tr>
</td>	Martina Arroyo</td>	Stay mentally high throughout</td>	“You should stay mentally high throughout the entire scale”</td></tr>
</td>	Luciano Pavarotti</td>	Position never changes, always high</td>	“For me the position never changes… it is high, even when I sing a low note”</td></tr>
PLACEMENT</strong></td>	Cornell MacNeil</td>	Sing backward, placement is nonsense</td>	“Placement is nonsense… utter, complete crap. You sing it backward”</td></tr>
</td>	Anna Moffo</td>	Extreme forward placement</td>	“The place is the most forward, focused sound, complete with overtones”</td></tr>
</td>	James McCracken</td>	Warns against over-focusing on mask</td>	“If somebody decides he’s going to sing in the mask before he knows how to sing… it’s thin. There’s not enough body resonance”</td></tr>
</td>	Placido Domingo</td>	Think way out in front</td>	“My thinking of singing is over there… Way out in front of me… I’m not thinking of placing the voice in the mask”</td></tr>
</td>	Paul Plishka</td>	Everything in the nose</td>	“To me, everything happens here in the nose”</td></tr>
SUPPORT/APPOGGIO</strong></td>	Luciano Pavarotti</td>	Push down, compress against cords</td>	“Appoggiare means… push down the voice… in a way that it doesn’t move anymore…”</td></tr>
</td>	Cornell MacNeil</td>	Floating ribs, diaphragm overrated</td>	“I think the diaphragm is enormously overrated… I think it comes more from the floating ribs in the back”</td></tr>
</td>	Rosa Ponselle</td>	Lean on diaphragm, push out</td>	“Appoggiare… lean on it… Always push out with the stomach and abdomen during the phrase, not in”</td></tr>
</td>	Beverly Sills</td>	Both diaphragm AND larynx</td>	“How does one exist without the other?… I use that ‘sitting on the diaphragm’ to produce the feeling that I want in my throat”</td></tr>
THROAT SPACE</strong></td>	John Alexander</td>	Same space throughout range</td>	“I try to keep as much space as possible, but always the same throughout the entire range of my voice”</td></tr>
</td>	Sherrill Milnes</td>	More space going higher</td>	“More spacing… More lift of the soft palate as you’re going higher, and… more opening in the throat”</td></tr>
</td>	Luciano Pavarotti</td>	Less space in passaggio</td>	“I think I give less space when I go through the passaggio, and then more space after I’ve left it”</td></tr>
</td>	Placido Domingo</td>	Horizontal to vertical shift</td>	“You are having a horizontal sound… to think vertical, in a way… up and down at the same time”</td></tr>
OPEN THROAT</strong></td>	Cornell MacNeil</td>	Meaningless concept</td>	“It doesn’t mean anything to me… I think it’s nonsense!”</td></tr>
</td>	Paul Plishka</td>	No sensation in throat at all</td>	“I don’t know. I know the vocal cords are in the throat, but when I’m singing I have no sensation in my throat at all”</td></tr>
</td>	Magda Olivero</td>	Don’t force larynx down</td>	“If you teach a young student to push down the larynx… goodbye! That he should do automatically”</td></tr>
LARYNX POSITION</strong></td>	Bonaldo Giaiotti</td>	Firmly lower it all the way</td>	“He demonstrated… breathing deeply, which caused his Adam’s apple to descend firmly all the way down”</td></tr>
</td>	Dr. Leo Reckford</td>	Minimal movement only (millimeters)</td>	“The larynx should make minimal excursions up or down… measured in millimeters… It’s definitely wrong!”</td></tr>
</td>	Louis Quilico</td>	Must bring it down independently</td>	“To bring down the larynx is not a natural thing… Yes!” (demonstrated independent up/down jiggling)</td></tr>
</td>	Pablo Elvira</td>	Never pull it down</td>	“The larynx should never be pulled down… You should never feel a down pull”</td></tr>
</td>	Sherrill Milnes</td>	Low larynx equals open throat</td>	“Open throat… low larynx… Those are in a sense the same thing. If your throat is open your larynx is low”</td></tr>
</td>	Cornell MacNeil</td>	Don’t think about it at all</td>	“As soon as you pull something somewhere, you’ve already created a tension”</td></tr>
</td>	James McCracken</td>	Larynx slightly down</td>	“Do we agree that an open throat has to do with the larynx being slightly down? Yep”</td></tr>
</td>	Régine Crespin</td>	Beginning of yawn sensation</td>	“The beginning of the yawn… Be suddenly, quickly astonished… Start breathing by the nose, and be astonished”</td></tr>
PASSAGGIO</strong></td>	Luciano Pavarotti</td>	Squeeze intensely for years</td>	“It took me six years… you become cyanotic, red in the face… you must really make the voice more squeezed”</td></tr>
</td>	John Alexander</td>	Don’t think about it</td>	“The less we think about passaggio, the better off we are”</td></tr>
</td>	Marilyn Horne</td>	Squeeze</td>	“We really tightened up the middle and especially the passaggio. We squeezed on that passaggio”</td></tr> </tbody></table> I spot-checked some entries and confirmed that the quotes I looked at were accurately sourced from the underlying interviews. (I’ve also fixed up some entries manually, and see other problems that I’m not going to bother fixing right now since this is just a quick post.)</p> Here are some specific ways in which the analysis above, and others like it, can be incomplete:</p> Non-exhaustive:</strong> There may be correlated omissions due to bias in the combination of LLM and prompt.</li> Incorrect data:</strong> LLMs generate data that can be hard to verify and the effort required to verify a claim often exceeds the effort required to generate the claim. As Jonathan Swift said, “Falsehood flies, and the Truth comes limping after it.” This can be a big deal</a>.</li> “Wrong” topics:</strong> Topics are proxies for the underlying data and ideally they “carve nature at its joints”. But the appropriateness of topics depends on the task at hand – for example, if you’re interested in the underlying factors that drive agreement or disagreement, the LLM-chosen topics might make it harder or easier to see those factors.</li> </ul> Nonetheless, the level of synthesis here is pretty impressive to me. While the result isn’t of particularly high quality, it nonetheless captures some real patterns from the underlying text that are worth investigating further. A real analysis would take orders of magnitude more time and this exploration at least shows that a table like the above is an interesting way to look at the data.</p> I think there’s a huge amount of potential in this kind of ad-hoc, task-specific dimensionality reduction, particularly for books and other rich source material, where the same knowledge can be projected into many different structures from which it is possible to much more easily make surprising connections.</p> Appendix: Amusing quotes</h3> It’s like Sir Thomas Beecham, who said, ‘Most sopranos sound like they’ve been living on seaweed.’</p> </blockquote> Jerome Hines: “Take away my extraordinary size and physique, take away my handsome face and my glorious voice, and what have you got?”</p> Jan Peerce: “Yeah, what have you got?”</p> Hines: “Jan Peerce.”</p> </blockquote> Louis sang to calm 149 cows and 18 chickens on a cattle boat crossing the Atlantic. When he finished his recital, “there was a breathless hush… not the sound of a single handclap. Cows don’t clap.”</p> </blockquote> LLMs for Nominative Determinism 2025-10-12T20:10:26-04:00 As a long-time fan of names</a> and nominative determinism</a>, I wanted to know how well language models could assess the fit between a person’s name and their chosen profession.</p> To investigate, I gathered a list of 1,000 scientists and asked Gemini</a> to score each one on how strongly their name connected with their chosen field of study.</p> Gemini came up with some obvious names such as Jules Angst</a> the mood researcher, Storm Field</a> the television meteorologist, and Russell Brain, 1st Baron Brain</a>, who studied brains. But it also claimed some amusing and more puzzling connections, insisting that Paul Mockapetris</a>’s name was directly connected to his invention of DNS</a>:</p> The name ‘Mockapetris’ has a phonetic similarity to ‘name’, particularly the ‘mock’ sound, which aligns directly with his invention of the Domain Name System (DNS).</p> </blockquote> His name, ‘Mockapetris’, contains ‘petris’ which sounds similar to ‘petrify’ or ‘petrifies’, evoking a sense of making something permanent or foundational, which aligns with his creation of a core internet system.</p> </blockquote> The name ‘Mockapetris’ has a phonetic resemblance to ‘mocked petris’, which could be interpreted as a playful or abstract reference to ‘mocking’ or manipulating ‘petris’ (perhaps stones or structures, analogous to data structures).</p> </blockquote> Then I realized I’d been querying Google’s smallest model, Gemini Flash Lite, rather than their premier model, Gemini Pro. Oops.</p> A few more amusing ones:</p> Gian Francesco Malfatti</a>, mathematician:</p> The surname ‘Malfatti’ translates from Italian to ‘badly made’ or ‘ill-made’. His most famous work is the ‘Malfatti problem,’ in which he posed the problem of packing three circles of maximal area into a triangle.</p> He conjectured a specific, aesthetically pleasing arrangement of three mutually tangent circles. However, his conjecture was later proven to be incorrect; it is not always the optimal solution.</p> Therefore, Malfatti’s own proposed solution to his namesake problem was, quite literally, ‘badly made’, creating an exceptionally strong and ironic case of nominative determinism.</p> </blockquote> Brian E. Dalrymple</a>, inventor of techniques for fingerprint detection:</p> The surname ‘Dalrymple’ bears a striking phonetic resemblance to the phrase ‘dull rimple’.</p> ‘Rimple’ is an archaic but valid term for a wrinkle or fold, which is an excellent description of the ridges that constitute a fingerprint.</p> Brian Dalrymple’s primary contribution to forensic science was developing techniques to detect latent fingerprints, which are faint, indistinct, or ‘dull’ by nature.</p> Therefore, his name, ‘Dalrymple’, can be interpreted as describing a ‘dull rimple’ or a faint fingerprint, the very subject of his groundbreaking work.</p> </blockquote> Benoit B. Mandelbrot</a> was given a high score, dubiously justified:</p> The name Mandelbrot is of German/Yiddish origin, where ‘Mandel’ means ‘almond’ and ‘brot’ means ‘bread’.</p> The primary object of study for Benoit Mandelbrot was the Mandelbrot set, the most famous of all fractals. The main body of the Mandelbrot set is a cardioid which is distinctly almond-shaped.</p> Therefore, Mandelbrot, the father of fractal geometry, is most famous for studying an ‘almond bread’ set that has an ‘almond’ shape.</p> </blockquote> I’d say instead that Benoit Mandelbrot is a great example of reverse</em> nominative determinism: following his discovery of fractals, he gave himself a recursive middle initial:</p> Benoît B. Mandelbrot (he added the middle initial himself, though it does not stand for a middle name) was born on Nov. 20, 1924, to a Lithuanian Jewish family in Warsaw.</p> — New York Times</a>, October 16, 2010</p> </blockquote> Tech Notes on Podcast Vibes 2025-03-15T19:24:39-04:00 I’m winding down my podcast project and wanted to write a note about some of the things it taught me. In short, all of the major tech choices in this project worked out really well, and I’ll probably use this toolkit to prototype future LLM-based processing pipelines.</p> Go</h2> Go</a> is a great choice for rapid prototyping. I’ve found it particularly useful for side projects since its simplicity makes it easy to context-switch from the other languages I use regularly. Go was a particularly good fit for this project since processing audio with APIs involved making lots of network requests and interfacing the database without doing much compute.</p> Why not Rust? I’ve done side projects in Rust before and while it is a very powerful tool, it has a lot of mental overhead compared Go, and the type system inhibits certain forms of exploratory programming that I find very valuable.</p> Postgres, PGX, and Postico</h2> I used Postgres</a> for the database with PGX</a> as the driver. PGX Top to Bottom</a> is a talk by the creator of PGX which explains the layered structure of the library, and taught me about a few useful API functions like `CollectRows</code></a>,` `BeginFunc</code></a>, and CopyFrom</code></a>.</p>` `This was also my first time trying out Postico</a>, a user interface to Postgres, which I found to be very well-designed. It made it easy to visually inspect the data in my tables and interactively edit schema definitions.</p>` River</h2> One lesson from my years at startups is to never build your own job execution engine if you can avoid it.</p> After looking at various options I decided to try River</a>, an open-source job queueing library, and am very happy I did. It is made by experts with good taste and comes with nice documentation and a well-designed API that exposes convenient abstractions for defining and scheduling jobs. River also comes with a web-based user interface that let me easily monitor execution and inspect job-specific error logs, which made tracking down processing bugs a lot easier since the UI presents the error in the context of other job information, such as its input arguments, total runtime, and number of retries.</p> DeepGram</h2> I used Deepgram</a> to transcribe podcasts. They give you $200 of free credits and the transcriptions are surprisingly good. There are cheaper options, but Deepgram is very convenient.</p> Rather than uploading the audio to Deepgram myself I send them a link to the podcast audio to avoid the extra round-trip. The API endpint I used for this was /listen</a>:</p> https://api.deepgram.com/v1/listen?punctuate=true&paragraphs=true&utterances=true&diarize=true&language=en&summarize=v2&topics=true&model=nova-2</p> By default, DeepGram will train on any audio you upload in return for a pricing discount. This was not a problem for me since the podcasts I was transcribing are already public, but might be a poor tradeoff for other use cases like personal voice notes. If you want, you can opt out</a> by adding a specific query parameter to your request.</p> Claude</h2> Claude did an excellent job at extracting emotionally-laden subjects from the podcast transcripts. The fact that this kind of AI processing is now possible and affordable opens up a lot of opportunities for fun analytics projects.</p> A few caveats:</p> While Claude was able to accurately identify subject-emotion pairs, one thing I don’t have as good a sense of is how many subjects were omitted</em> from the extraction since I only did a limited number of spot checks and never created a manually-constructed dataset for comparison, relying instead on my intuitive sense of whether the extracted data was reasonable.</p> I noticed was that Claude would produce a similar number of results regardless of the length of the provided transcript snippet. I worked around this by feeding it the transcript in small overlapping chunks and de-duplicated the results using a vector similarity heuristic.</p> Claude’s citation feature</a> launched while I was working on this project and turned out to be a great fit for connecting the emotional inferences back to the supporting text.</p> Previous posts in this series:</p> Note: Podcast Vibes</a></li> Podcast Vibes Prototyping</a></li> Podcast Vibes Presentation</a></li> </ul> Podcast Vibes Presentation 2025-02-27T21:21:18-05:00 I gave a talk today about my podcast analysis project (previously: Note: Podcast Vibes</a>, Podcast Vibes Prototyping</a>), connecting it to some visualization design work from over a decade ago which explored ways to visualize and organize text corpora with vector-based embedding models.</p> I’ve put annotated slides up here</a>.</p> This was in 2013 when I was working at an small machine learning startup spun out of the MIT Media Lab, and AI research was just beginning to be taken over by deep learning. I remember being amused when a team member referred to this phenomenon as deep lemming</em>.</p> Our main visualization was called a concept cloud</em> and it used semantic vectors to make a more meaningful word cloud visualization, in which not only the size but also position & color of the words was used to convey the structure of conceptual relationships in the underlying text.</p> </p> This work was done in collaboration with many people on the team, including Elia Robyn Lake</a>, Jason Alonso, Ken Arnold</a>, Avril Kenney, Christina Laverentz, Alice Kaanta, and Andrew Lin.</p> Writing about this brought back memories of the infamous Ass Headache Problem</em>, and the time when our stemming pipeline thought that “Emily” was an adverb and “Coca Cola” was the plural of a singular “Coca Colon”…</p> Language modeling has come a long way.</p> XOR Doubly-Linked List 2025-02-06T23:35:13-05:00 A neat idea I learned today:</p> You can make a doubly-linked list that only needs to store a single pointer – the xor</code> of the prev</code> and next</code> pointers – to allow traversal of the list from either direction.</p> When moving forward we xor</code> current ptr with prev</code>. Moving backward xor</code> with next</code> to get prev</code>. That it. How do we get prev</code>/next</code> for xor</code>? The only way to get in the middle of the list is using iterator. Which can hold one extra pointer. For the first/last nodes extra pointer is NULL</code>.</p> — numba888 on Hacker News</a></p> </blockquote> This concept has a Wikipedia page</a>.</p> Podcast Vibes Prototyping 2025-02-02T21:39:36-05:00 I’m considering turning my little research note on podcast vibes</a> into a bit more of a product. The idea would be to track the daily pulse of the podcast ecosystem in terms of the subjects it discusses and the emotions it puts out.</p> This weekend I played around with a few visualization ideas for looking at the emotional subjects for a single episode at a time.</p> The picture below shows all of the subjects discussed on a recent podcast interview with a bean enthusiast</a>, with subjects discussed positively colored green and those discussed negatively colored red.</p> </p> The annotations are picked at regular samples across the emotional spectrum so that the labels don’t overlap. I think it would be better to instead draw attention to the most salient discussion points.</p> Here’s how the same kind of picture looks for a longer podcast episode:</p> </p> And here’s one for a podcast with a much more negative slant. Amusingly, the most positive segments of this one are the ads. While looking at the transcript DeepSeek R1 complimented the host on his ability to smoothly and undetectably transition from the interview into an advertising segment.</p> </h1> Here’s how it might look to lay out the dots on a horizontal timeline, with negatives segmented from positives.</p> </p> Anyway, just some early explorations. Still getting used to publishing this stuff in such an early state, but here we go.</p> Note: Podcast Vibes 2025-01-26T23:02:05-05:00 Here’s a research note</a> on my recent explorations into using LLMs to analyze the emotional content of podcasts.</p> I’m amazed that we have technology that can do this so well.</p> This project supports my view that one of the most useful features of language models is that they can function as transducers from messy reality to a structured format that can then be dealt with using standard tools that are more efficient, controllable, and well-understood.</p> It’s also a reminder of just how open the design space is for projects like this, how valuable good tooling for data pipelines is, and the importance of taste.</p> The Structure of the Wavelet Tree 2025-01-21T15:43:37-05:00 This is a follow-up to Wavelet Matrix Construction</a>.</p> (I’m working on a larger post about the wavelet matrix and writing and publishing extremely rough drafts as I go.)</p> To understand the wavelet matrix it can be useful to understand the wavelet tree first, so here’s a quick primer.</p> As I’ve mentioned before, I think of the wavelet tree (and matrix) as representing the process of sorting</em> a sequence of integer symbols.</p> Let’s take an example. Imagine that our integer symbols are picked from a universe of size 2³, consisting of the numbers from 0 to 7.</p> The wavelet tree represents a quicksort</a> whose pivots are all powers of two. At each level of recursion, the value space is partitioned into two halves based on the universe size, ie. into a lower half containing the numbers 0 to 3, and an upper half containing the numbers 4 to 7. After this first step, each half is recursively sorted using the same approach, terminating when the halves contain a single integer each.</p> You can see this in the diagram below, where each node is labeled with the range of the as-yet-unsorted integers inside it.</p> Each level of the diagram corresponds to the next stage of the sorting process. The full sequence of numbers starts in its original order on the top level, made up of symbols from 0 to 7. On the next level those numbers are partitioned into two sublists, with the one on the left containing all of the numbers from 0 to 3 in their original order, and the one on the right containing all numbers from 4 to 7, in their original order.</p> Then the same process happens again, with the left (0 to 3) node further partitioned into two child nodes, one from 0 to 1 and another from 2 to 3. Then we do this again and arrive at the fully-sorted sequence on the bottom level.</p> </p> The diagram above shows the process abstractly without any particular sequence of numbers, but let’s see how this would actually work in practice. Here’s a visualization that shows the incremental sorting process for the sequence 5 2 1 3 7 2 6 1 5 2</code>:</p> </p> We start off with the full sequence in its original order on the top layer, and end up with those same numbers sorted in ascending order on the bottom. In between, you see the process of recursive partitioning, where the numbers “slide” down either the left or the right arrow, depending on whether they belong to the bottom or the top half of the universe.</p> Now, because the entire sequence is represented in full on each level of the sort, you might think this is a very expensive way to represent the data. But in fact, this is the beautiful trick: the wavelet matrix stores not the boxes, but the arrows!</p> Here’s the same diagram again, this time with the numbers in parentheses notating the data that actually gets written into the wavelet matrix as individual 0- and 1-bits.</p> The drawing tool</a> I’m using does not allow me to color the individual zeros and ones red or blue but you should think of the the zeros as colored blue, indicating that the number goes down and to the left along the blue arrow, and ones as red, indicating that the number goes down and to the right following the red arrow. </p> The data in the wavelet matrix consists of a bit vector per level containing the bits in parentheses, With the one small tweak that, rather than writing the bits down from left to right as they’re shown here, we first write down the bits for all the left nodes, followed by the bits for all the right nodes.</p> If those bits are all you’ve got it might look like we’ve scrambled the numbers beyond all possibility of repair. But we will soon see that it’s easy to reconstruct the original data from this new representation.</p> The reason for this is that at each level we’ve actually written out bits from the numbers themselves. For example, the number 5 is represented in binary as 101</code>, and those are the exact bits written out for each 5 in the sequence, in that order – a 1-bit on the top layer, a 0-bit on the middle layer, and another 1-bit on the bottom layer.</p> (I’d like to make an animated version of this diagram later, but for now I’m happy since this is already quite an improvement from my last post.)</p> More to come.</p> Appendix: Scraps</h3> Here are some scraps of visualizations I’m working on that are quick sketches to see whether an idea works. This first one has some promise: </p> This tree view is an interesting idea, too, and inspired the boxy diagrams below:</p> </p> The next few are a bit misleading because it is not the case that the bottom half of each node sorts to the left:</p> </p> I experimented with showing the flow from the full parent node, but that seems even less enlightening, and I think the arrows in the main post are better illustrations of the idea.</p> </p> Wavelet Matrix Construction 2025-01-20T18:32:05-05:00 This is a follow-up post to Enter the Wavelet Matrix</a>. Compared to my previous entries this one is much more of a stream-of-consciousness that’s not really designed for anyone else to read (though hopefully on the way to a much more coherent and comprehensive explanation!)</p> Sidenote:</em> This is very much an early draft, and not really understandable on its own, but I’m practicing the art of forging ahead with partial work.</p> The wavelet matrix can be constructed using an extremely simple algorithm, though explaining the meaning of the resulting structure will take a little bit more work.</p> The wavelet matrix is a way to represent a sequence of integers, which we will call symbols</em>, by representing a sorting process – the symbols are incrementally sorted by their successive bits, from high to low, using a stable bucket sort, and the incremental results are written down into the wavelet matrix structure, and later used for reconstituting the elements or performing other algorithms such as ranged quantile queries.</p> Here’s an image illustrating the construction of a wavelet matrix for the sequence 1 4 2 3 7</code>. The sequence starts out in its original order on the top layer and moves down incrementally through a series of bitwise bucket sorts that sort the elements into ‘left’ and ’right buckets based on whether the element has a 0-bit or 1-bit on that level. The</p> </p> The big bold numbers are the symbols</li> The gray numbers underneath are their bit encodings</li> This diagram represents three successive applications of a bucket sort, from high bits to low bits. The unsorted elements on the first row rearrange themselves until they’re in their final positions on the bottom row.</li> This visualization will need to be improved in many ways, but I’m trying to get better at publishing drafts, so here we are. Don’t re-use colors between the numbers and their bits, and the links which represent the data encoded by the matrix</li> Consider introducing a version of this that sorts according to the wavelet tree rule</li> </ul> </li> </ul> The data stored in the wavelet matrix is represented by the “pipes” in between the numbers rather than the numbers themselves.</p> This is what I mean when I say the wavelet matrix represents the process of sorting. It records whether each element “went left” or “went right” in the successive applications of the bucket sort.</p> The idea behind this picture is that all elements that go left appear before the elements that go right. So for example, the numbers 1, 2, and 3 have a zero for their top bit, and therefore appear on the left of the second row.</p> The numbers 4 and 7 have a 1 in that top bit, and therefore go right and appear at the end. The numbers that went left are indicated by gray pipes, and the ones right are indicated by black pipes.</p> The next sorting is based purely on the second bit. The numbers 2 and 3 have a 1 in that position, so they go right.</p> The same process occurs again on the third layer. Note how 7 was the maximal element on each row indicated and therefore went right on each level.</p> This is my sixth attempt to try to explain this process, and clearly it’s not working very well, but I will forge ahead, because I think this section does capture something about the practical algorithm I use for large-alphabet wavelet matrices.</p> I’m beginning to think it’s probably best to start with an introduction to the wavelet tree where the effects of the sort are much more salient, since the bottom row ends up with the symbols sorted by value. In the wavelet matrix, the symbols end up sorted by the bit-reversal of their value.</p> Actually…</p> Here’s a visualization of the wavelet tree representation of the sequence</p> 7 6 5 4 3 2 1 </code></pre> </p> The wavelet tree has a nice hierarchical structure where all of the children stay underneath their parent node, though this is not obvious from these matrix-style visualizations that lose the node borders.</p> And here’s the wavelet matrix version:</p> </p> Here we can see that each layer is sorted independently of the previous – ie. other than the stability of the sort, whether the elements on each layer go either left or right purely based on the bit at that level, rather than taking into account the previous levels too (which is what gives the preceding picture its tree structure).</p> As a side note, the reason the wavelet matrix is an improvement over the wavelet tree is because the wavelet tree requires you to store the node offsets at each level, which scales linearly with the size of the maximum symbol. Without this information, it becomes impossible to navigate from one level of the wavelet tree to the next.</p> The wavelet matrix is based on this ingenious idea of ordering all of the left children before all of the right children on each level, rather than keeping the child nodes “underneath” the parent. This helps because when you’re on a particular level you can compute the number of left children that precede you and the number of right children that precede you with a simple rank query on the bit vector holding that level. Therefore, navigating to a left child involves looking at the number of zeros to your left, and jumping to that location on the next level. Navigating to the right child involves looking at the number of ones to your left and jumping to that location on the next level (plus the total number of zeros, ie. left children, since all the left children precede the right children).</p> It’s very clever but hard to explain. Hopefully I can find a nice way to present all of this.</p> References</h2> See Translating Between Wavelet Tree and Wavelet Matrix Construction</a> for more details on the relationship between the wavelet tree and wavelet matrix, including a discussion of role bit reversal plays in the translation.</li> See Practical wavelet tree construction</a> (2021) for description of the bucket sort algorithm visualized above (page 31).</li> </ul> Enter the Wavelet Matrix 2025-01-15T20:32:08-05:00 Imagine you have an array of integers and want to preprocess it so that you can answer quantile queries, such as computing the median or the 99th percentile, in constant time.</p> One possible solution is to sort the array and then answer the quantile queries by looking up the relevant entries in the array.</p> But what if, instead, I asked you to preprocess the array for range</em> quantile queries, where rather than computing the quantile over the entire array, you want to compute quantiles for any sub-slice arr[i..j]</code> in constant time?</p> Computing range quantiles involves combining information from both the original order and the sorted order, so sorting the entire sequence in advance no longer suffices. And the quickselect</a> algorithm, which is often used for computing specific quantiles directly from an unsorted array, does not help us either, since rather than being constant-time its runtime depends on the length of the slice of the array.</p> Enter the wavelet matrix, a data structure that represents the process</em> of sorting.</p> The wavelet matrix reorganizes the data in the array in a way that allows answering range quantile queries, and a lot more, in constant time:</p> You can access any element of the original sequence in its original order in constant time.</li> You can access any element of any slice of the array in sorted</em> order in constant time.</li> You can count the number of times any element occurs in any slice of the array in constant time, which is known as a rank query.</li> You can count the number of times all elements occur in any slice of the array in constant time with respect to the length of the slice, with the runtime scaling with the number of unique elements that are counted.</li> You can find the location of any occurrence of any element of the array in close-to-constant time, which is known as a select query.</li> You can find all of the elements that occur more than X</em>% of the time, in any slice of the array, in an amount of time that depends only on X</em> and not the length of the slice. This is known as a k-majority query.</li> </ul> This all sounds very attractive, but there are some trade-offs.</p> The main downsides are that the data representation has some space overhead, on the order of 10-20% of the space required to store the bits of the integers themselves, and a high constant time factor due to the distributed nature of the data representation. </p> Internally, the wavelet matrix is composed of an array of bit vectors, each representing the bit planes of the original integers, which means that the bits of any specific element are scattered throughout memory and need to be reassembled as queries are executed, which hurts performance due to a lack of data locality and somewhat unpredictable access patterns, which can be somewhat alleviated by batching.</p> I’d like to explain more about the mechanics of how this thing works in the future, but for now there’s a great paper written by Gonzalo Navarro, one of the co-discoverers of the wavelet matrix, which serves as a great introduction of the basic idea along with a whirlwind of the literature around it. The article is called Wavelet Trees for All</a> because the wavelet matrix is actually an optimized data representation for a conceptual tree structure called the wavelet tree. The word wavelet</em> hints at the fact that the integers are decomposed into something like their frequency components, but no actual wavelets are used in the process of making a wavelet tree.</p> I’ll leave you with a picture I made a while back of an algorithm that constructs a wavelet tree through a series of incremental stable sorts. </p> Further reading</h2> A nice description of wavelet trees</a> by Alex Bowe</li> Original wavelet matrix paper</a> by Francisco Claude and Gonzalo Navarro</li> Paper: Practical Wavelet Tree Construction</a></li> Paper: New algorithms on wavelet trees and applications to information retrieval</a></li> Paper: Range Quantile Queries: Another Virtue of Wavelet Trees</a></li> </ul> Gilbert Curve 2025-01-10T17:03:28-05:00 The Hilbert Curve</a> is space-filling fractal that, in the limit, will fill any power-of-two-sized square with an increasingly dense squiggle that fills the plane.</p> The Gilbert Curve</a> is a generalization of the Hilbert curve to rectangular spaces. The main code is in Python but there are also implementations in JavaScript</a> and C</a>.</p> </p> Related: A very fast C++ Hilbert curve implementation</a> in 2D and 3D (details here</a> and here</a>).</p> LLMs Make Context More Valuable 2024-12-31T16:49:48-05:00 Reader-side LLMs enable information filtering and synthesis to be done on behalf of the reader.</p> This increases the value of curated contextual information, which can now be sifted through efficiently, and automatically integrated with the main text.</p> LLMs have issues with provenance and reliability, though, which is why I’m interested in author-provided context, which not only serves as a trust signal but also tames hallucinations, making catastrophic failure less likely.</p> Existing media formats have lightweight interaction mechanisms such as footnotes and annotations which augment a text with additional information. However, these interactions are usually limited to a small set of pre-determined annotations and footnotes, and the author likely had ideas they decided were not worth incorporating into the final work.</p> Similarly, commented code only contains a fragment of its relevant context and rarely explains every design decision. Some design decisions can be inferred from the surrounding context of the code itself, but other information cannot be inferred, and has to come from knowledge of the broader system and its environment.</p> I wonder what might be a good testing ground for these ideas. One thought is that large programming projects often have public design discussions about the broader tradeoffs involved in particular architectural or feature decisions. And as far as other kinds of writing, the Bible is probably one of the most analyzed and annotated</a> pieces of text in human history, though that one’s a bit out of my area of expertise…</p> See also: Advanced Essay</a></p> Splash Color Format 2024-12-25T17:00:51-05:00 Splash</a> is a color format that reduces the space of choices in a playful way and lends itself to interesting UI design possibilities.</p> Each color is specified as a three-digit number with one digit per color channel, for a total of one thousand possible colors. For example, 407</code> stands for 4/9</span> units of red, 0/9</span> units of green, and 7/9</span> units of blue.</p> Voice Note Browser 2024-12-24T19:37:05-05:00 I like thinking through ideas on a walk and sometimes record voice notes to myself as I go. The result is a collection of related notes, and I wanted a way to see their transcripts together on one page. But to my surprise, couldn’t find any apps that show more than a single transcript at a time.</p> So I built a little web app to do this using the newly-added transcripts</a> in Apple’s Voice Memos, showing the transcripts from the couple of days’ worth of notes in an simple list.</p> I used an Apple Shortcut to copy out the most recent week of voice memos. I originally wanted to use rsync</code>, but ran into permission issues and did not feel sufficiently bold to give my terminal app “Full Disk Access”.</li> </ul> </li> I used SvelteKit to make the app. The transcript is stored as JSON alongside the audio in a metadata section. Here’s how I parsed it, treating the audio file contents as a utf-8 string:const jsonStart = text.indexOf(`{"attributedString"`); if (jsonStart > -1) { const jsonEnd = text.indexOf("\x00", jsonStart); const slice = text.slice(jsonStart, jsonEnd); const transcript = JSON.parse(slice); } </code></pre> </li> I learned that SvelteKit can stream promises</a>, making it easy to load individual voice notes asynchronously and incrementally. The index page returns an array of Promise</code>s representing individual notes, and the frontend shows data as it arrives. This was very cool.</li> </ul> </li> </ul> This came together quickly and I’m really happy with it – it’s nice when something so simple can be so useful.</p> A few ideas for future work:</p> Provenance: Make it easy to listen to the underlying audio for the cases when the transcription doesn’t get things right. For example, making a highlight on the page should pop up a little tooltip that lets you listen to the audio corresponding to that part of the transcript.</li> </ul> </li> Structured extraction: Use LLMs to extract to-dos so that the ideas are easier for me to act upon. It would be nice if there was a sidebar with a bunch of to-do items next to each note.</li> There might be other kinds of structured extraction that would also be useful. I could imagine a set of local language models making parallel passes over each note with each focused on extracting a particular kind of structure.</li> </ul> </li> </ul> Hyparquet 2024-12-18T21:22:19-05:00 I came across a nice dependency-free library today, hyparquet</a>, for parsing Parquet files in JavaScript.</p> It supports many compression formats</a> including Snappy and Zstd, and reads Parquet files in chunks using range requests</a> rather than loading the entire file. You can read just the metadata, or particular rows and columns.</p> It’s a good companion to Jeff Heer’s flechette</a>, a lightweight library for reading and writing Arrow files. (API reference</a>)</p> See also: fzstd</a>, a small library for decompressing zstd files.</p> Advanced Essay 2024-12-15T22:07:32-05:00 In the 1970s, decades before computers began to compete at the highest levels of chess, a new variant of the game was invented, called Advanced Chess</a>. The idea was that instead of a human player competing against a computer, the human and machine would play together on the same side.</p> Rust-analyzer can expand macros 2024-12-13T13:10:12-05:00 If you use an editor with Rust’s LSP integration</a>, you can put your cursor on a particular derive</code> macro, such as Clone</code> in #[derive(Clone)]</code>, or an the name of a macro application, such as matches!</code> in matches(foo, bar)</code>, and select “Expand Macros Recursively”, which will open up a side buffer showing the expanded code from that macro.</p> This is extraordinarily useful when trying to understand what code is generated from a derive, or when debugging an issue with your own macros.</p> I discovered this through a comment made by David Barsky</a> in the Rust Zulip</a>.</p> Formatting Code with a Git Hook 2024-12-11T18:21:38-05:00 Here’s a git pre-commit hook</a> to auto-format code on commit. It’s useful if you’re working with code whose formatting guidelines differ from the ones you’ve configured in your code editor.</p> #!/bin/bash # .git/hooks/pre-commit # Store list of staged files that match your target pattern. # The `git diff` command returns a list of staged files. files=$(git diff --cached --name-only \| grep -E '\.(js\|ts\|svelte)$') if [ -n "$files" ]; then pnpm format # Add all of the previously staged files back to the staging area # to check in the formatted files. git add $files fi # Exit with an error when there are no longer any staged changes, # since otherwise git will create an empty commit. if git diff --cached --quiet; then echo "Error: After formatting, there are no longer any files are staged for commit" >&2 exit 1 fi </code></pre> Quicker Netlify Deploys 2024-12-08T11:16:20-05:00 I use Netlify</a> to host small static sites like this one. It does its job well and is very convenient to use.</p> But I noticed that since the standard way of setting things up is to link Netlify directly with your Git repo, deploys could take a while. When you push your code, Netlify builds the site on their own infrastructure before deploying it.</p> I recently came across</a> a very neat alternative where you can build your site locally and deploy it to Netlify directly with a single HTTP request. This is really convenient when you don’t need the overhead that comes with more careful deploy management.</p> The link above does a good job of describing the process, but the upshot is that the following command is all you need to have a deploy running live just a few seconds later.</p> curl -H "Content-Type: application/zip" \ -H "Authorization: Bearer PERSONAL_ACCESS_TOKEN_FROM_STEP_1" \ --data-binary "@FILE_NAME.zip" \ https://api.netlify.com/api/v1/sites/SITE_NAME.netlify.com/deploys </code></pre> Right now I use it like this (these build commands go in a justfile</code></a>) to ensure that there’s a corresponding commit for every deployed version:</p> pub: no-uncommitted-changes build @rm -f dist.zip @zip -q -r dist.zip dist; @curl -H "Content-Type: application/zip" \ -H "Authorization: Bearer $(cat .netlify-token)" \ --data-binary "@dist.zip" \ https://api.netlify.com/api/v1/sites/what.yuri.is/deploys \| jq @rm -f dist.zip @echo "" @echo "Deployed!" # Halts with an error if the repository contains uncommitted changes no-uncommitted-changes: @ git diff --exit-code > /dev/null \|\| (echo "Please commit changes to the following files before proceeding:" && git status --short) build: zola build </code></pre> Pareto Frontier 2024-10-25T22:21:38-04:00 Status: This was written in a hurry. I should find some way to label these with how much effort went into them / how much attention should be put on them…</em></p> The Pareto frontier can help you find a subset of your data points that are the ones that represent trade-offs between maximizing the various dimensions.</p> Imagine you’re looking at data about thousands of cars, and for each car you know its cost, mileage, and top speed. Maybe you’re trying to get a general sense of which cars perform the best, but this can be difficult when there are multiple attributes along which they can be compared.</p> All else being equal, though, you want a lower cost, higher mileage, and higher top speed, and it could be the case that some cars are strictly better than others on all three of these aspects. That set of those points is called the Pareto frontier</a>.</p> The Pareto frontier will reveal the points that represent interesting trade-offs: no point on the frontier is strictly better than any other. Of course, there may be important real-world dimensions not captured in your data – perhaps you also care about the size of the car, for example – and in that case, looking at only the frontier of cost, mileage, and top speed can be a dangerous thing since it would cause you to ignore potentially relevant points along those other dimensions or skew your data (since size is correlated with all three dimensions). But if you can quantify these “hidden” aspects and add them to your data, the new higher-dimensional frontier might prove to be a source of insight.</p> Implementation</h2> Here’s a simple JavaScript implementation of a Pareto frontier algorithm from the paper Fast Linear Expected-Time Algorithms for Computing Maxima and Convex Hulls.</a>, coauthored by Jon Bentley</a> of Programming Pearls fame.</p> It takes an array of points, each represented by an array, and returns the indices of the points that are on the “max” Pareto frontier (intuitively, these points are generally “strictly larger” than the others when considering all dimensions).</p> // Return the indices of points on the max Pareto frontier (maxima of a point set) using algorithm // M3 from "Fast Linear Expected-Time Algorithms for Computing Maxima and Convex Hulls". // Note that if multiple equal points are potentially on the frontier, only one will be returned. function paretoFrontier(points) { const n = points.length; if (n === 0) return []; let topMax = 0; const max = [0]; for (let i = 1; i < n; i++) { let j = 0; while (j <= topMax) { if (dominates(points[max[j]], points[i])) { // move max[j] to front of max[1..j] const [move] = max.splice(j, 1); max.unshift(move); j = topMax + 2; } else if (dominates(points[i], points[max[j]])) { // move max[j] to max[topMax] const [move] = max.splice(j, 1); max.splice(topMax, 0, move); topMax -= 1; } else if (equals(points[i], points[max[j]])) { j = topMax + 2; } else { // pts i, max[k] incomparable j += 1; } } if (j === topMax + 1) { topMax += 1; max[topMax] = i; } } return max.slice(0, topMax + 1); } function equals(p1, p2) { const k = p1.length; for (let i = 0; i < k; i++) { if (p1[i] !== p2[i]) return false; } return true; } // p1 dominates p2 if it is greater than it in some components and equal // in the remainder (incomparable in no components). Allows for dimensions // s. t. neither p1 nor p2 is less than the other, eg. if the values // are distinct Objects. function dominates(p1, p2) { const k = p1.length; let i = 0; while (i < k && p1[i] === p2[i]) i++; if (i === k) return false; // equal in all dimensions for (; i < k; i++) { // if p2 is greater than p1 in this dimension, then p1 does not dominate p2. if (p2[i] > p1[i]) return false; // if p2 is incomparable to p1 in this dimension, the p1 does not dominate p2. if (!(p2[i] < p1[i] \|\| p2[i] === p1[i])) return false; } return true; // p1 dominates p2 } </code></pre> Cheerio for Web Scraping 2024-10-13T19:22:53-04:00 A great JavaScript library has recently reached 1.0: Cheerio</a>. It gives you a jQuery-like syntax for quickly parsing out content from an HTML string.</p> I recently used it to parse out some data from a structured list using a nice high-level DSL that’s used by Cheerio’s extract</a> function.</p> const $ = cheerio.load(htmlContent); // The page has many of these: // <div class="ts-segment"> // <span class="ts-name">Jimmy Wales</span> // <span class="ts-timestamp"><a href="https://youtube.com/watch?v=XXXX&t=5112">(01:23:45)</a> </span> // <span class="ts-text">Hmm?</span> // </div> const result = $.extract({ segments: [{ selector: '.ts-segment', value: { name: '.ts-name', timestamp: { selector: '.ts-timestamp', }, href: { selector: '.ts-timestamp a', value: 'href' }, text: '.ts-text', }, }] }); </code></pre> The input HTML content contains many snippets like the one in the comment above. One nice thing about Cheerio’s extract function is that if parts of a composite value are missing, then the entries in each segment will remain undefined while the existing values are extracted. For example, if the name</code> is missing from a particular segment and its selector fails to match, then its timestamp</code>, href</code>, and text</code> will still be extracted.</p> Running the extraction will return a JavaScript object with the key segments</code>, whose value is an array of objects each with a name</code>, timestamp</code>, href</code>, and text</code>, mirroring the shape of the input.</p> One nice thing about this interface is its flexibility, which allows you to extract arbitrary properties at multiple levels of a nested tree-like query. The docs</a> for the function aren’t very informative as to what can be extracted but this tutorial</a> explains many of the possibilities, including using a function as the extractor.</p> Visualizing Sentences with Finite State Automata 2024-10-07T13:10:38-04:00 The National Electronic Injury Surveillance System</a> in the US collects data on product-related injuries. It has a dataset of emergency room visits, with granular information on over 8.5 million visits across 100 hospitals going back over 20 years.</p> An interesting feature of the data is that each ER visit comes with a narrative story. For example:</p> A 50YOF DROPPED A CRYSTAL BALL ON FOOT, SPRAINED FOOT</p> </blockquote> PATIENT ROLLED OFF BED, HIT EAR ON DRESSER OR ON WOODEN KNOB SHAPED LIKE AN ELEPHANT HEAD AT HOME; EAR LAC.</p> </blockquote> 2 NIGHTS AGO A FRIEND MICROWAVED A MARSHMALLOW AND PUT IT ON PT’S NECKPT HAS BURN TO NECK WITH BLISTERING</p> </blockquote> One way to get a qualitative sense of what’s in this dataset is to do some thematic keyword searches. For example, for bicycles:</p> PT FELL OFF BICYCLE AND HIT HEAD</p> </blockquote> PT RIDING BICYCLE AND FELL AND LAC ARM</p> </blockquote> 13YOBF FELL OFF BICYCLE;RADIAL HEAD FX</p> </blockquote> Avocados:</p> PT CUTTING AN AVOCADO AND LAC HAND ON KNIFE</p> </blockquote> LAC FINGER ON KNIFE CUTTING AVOCADO</p> </blockquote> CUT WITH KNIFE TO FINGER WHILE CUTTING AVOCADOSDX: LAC FINGER PT</p> </blockquote> Cowboy boots:</p> PT DANCING IN COWBOY BOOTS STRAINED RIGHT ANKLE</p> </blockquote> PT. HAS FOOT DISFIGUREMENT FROM CHRONIC COWBOY BOOT WEARING. DX:FOOT DISFIGUREMENT</p> </blockquote> PATIENT WAS ACCIDENTALLY KICKED IN RIBS PLAYING WITH FRIENDS WHOWORE COWBOY BOOTS. DX. RIGHT RIB CAGE STERNAL SOFT TISSUE.</p> </blockquote> Banana peels:</p> WALKING DOWN HANDICAP RAMP CARRYING BOXES AND STEPPED ON A BANANA PEELAND FELL STRAINING KNEE</p> </blockquote> 32 YO FEMALE SLIPPED ON BANANA PEEL AND FELL TO FLOORKNEE STRAIN</p> </blockquote> PT SLIPPED ON BANANA PEEL AND FELL ON FLOOR IN STORE</p> </blockquote> I wanted to visually summarize the variation within sets of similar stories like these, and after tinkering a bit realized that this problem was very similar to a collaboration</a> I was a part of a while ago that focused on visualizing variations across a set of lede sentences in newspaper articles. One of the goals there was to reveal authorial/media bias, which led us to stay close to the surface texts since phrase ordering and word choice are part of how bias is expressed. In the ER visit stories, though, the texts are written in medicalese and the event semantics matter more than surface syntax, so I think it would be fine to dramatically restructure the text if doing so reveals interesting higher-level patterns. It’s also possible that visualizing something like the temporal sequence of events and their consequences directly will wind up being a good idea here.</p> The story so far</h2> After getting the dataset into shape I spent some time learning about minimal finite state acceptors, which I thought could be a mechanism for merging similar sentence fragments together. FSAs compactly represent a set of strings by reusing suffixes wherever possible:</p> Input: Yuri accidentally went to the store Ruby deliberately went to the store Output: Yuri accidentally ⤵ went to the store Ruby deliberately ⤴ </code></pre> This post</a> by Andrew Gallant has a great overview of FSA construction and usage. In the section on FSA construction</a> Andrew visualizes the algorithm for constructing a minimum finite-state acceptor for a given set of strings. I implemented the algorithm described in the post, which comes from a paper called Incremental Construction of Minimal Acyclic Finite State Automata</a> , and tried it out on some of the stories from the emergency room data.</p> My plan was to see whether FSAs could be used to capture the variation in a curated set of sentences based on a search term, such AVOCADO or BOOTS.</p> An issue that came up is that, since each node in an FSA represents a fixed set of strings, finite state automata will only compress strings if they share a suffix. For example, the following two sentences cannot be compressed using an FSA because they differ in their last word:</p> Yuri accidentally went to the bar Ruby deliberately went to the store </code></pre> One way around this is to focus the compression on a local neighborhood around the word of interest (eg. WENT above) and to build two automata: one for the prefix leading up to the word, and another for the reversed suffix of the word. The reversal of the suffix makes it so that the “end” of the FSA is closest to the word of interest, which often leads to a more compact automaton.</p> For example, here’s a pair of FSAs for BOOTS, visualizing local neighborhoods across 10 sentences: </p> Here’s BOOTS with 20 sentences: </p> And here’s BICYCLE with 25 sentences: </p> It feels like there’s a glimmer of promise here, but I need to do some more thinking about how to proceed.</p> One idea is to use LLMs to conform the sentences to a more standard structure, then build automata for each “semantic fragment” of the sentence, and visualize the result using something like a parallel coordinates plot with ordinal Y axes.</p> Another thing I want to try is to identify collocations (multi-word phrases like “christmas tree”) and use those as the elements over which the FSA is built, rather than always simple-mindedly splitting on word boundaries.</p> There’s also a lot of work to do on visual presentation. The current diagrams are just the raw output from Mermaid</a> and are hard to read. Perhaps there’s a layout algorithm that is a better match for this family of graphs. My suspicion is that, in the end, the visualization should look significantly less graph-like, with the FSA graph serving as more of an inner structure to guide layout decisions. Related</a>:</p> First there is old work by Book and Chandra, with the following abstract.</p> Summary. It is shown that for every finite-state automaton there exists an equivalent nondeterministic automaton with a planar state graph. However there exist finite-state automata with no equivalent deterministic automaton with a planar state graph.</p> </blockquote> </blockquote> I wonder if it would be reasonable to “planarize” nonplanar FSA graphs by duplicating nodes and edges. I’d like to make it easier to read these from left to right without so much eye movement, and being able to lay sentences out without intersection seems like it would help.</p> A few things this project reminds me of:</p> The data is similar to that underlying some visualizations I worked on for large collections of distributed traces. For that effort we aggregated individual traces into a prefix tree, with edges representing service dependencies. (FSAs are compressed representations of prefix trees). One of the powerful ideas in that project was that we built service-centric views that merged together all of the prefix tree nodes for the target service in postprocessing, which is similar to what I did above in building the prefix trees for the prefix/suffix of a local window around the target word.</li> Both forms of merging result in ambiguous graphs. I think this is fine so long as there is a way to disambiguate interactively – it could let us scale to much larger sentence sets.</li> </ul> </li> This post on dominator trees</a>.</li> The Gyllenhaal Experiment</a>, which visualizes people’s attempts to spell a famous person’s name:</li> </ul> </p> Creating a Visualization Library in Mathematica 2024-10-03T21:24:26-04:00 Status: First draft; written in a hurry. Publishing since I want to get better at publishing incremental work and nobody reads these things anyways.</em></p> I recently came back to using Mathematica</a> as my tool of choice for early-stage prototyping and data exploration. Unlike many other language ecosystems, Mathematica comes with a lot built in, including many data visualization functions. However, plots are scattered throughout the landscape without a high-level system unifying them: every new chart lives in its own world, and there’s nothing resembling The Grammar of Graphics</a> unifying them.</p> So I decided to see how much work it would take to create a more uniform and flexible plotting system like Plot</a>, my current tool of choice for browser-based visualizations, in Mathematica. It’s been a few days and I’ve made some progress and have the following picture to show for it:</p> </p> Which was made by this code:</p> plot[ dot[ "x" -> data[[All, 1]], "y" -> data[[All, 2]], "stroke" -> data[[All, 5]]]] </code></pre> The library works along very similar lines to Observable Plot, which is the existing library I’m most familiar with, and directly inspired this effort.</p> Technical notes</h2> The following was written in something of a hurry, and I plan to refine it later.</p> The basic idea is that a plot is a collection of marks overlaid atop each other. (This plot has one mark, dot</code>.)</p> A plot encodes meaning through visual channels</em>, such as x</em> or y</em> or color</em>. Each mark provides data to some number of channels. Based on this data, together with preferences provided by the user, the plot function comes up with an encoding</em> for each channel that maps from the data domain to the visual domain (eg. taking a category label such as “a” and turning it into a color, or taking a numeric data value and turning it into a position along the x-axis.</p> The usability of the library depends strongly on the amount of deduction the library can make for you, and its ability to add explanatory marks to the plot, such as legends and axes, so that the resulting picture can be made sense of.</p> An interesting point that I had not sufficiently appreciated is the importance of representing scales in a way that allows you to make these explanatory marks. For example, the library needs to be able to tell when a log scale is used so that it can generate logarithmic axis ticks. Another example is that in Plot, the “shorthand” field specification syntax is what allows the library to know how to label your axes. A possibility in Mathematica is to use controlled evaluation for the same purpose.</p> This also raises some questions about evaluation order. I was hoping to use regular marks to build the axis legends, but had to figure out a solution to the time-ordering problem: Normal marks determine the set of inputs for each channel, from which scales for each channel are inferred, and ticks generated. But axis marks need to know the tick values. My current solution is to allow a mark to be a function which will be invoked with scale information. Any points plotted by such marks will not be used to determine the domain of the plot.</p> Another Plot design point that I hadn’t fully appreciated is that one reason it uses a z</em> channel to allow eg. the line mark to make multiple lines is because sometimes one data column wants to be plotted as multiple curves, eg. with categorical time series, where you don’t want the user to need to reshape their data to calling your mark function.</p> My implementation so far is about 250 lines of code, and doesn’t yet include the ability to make bar charts, does not include faceting, or support for something like Plot’s z</em> channel. I’m also not yet sure about the best way to support interactive selection, highlighting, and animated transitions. But you can use continuous, ordinal, and categorical encodings to plot text, dots, and lines, with scale and tick inference and a composable mark system.</p> We’ll see if my current design survives the addition of any of these new features. Part of the pleasure of using such a high-level programming language is that rewrites are cheap: when you know what you want to say, and if performance isn’t too much of a concern, you can express yourself very concisely.</p> References</h2> Plot</a> – My favorite plotting library, from the folks behind d3</a>.</p> Wolfram Videos: ggplot: A Grammar of Graphics for the Wolfram Language</a> – I came across this while writing this note. Haven’t watched it yet.</p> Appendix: Code</h2> (* Names of all channels ) channelNames = {"x", "y", "r", "color", "opacity", "fx", "fy"}; ( Mapping from option names to channel names ) optionChannels = <\|"x" -> "x", "x1" -> "x", "x2" -> "x", "y" -> "y", "y1" -> "y", "y2" -> "y", "r" -> "r", "stroke" -> "color", "fill" -> "color", "opacity" -> "opacity", "strokeOpacity" -> "opacity", "fillOpacity" -> "opacity", "fx" -> "fx", "fy" -> "fy"\|>; ( Converts its argument to a list, if it isn't one already ) toList[x_] := If[ListQ[x], x, {x}] ( "Broadcasts" scalar values in an association to lists, so the \ result is tabular in structure. ) toTable[assoc_] := AssociationThread[ Keys[assoc] -> Map[toList, Transpose[Thread[Values[assoc]]]]] ( Normalization functions to rescale a domain to [0, 1] in the case \ of continuous and ordinal, or to [0, n] in the case of indexed ) continuous[domain_] := Rescale[#, MinMax[domain]] & indexed[domain_] := AssociationThread[domain -> Range[Length[domain]]] ordinal[domain_] := indexed[domain]/Length[domain] - 1/(2 Length[domain]) ( Map an association to another association: <\| k->v \|> -> <\| \ k->f[k,v ]\|> ) ( This is a bit confusing, but have a look at its usage in mark \ render functions. ) kvMap[f_, assoc___] := AssociationMap[First[#] -> f[First[#], Last[#]] &, assoc] ( Applies the appropriate scale function to each option in assocs ) scaleMap[scale_, assocs__] := kvMap[ Map[scale[optionChannels[#]]["apply"], #2] &, toTable[Association[assocs]]] ( Returns a list of k->v where k is the channel name and v is a list \ of values ) markChannelValues[mark_] := mark // List // Prepend[optionChannels] // KeyIntersection // Values // Transpose // MapApply[#1 -> toList[#2] &] // KeyDrop[Keys[mark["scales"]]] ( Dot ) render[dot, opts_, scale_, ctx_] := Module[{ m = scaleMap[scale, <\|"x" -> 0, "y" -> 0\|>, KeyTake[opts, {"x", "y", "r", "stroke"}]]}, MapThread[ Style[Circle[#1, #2], #3] &, {Transpose[ ctx["dims"]{m["x"], m["y"]}], m["r"], m["stroke"]}]] dot[opts___] := Association[Options[dot], opts] Options[dot] = {"type" -> dot, "fillOpacity" -> 0.1, "strokeOpacity" -> 0.5, "r" -> 1, "stroke" -> Automatic, "scales" -> <\|\|>}; (* Line ) render[line, opts_, scale_, ctx_] := Module[{ m = scaleMap[scale, KeyTake[opts, {"x", "y", "stroke"}]]}, ( For now we draw a single curve with a single color. ) Style[Line[Transpose[ctx["dims"]{m["x"], m["y"]}]], First[m["stroke"]]]] line[opts___] := Association[Options[line], opts] Options[line] = {"type" -> line, "x" -> 0, "y" -> 0, "fillOpacity" -> 0.3, "stroke" -> Automatic, "scales" -> <\|\|>}; (* Text ) render[text, opts_, scale_, ctx_] := Module[{ m = toTable[Association[ scaleMap[scale, KeyTake[opts, {"x", "y", "stroke"}]], KeyTake[opts, {"dx", "dy"}], <\|"align" -> opts["align"]\|>]]}, MapThread[ Style[Text[#1, #2, #3], #4] &, {opts["text"], Transpose[ctx["dims"]{m["x"], m["y"]} + {m["dx"], m["dy"]}], m["align"], m["stroke"]}]] text[opts___] := Association[Options[text], opts] Options[text] = {"type" -> text, "x" -> 0, "y" -> 0, "dx" -> 0, "dy" -> 0, "align" -> Center, "stroke" -> Automatic, "scales" -> <\|\|>}; (* Link ) render[links, opts_, scale_, ctx_] := Module[{ m = Association[ scaleMap[scale, KeyTake[opts, {"x1", "y1", "x2", "y2", "stroke"}]], KeyTake[opts, {"dx1", "dx2", "dy1", "dy2"}]], p1, p2, xy }, p1 = Transpose[ctx["dims"]{m["x1"], m["y1"]} + {m["dx1"], m["dy1"]}]; p2 = Transpose[ctx["dims"]{m["x2"], m["y2"]} + {m["dx2"], m["dy2"]}]; Thread[Style[Map[Line, Thread[List[p1, p2]]]], m["stroke"]]] Options[links] = {"type" -> links, "x1" -> 0, "y1" -> 0, "x2" -> 0, "y2" -> 0, "dx1" -> 0, "dx2" -> 0, "dy1" -> 0, "dy2" -> 0, "stroke" -> Automatic, "scales" -> <\|\|>}; links[opts___] := Association[Options[links], opts] ( Axes are given as function marks, which accept the scales as an \ argument, and whose values do not inform the scale domains. ) ( This solves the order-of-operations problem where the domain is \ inferred from marks, and wants to be "visualized" by axis marks. ) ( We still need a way to signal to Plot to not add its own axes if \ the user provided some; maybe with some magic options. ) xAxis[] := If[KeyExistsQ[#, "x"] && Length[#["x"]["domain"]] > 0, {text["x" -> #["x"]["ticks"], "y" -> 0, "text" -> Map[NumberForm[#, {Infinity, 1}] &, #["x"]["ticks"]], "dy" -> -18, "stroke" -> Black, "align" -> Top, "scales" -> <\|"y" -> <\|"apply" -> Identity\|>, "color" -> <\|"apply" -> Identity\|>\|>], links["x1" -> #["x"]["ticks"], "x2" -> #["x"]["ticks"], "y1" -> 0, "y2" -> 0, "dy1" -> -12, "dy2" -> -16, "scales" -> <\|"y" -> <\|"apply" -> Identity\|>, "color" -> <\|"apply" -> (Black &)\|>\|>]}, {}] & yAxis[] := If[KeyExistsQ[#, "y"] && Length[#["y"]["domain"]] > 0, {text["x" -> 0, "y" -> #["y"]["ticks"], "text" -> Map[NumberForm[#, {Infinity, 1}] &, #["y"]["ticks"]], "dx" -> -20, "stroke" -> Black, "align" -> Right, "scales" -> <\|"x" -> <\|"apply" -> Identity\|>, "color" -> <\|"apply" -> Identity\|>\|>], links["x1" -> 0, "x2" -> 0, "y1" -> #["y"]["ticks"], "y2" -> #["y"]["ticks"], "dx1" -> -12, "dx2" -> -16, "scales" -> <\|"x" -> <\|"apply" -> Identity\|>, "color" -> <\|"apply" -> (Black &)\|>\|>]}, {}] & ( Default color schemes ) indexedColorScheme := Module[{colors = Map[Interpreter["StructuredColor"], {"#1f77b4", "#ff7f0e", "#2ca02c", "#d62728", "#9467bd", "#8c564b", "#e377c2", "#7f7f7f", "#bcbd22", "#17becf"}]}, colors[[#]] &] ordinalColorScheme := Blend[Map[ Interpreter["StructuredColor"], {"#f7f4f9", "#e7e1ef", "#d4b9da", "#c994c7", "#df65b0", "#e7298a", "#ce1256", "#980043", "#67001f"}], #] & ( Scale inference fills in the subcomponents of a channel scale \ based on the channel, whether the values are numeric, and \ already-specified subcomponents. ) ( Base case ) inferScale[channelName_, isNumeric_, {dom_, norm_, out_}] := {dom, norm, out} ( Default "out" for the radius channel ) inferScale["r", isNumeric_, {dom_, norm_, Automatic}] := {dom, norm, (2 + 5 #) &} ( Default domain for the "opacity" channel ) inferScale["opacity", isNumeric_, {Automatic, norm_, out_}] := {{0, 1} &, norm, out} ( Default domain and normalization for channels with all numeric \ values ) inferScale[channelName_, True, {Automatic, Automatic, out_}] := {MinMax, continuous, out} ( Default domain and normalization for channels with nonnumeric \ values ) inferScale[channelName_, False, {Automatic, Automatic, out_}] := {Union, ordinal, out} inferScale["color", False, {Automatic, Automatic, out_}] := {Union, indexed, out} ( Default continuous\|ordinal color scheme ) inferScale["color", isNumeric_, {dom_, norm : continuous \| ordinal, Automatic}] := {dom, norm, ordinalColorScheme} ( Default indexed color scheme ) inferScale["color", isNumeric_, {dom_, norm : indexed, Automatic}] := {dom, norm, indexedColorScheme} ( Default normalization is continuous ) inferScale[channelName_, isNumeric_, {dom_, Automatic, out_}] := {dom, continuous, out} ( Default out transform is Identity ) inferScale[channelName_, isNumeric_, {dom_, norm_, Automatic}] := {dom, norm, Identity} ( Tick inference figures out the ticks for a scale based on the \ properties of its subcomponents. ) inferTicks[in_, norm : ordinal \| indexed, domain_] := domain inferTicks[in : Log10, norm : continuous, domain_] := Range[Floor[in[First[domain]]], Ceiling[in[Last[domain]]], 1/4.] inferTicks[in_, norm : continuous, {lo_, hi_}] := Range[lo, hi, (hi - lo)/4.] ( Remove Null elements from a list, since it's a common mistake to \ call plot[mark1, mark2, ] with a trailing comma. ) removeNulls[xs_] := xs /. Null -> Sequence[] ( The plot function accepts a list of marks, and returns a graphics \ object. *) Options[plot] = {"dims" -> {100, 100}, "margins" -> {{50, 10}, {50, 10}}, "metaMarks" -> ({} &)}; plot[marks___, OptionsPattern[]] := Module[{ dims = OptionValue["dims"], margins = OptionValue["margins"], markList = Join[{xAxis[], yAxis[]}, removeNulls[Flatten[{marks}]]], markTypes, channelValues, channelNumericQ, scales, markScales, filledMarks, renderedMarks }, channelValues = Merge[Map[markChannelValues, Select[markList, AssociationQ]], Catenate]; channelNumericQ = Map[AllTrue[NumericQ], channelValues]; scales = Association[ <\|"x" -> <\|"domain" -> {}, "ticks" -> {}, "apply" -> (0 &)\|>, "y" -> <\|"domain" -> {}, "ticks" -> {}, "apply" -> (0 &)\|>\|>, channelValues // kvMap[Module[{ channel = #, values = #2, in = Identity, dom, norm, out, domain}, {dom, norm, out} = ConstantArray[Automatic, 3] //. opts_ :> inferScale[channel, channelNumericQ[channel], opts]; domain = dom[Map[in, values]]; <\|"domain" -> domain, "ticks" -> inferTicks[in, norm, domain], "apply" -> Composition[out, norm[domain], in]\|> ] &] ]; markList = FixedPoint[Replace[Flatten[#], f_Function :> f[scales], {1}] &, markList]; markScales = markList // Map[Merge[{scales, #scales}, Merge[Last]] &]; markTypes = Comap[markList, "type"]; renderedMarks = Thread[render[markTypes, markList, markScales, <\|"dims" -> dims\|>]]; Graphics[ Translate[renderedMarks, Map[First, margins]], ImageSize -> dims + Map[Total, margins], PlotRange -> Transpose[{{0, 0}, dims + Map[Total, margins]}]]] raw = ExampleData[{"MachineLearning", "FisherIris"}, "Data"]; data = raw // Map[Append[First[#], Last[#]] &]; plot[ dot[ "x" -> data[[All, 1]], "y" -> data[[All, 2]], "stroke" -> data[[All, 5]]]] </code></pre> It depends what race you’re running 2024-09-20T20:04:25-04:00 Imagine there are three teams of runners who are about to compete against each other in a team-vs-team race.</p> The first team has some very fast runners, but also some that are very slow. Despite the slowpokes, this team has highest average speed.</p> 🯅 🯅 🯅 🯅 🯅 🯅 🯅 ----------------------------- Slow Fast </code></pre> The second team is made up entirely of unremarkable runners, all of whom run at roughly the same middling pace.</p> 🯅🯅🯅🯅🯅🯅🯅 ----------------------------- Slow Fast </code></pre> The third team consists of runners who are far below average. But it has one incredible athlete – a world-class talent who’s easily the fastest runner on any of the teams.</p> 🯅🯅🯅🯅🯅🯅 🯅 ----------------------------- Slow Fast </code></pre> Which of these teams is most likely to win the race? Well, it depends entirely on what kind of race it is.</p> It might be a relay race, in which the total time taken by a team is the sum of the times taken by the individual runners. In this scenario, the first team would be most likely to win since it has the highest average speed.</li> Or maybe the race is one in which all of the runners start at the same time and teams are scored by how long it takes everyone on the team to finish the race. In this case, the time for a team depends exclusively on its slowest runner, which would give the second team the advantage since the other two teams both have slower runners.</li> Or maybe teams are scored not by their slowest runner, but by their fastest. Then the third team would be the favorite since it has the fastest individual runner.</li> </ol> This analogy is an attempt to explain some simple but deep ideas in software performance that I came to relatively recently, despite writing software for more than a decade:</p> It’s useful to think of the execution time of a program not as a number, but as a distribution.</li> Which aspects of this distribution are worth optimizing depends not on the program, but on the context in which the program is invoked.</li> </ul> In the analogy, each team represents a computer program, and each runner represents a single execution of that program. The differences in the runner’s “run times” captures the idea of performance variation across multiple executions of the same program.</p> The relay race is like a plain for</code> loop. Executions of the program occur serially, and the total time taken is the sum of the times of the individual executions. To win this race, it makes sense to optimize the mean execution time, which is just a scaled version of the sum.</li> The second race, in which all of the runners have to cross the finish line, is like invoking a program many times in parallel, then waiting for all of them to complete, as one might in a map-reduce</a> computation. For example, imagine you want to search a partitioned dataset. Individual partitions can be searched in parallel, and then the results combined, with the total run time depending on the maximum time it takes to conduct an individual search (since you need to wait for the slowest search to finish before returning the combined results).</li> The third race, in which the team with the fastest runner wins, is like issuing a bunch of parallel calls and but only waiting around for the first result. A practical example of this pattern is the idea of hedged requests</a> where, simplifying somewhat, to improve the response times from a service you send it multiple identical requests and take the response that arrives first, ie. the one with the minimum run time.</li> </ol> References</h2> The Tail at Scale</a> (summary</a>)</p> Just as fault-tolerant computing aims to create a reliable whole out of less-reliable parts, large online services need to create a predictably responsive whole out of less-predictable parts; we refer to such systems as “latency tail-tolerant,” or simply “tail-tolerant.” Here, we outline some common causes for high-latency episodes in large online services and describe techniques that reduce their severity or mitigate their effect on whole-system performance.</p> </blockquote> Minimum Times Tend to Mislead When Benchmarking</a></p> In this short post, I want to make a simple argument against using the minimum time of a benchmark as a proxy for that benchmark’s performance.</p> </blockquote> Tail Latency Might Matter More Than You Think</a></p> I continue to believe that if you’re going to measure just one thing, make it the mean. However, you probably want to measure more than one thing.</p> </blockquote> This draft is based on ideas I learned about in discussions with Yao Yue, Dan Luu, and Rebecca Isaacs over the last few years, all of whom know way more about this stuff than I do and have</a> written</a> articles</a> and</a> papers</a> that cover the same ground but in much more conceptual depth and technical detail.</p> Also, all this assumes that you care about winning the race</a> in the first place.</p> Indie Apps I Enjoy 2024-09-13T00:11:01-04:00 Sublime Text</a></h3> This is my text editor of choice for many years and I’m a happy paying customer. It’s fast with a very light feel, and does the basics well – editing, navigation, search – while staying responsive and almost bug-free.</p> I’ve tried VSCode before but couldn’t get past inelegancies in the user interface or the laggy feeling of the editor. The folks behind Sublime Text also make a visual Git client, Sublime Merge</a>, which I also use.</p> superwhisper</a></h3> This is a great little app for speech-to-text transcription. In fact, I’m writing this post with it. I like it for its relatively reliable and responsive text-to-speech functionality (press a keyboard shortcut, talk, press another one, and have it paste the transcribed text). I haven’t been interested in any of their more recent developments that add additional AI features.</p> Little Snitch</a></h3> A nice firewall and network monitor app for Mac that comes with some nice visualizations of data rates over time, and geo-located servers that your computer is talking to.</p> Things</a></h3> This is a to-do app that does the basics well and has an extremely pleasant user interface. I mostly use it to add tasks on the go using a shortcut on my home screen. These tasks go into an Inbox area that I can triage later. It’s mostly helpful during the times of my life when things get busy and I need some external memory to stay organized.</p> Bike</a></h3> Bike is a text editor and outlining tool. I like its approach to text editing and use it for some of my freeform notes because it makes it easy to create a hierarchy and jump around a big outline. There are some features that I have consistently missed (I want to be able to paste in images & files), so I’m in the process of moving on to other apps for my writing, but I still like many of its ideas, including a novel caret affinity</a> system that solves a common issue when editing rich text.</p> Weather strip</a></h3> This app has a really nice visual way of presenting the weather. I’m always excited about clear and clever data visualizations. </p> Capo</a></h3> This is a fun app that helps you learn how to play a song by doing some helpful signal processing and showing you an annotated spectrogram with chord changes, and gives you the ability to slow down and loop particular sections of a song.</p> Others</h3> This is getting a bit long so I’ll end with a few other indie apps (all created by independent developers or small teams) that I either use or spiritually approve of: Bear</a>, Overcast</a>, Halide</a>, Puzzmo</a>, Datasette</a>, and Kirby</a>.</p> And on a related note, there are a few individuals who make fonts I like a lot: MB Type</a> and Fabrizio Schiavi</a>.</p> Setting up an auto-publishing workflow 2024-09-12T23:08:24-04:00 A quick note on how I’m syncing Bear with this site, since I haven’t done this before.</p> I’m using launchd</code></a> to update this site the with the most recent content from Bear</a> every minute. Right now, there’s hardly any content here so the inefficiency of re-generating and re-deploying everything is not a problem.</p> The launchd</code> bit was a bit prickly to figure out, so I figured I’d write it down here. ChatGPT helped with a lot of the basics. Here are the instructions I ended up with (just pub</code> is a recipe that uses the excellent Just</a> command runner):</p> Save this to ~/Library/LaunchAgents/is.yuri.pubcommand.plist</code>:</p> <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>is.yuri.pubcommand</string> <key>StartInterval</key> <integer>60</integer> <key>ProgramArguments</key> <array> <string>/bin/zsh</string> <string>-c</string> <string>just pub</string> </array> <key>WorkingDirectory</key> <string>/Users/yurivish/Dropbox/Projects/what</string> <key>StandardOutPath</key> <string>/tmp/what_yuri_is_output.log</string> <key>StandardErrorPath</key> <string>/tmp/what_yuri_is_error.log</string> <key>EnvironmentVariables</key> <dict> <key>PATH</key> <string>/Users/yurivish/.cargo/bin/:/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string> </dict> <key>RunAtLoad</key> <true/> </dict> </plist> </code></pre> Then run:</p> launchctl unload ~/Library/LaunchAgents/is.yuri.pubcommand.plist launchctl load ~/Library/LaunchAgents/is.yuri.pubcommand.plist </code></pre> This will automatically run the publication command every minute, publishing the #pub</code> notes from Bear.</p> To temporarily disable the scheduled job, you can unload the plist using launchctl:</p> launchctl unload ~/Library/LaunchAgents/com.user.pubcommand.plist </code></pre> This will stop the job from running without deleting the plist file. To re-enable it later, simply load it again:</p> launchctl load ~/Library/LaunchAgents/com.user.pubcommand.plist </code></pre> You can see logs by tail</code>ing the paths specified in the plist file under StandardOutPath</code> and StandardErrorPath</code>.</p> Trying something new. 2024-09-12T20:52:36-04:00 This is my place for rough ideas and half-baked thoughts. I’ve set it up to automatically synchronize with a new note app that I’m trying out, Bear</a>.</p> I hope this workflow will lower the barrier to writing online. I’ve wanted to be a writer for a long time but have never been very comfortable with the process, and I often feel like my taste handily exceeds my skills. This seems fairly common</a> and I’m finally going to do something about it.</p> In the spirit of making things easy I’m going to try another experiment, which is to draft my notes with voice using superwhisper</a>. This will either increase the length of my posts or improve my speaking concision…</p> The main goal is to get comfortable publishing messy notes and unrefined ideas. I’m curious to see how it goes.</p> </p> Euclidean Distance and Pearson Correlation 2021-12-28T00:00:00+00:00 I recently learned there’s a useful connection between Euclidean distance</a> and Pearson correlation</a>: the squared Euclidean distance between two z-normalized</a> vectors equals their Pearson distance</a>, up to a scale factor.</p> This short note walks through a derivation.</p> Z-normalization</h3> Let’s say we have two z-normalized vectors $x$ and $y$ in $N$ dimensions.</p> Z-normalizing</a> a vector involves subtracting its mean then dividing by its standard deviation, so we know that the means $\mu_x$ and $\mu_y$ of our vectors $x$ and $y$ are zero:</p> $$ \mu_x = \mu_y = 0 $$</p> and their standard deviations $\sigma_x$ and $\sigma_y$ are one:</p> $$ \sigma_x = \sigma_y = 1 $$</p> We’re going to introduce some statistical concepts and see how they simplify under these constraints. Expressions in color</span> are simplifications valid only under z-normalization.</p> Summations implicity sum over all terms: $\small{\sum{x_i} = \sum_i^Nx_i}$</p> Covariance</h3> Since the means $\mu_x$ and $\mu_y$ are both zero, the numerator reduces to a dot product:</p> $$ \operatorname{cov}(x, y) = {\sum{(x_i - \mu_{x})(y_i - \mu_{y})} \over N} $$</p> $$ \htmlClass{special}{\operatorname{cov}(x, y) = {{\sum{x_iy_i}} \over N}} $$</p> Standard deviation</h3> The standard deviations $\sigma_x$ and $\sigma_y$ both simplify to one by the definition of z-normalization:</p> $$ \sigma_x = \sqrt{cov(x, x)} $$</p> $$ \htmlClass{special}{\sigma_x = \sqrt{\sum{x_i^2} \over N} = 1} $$</p> Pearson correlation</h3> The denominator reduces to one, leaving us with just the covariance:</p> $$ \operatorname{P}(x, y) = {\operatorname{cov}(x, y) \over \sigma_x\sigma_y} $$</p> $$ \htmlClass{special}{\operatorname{P}(x, y) = \operatorname{cov}(x, y) = {{\sum{x_iy_i}} \over N}} $$</p> Euclidean distance</h3> Now let’s look at squared Euclidean distance</a> between z-normalized vectors.</p> $$ \operatorname{dist^2}(x, y) = \sum{(x_i - y_i)^2} $$</p> We can expand out the squared differences inside the sum:</p> $$ {\operatorname{dist^2}(x,y) =}\sum{(x_i^2 - 2x_iy_i + y_i^2)} $$ then separate out the terms into their own sums so we can tackle them one by one:</p> $$ {\operatorname{dist^2}(x,y) =} \sum{x_i^2} - 2\sum{x_iy_i} + \sum{y_i^2} $$</p> Let’s examine this. The two sum-of-squares terms, $\sum{x_i^2}$ and $\sum{y_i^2}$, are both $N$ times the squared standard deviation, which reduces to $N$:</p> $$ \underbrace{\sum{x_i^2}}_{N \text{ times } \sigma^2_x ~=~ N} \htmlClass{light}{ - 2\sum{x_iy_i} +} \underbrace{\sum{y_i^2}}_{N \text{ times } \sigma^2_y ~=~ N} $$ </div> Meanwhile, the middle term is $2N$ times $\operatorname{P}(x, y)$:</p> $$ \htmlClass{light}{\sum{x_i^2} -} \underbrace{2\sum{x_iy_i}}_{2N \text { times } \operatorname{P}(x, y)} \htmlClass{light}{+ \sum{y_i^2}} $$ Putting these pieces together, our formula becomes:</p> $$ \htmlClass{special}{\operatorname{dist^2}(x, y) = 2N - 2N \cdot \operatorname{P}(x, y)} $$</p> which can be simplified to:</p> $$ \htmlClass{special}{\operatorname{dist^2}(x, y) = 2N(1 - \operatorname{P}(x, y))} $$</p> showing that the squared Euclidean distance between two z-normalized vectors is $2N$ times their Pearson distance</a>.</p> Why does this matter?</h3> A common trick to improve Euclidean nearest-neighbor searches is to z-normalize the vectors that you’re searching over. The connection to Pearson correlation gives us an intuition for why this works so well in practice.</p> Additionally, representing correlation as a distance metric allows us to use it with algorithms that rely on properties of a distance metric, such as the triangle inequality</a>. Despite the name, Pearson distance is not actually a distance</a>.</p> References</h3> On Clustering Time Series Using Euclidean Distance and Pearson Correlation</a>:</p> For time series comparisons, it has often been observed that z-score normalized Euclidean distances far outperform the unnormalized variant. In this paper we show that a z-score normalized, squared Euclidean Distance is, in fact, equal to a distance based on Pearson Correlation.</p> </blockquote>

yuri.is

LLM Experiments #5: Design by API

2026-05-22T21:25:56-04:00

Note: Experimenting with quick writeups as I play around with LLMs and my homegrown agentic workflow system. Previously.</a>

When working on my workflow system I’m starting to find that the most effective use of my attention is to focus on API surfaces of the core subsystems. With a layered API design</a>, the surface will span multiple levels of subsystem, and so give a rough shape to everything underneath. I’ve found property testing</a> and synctest</a> to be valuable tools for defining and testing the properties of the concurrent execution engine: the contracts of the API serve as useful inspiration for tests. In general, a lot about this project feels like it’s about artfully and verifiably constraining growth.

LLM Experiments #4: Intermezzo

2026-05-18T20:59:34-04:00

Note: Experimenting with quick writeups as I play around with LLMs and my homegrown agentic workflow system. Previously.</a> Scattered notes this time.
Agent swarms for bug hunting</h2>
Inspired by Mythos, I wanted to see what I could find with a small army of sub-agents hunting for bugs in Julia. It helped to have done this before</a> since I could direct the agents to known-sus parts of the codebase, and instruct them on tactics.
With the lower-level work automated I was able to put more of my efforts into qualifying the legitimacy/severity of the bugs and organizing the results, ending up with a compact table of ~10 bugs each with a one-line summary and one-line reproduction. A few turned out to be documented/allowed behavior, others were quickly</a> fixed</a>, and some remain open, awaiting agents from the future.
Example-driven evaluation & quick cross-language prototyping</h2>
I’ve been working on the design of my agentic workflow/podcast analysis system by writing lots of pipelines and seeing how they surface gaps in the underlying system and API. It’s so cool that the example-driven feedback loop approach can now be scaled.
I’ve also been playing around with cross-language rewrites, but am finding that when this is done with too much AI assistance it’s much less valuable since it’s difficult to judge the results. Implementation complexity can vary by orders of magnitude depending on the skill of the implementer, and evaluating the results a deep familiarity with the possibility space. 

LLM Experiments #3: Noodling

2026-05-10T10:18:33-04:00

Note: Experimenting with quick writeups as I play around with LLMs and my homegrown agentic workflow system. Previously.</a>
Nothing much to write about today, but here are some things I’ve explored since my last post.
Gemma 4 has goblin tokens</h2>
Gemma 4 tokens can be seen in tokenizer.json here</a>. I took a look, and it turns out that Gemma has a “ Yuri“ token: "▁Yuri": 142143</code>, as well as a few goblin-related tokens: "▁goblin": 218798</code>,"▁Goblin": 171680</code>, "oblins": 236144</code> (the underscore-looking character encodes a space). The tokenizer is BPE (line 311 of tokenizer.json) so I wonder what the presence of these tokens says about the frequency of goblins in the training data. Probably not much, since each of tacos</code>, Tocco</code>, getColumn</code>, StudentVector</code>, flipping</code>, counselling</code>, and combustibles</code> are also tokens, as well as other infrequent words. There are multiple tokens for each of Google and OpenAI, and tokens for anthrop</code> and ic</code>.
The existence of these tokens implies that you can use llama-server’s --logit-bias</code> option to selectively upweight goblins, for example, or steer the model into profanity by increasing the probability of bastard</code>.
You need to know the goal to refine the task</h2> I relearned this lesson today, this time in the context of optimizing data pipelines whose outputs are subjective. When you’re just starting out it can be hard to know what you want precisely enough to describe to a small language model, since small models fail loudly in visible ways that force you to specify increasing amounts of procedural detail in your prompt. In contrast, big models are often right enough that you can defer tweaking until you know more about how the results will be applied.Sampling in llama-server</h2> One way to control model output is by changing your sampling strategy. llama-server</code> has many options for this. Many strategies are represented as functions that accept and return a probability distribution, and can thus be composed. Others, like adaptive-p</a>, are intended to be used as the last sampler in the chain: adaptive-p selects a token ID rather than just mutating candidates, so it must be last in the sampler chain. It shares this behaviour with some existing samplers like mirostat, dist, and greedy (mirostat being the closest relative). </blockquote> Lots of good descriptions in the completions tool readme</a>. Gemma 4 26B is pretty good at extracting emotion-subject pairs</h2> Gemma 4 does surprisingly well, but has false positives, eg. failing in simple cases like “I want my pilot to lie to me.” which returns { subject: "pilot", emotion: "desire" }</code> where the referent of the emotion is inaccurate (and I don’t think wanting is an emotion, though it’s apparently debatable). Other edge cases involve tracking the subject through nested conversations, eg. it emitted { subject: "speaker", emotion: "ridicule" }</code> for the following quote: And the funny thing is I own Final Cut Pro. I could just click, click, click and make it happen, but I’m like, but I want to figure it out using ffmpeg. And somebody could objectively say, "You idiot, it’s going to take you an hour to do it this way. It would take a minute to do it this way." I’m like, "But this makes me happier." And that matters. </blockquote> Whether or not this is acceptable depends on what you’re planning to do with it, but to me, a quote where the speaker imagines a critic ridiculing the speaker feels like it should be represented differently from one in which the the speaker directly ridicules themselves. I wonder if, rather than trying to anticipate all possible edge cases in the prompt, a generator-verifier pattern might be useful here. I have an unvalidated intuition that decomposing into workflows can be a way to work with the grain of a model, and, more generally, that designing robust pipelines that inherently compensate for expected failure may be a better strategy than to trying to engineer it out.
LLM Experiments #2: Eval Planning 2026-05-08T17:54:55-04:00 Note: Experimenting with quick writeups as I play around with LLMs and my homegrown agentic workflow system. Previously.</a> LLM costs scale with a request volume, and it turns out that my extractive summarization approach is much too expensive when run at scale on frontier-ish cloud models. So I’m seeing if I can get a small off-the-shelf model, Gemma 4 E2B</a>, to do the task more efficiently. Since this is a side project I’m been experimenting with letting an agent autonomously drive both the creation and optimization of the pipeline, and view my job as giving the agent a good home with easy access to tools for exploring data, debugging code, and taking performance measurements. As part of this, the workflow system itself is event-sourced</a> and includes a provenance tracking system for every data output, allowing us to effortlessly connect outputs back to inputs and intermediate results. One point of concern is that extractive text compression doesn’t have an obvious scoring function. Fortunately language models are pretty good at language so I’m letting Opus 4.7 judge the quality of the output, with small bits of targeted feedback from me. And for inspiration I did have it read John McPhee’s Draft No. 4</a>. In terms of technology, I’ve installed llama-cli</code> and am running with enough parallelism to almost saturate the GPU at peak load as measured by mactop</a>. Beyond that, I haven’t done any additional tuning beyond making sure that we’re caching common prefixes across prompts: { "model": "ggml-org/gemma-4-E2B-it-GGUF", "messages": […], "cache_prompt": true, "max_tokens": 2048, "response_format": { … } } </code></pre> To look at the results I had the agent build me a viewer that lets me browse the iterative simplifications and easily leave individualized feedback on sentences or paragraphs whose outputs look incorrect. The feedback goes into a file that the agent looks at to guide its optimization. LLM Experiments #1: Shorter Sentences 2026-05-07T12:56:02-04:00 Note: Experimenting with quick writeups as I play around with LLMs using my homegrown agentic workflow system. Today’s idea (and the workflow system itself) both came from a dream of building a “daily pulse” of the podcast news ecosystem. As part of that I’m exploring ways to concisely present podcast transcripts. In particular, I want to create summaries that preserve the voice of the speakers while still condensing the text. A summary can be abstractive or extractive: abstractive summaries paraphrase the original text; extractive summaries “extract” by pruning — removing sentences, phrases, or words while preserving structure where possible (see this paper</a> for a discussion). For my first attempt I tried a simple two-agent approach with a generator and a verifier. Given a sentence, the generator proposes multiple candidate simplifications, and then the verifier accepts the best one, or rejects them all. This proceeds iteratively until there is no simpler acceptable candidate. Here’s the first try on a sentence from this podcast episode</a>: What’s worth keeping depends on context, so I provide the LLM a conversation snippet to help it judge what’s semantically meaningful: In its current form the algorithm is a bit overeager and can sometimes lop off semantically significant phrases, so I’m already re-running with adjustments based on this initial result, tweaking the prompts and experimenting with how to provide guidance to the model so it can more accurately tell which parts of the sentence are most important. I like this idea – it’s an antidote to the tendency of generative AI systems to increase the separation between raw materials and the final result. Instead, we can summarize people in their own words. Abraham Lincoln & English as She Is Spoke 2025-12-14T23:41:28-05:00 Sometimes the threads of the universe weave together in unexpected ways: one of my favorite books, English as She Is Spoke</a>, turns out to have also been enjoyed by Abraham Lincoln. The book is a primer on how to speak English written by somebody who does not speak it themselves. Mark Twain said of it that Nobody can add to the absurdity of this book, nobody can imitate it successfully, nobody can hope to produce its fellow; it is perfect. </blockquote> while Stephen Pile mentions this work in The Book of Heroic Failures</a> and comments: “Is there anything in conventional English which could equal the vividness of ‘to craunch a marmoset’?” The original has “to craunch the marmoset”, an entry under the book’s “Idiotisms and Proverbs”. This is the author’s attempt to translate the French slang idiomatic expression croquer le marmot, used to indicate “waiting patiently for someone to open a door” — Wikipedia</a> </blockquote> But most remarkably, this: In January 1864, then US President Abraham Lincoln</a> and Secretary of State William H. Seward</a> laughed as Lincoln’s private secretary John Hay read aloud from the book. The book has been cited as one example of many diversions that Lincoln used to lighten his heart and mind from the weight of the US Civil War and his cabinet’s political infighting. — Wikipedia</a> </blockquote> I folllowed the citation and sure enough</a>, the story is there, in a book chapter on Lincoln’s humor: One evening, after a day of harassing anxiety and work, the President, accompanied by his private secretary, walked into the library of Seward’s house. “No, more bad news, I hope, Mr. President?” “No; nothing further; but Hay happened to find a book that amused us; so I told him we would walk over to Seward’s, and read it to him and have a laugh over it.” The book was the little Portuguese guide to English conversation, since become famous under the title of “English as She is Spoke.” As John Hay read aloud its queer inverted sentences, Lincoln and Seward laughed heartily, their minds finding a brief but welcome relief from care. </blockquote> H2 Morris Counter 2025-10-23T21:14:53-04:00 I was thinking about the Morris counter</a> algorithm and wondered if it would be possible to combine it with the H2 histogram</a> encoding to have more control over the standard deviation of the approximation in exchange for spending more bits to represent the same value range. It turns out the answer is yes: // Probabilistic increment based on the bin width at the current counter value function increment(counter, a, b) { return counter + (Math.random() < 1/h2(counter, a, b).binWidth) } // Expected value of the H2 counter function estimatedValue(counter, a, b) { return h2(counter, a, b).lower } // A mini H2 histogram decoder implementation for values <= 2^32-1. // Returns the lowest value in, and bin size of, the `index`-th bin. function h2(index, a, b) { const c = a + b + 1; const binsBelowCutoff = u32(1 << (c - a)); if (index < binsBelowCutoff) { // we're in the linear section of the histogram; each bin is 2^a wide return { lower: u32(index << a), binWidth: u32(1 << a) } } else { // we're in the log section of the histogram; 2^b bins per log segment const logSegment = c + ((index - binsBelowCutoff) >>> b); const binOffset = index & (u32(1 << b) - 1); return { lower: u32(1 << logSegment) + u32(binOffset << (logSegment - b)), binWidth: u32(1 << (logSegment - b)) } } } function u32(x) { return x >>> 0; } </code></pre> The idea is that every integer represents a random variable whose expected value is the lower edge of the corresponding histogram bin, since this way the index 0 can represent the value 0. The number of integers that share this encoding is the bin width, so the probability of ascending to the next bin is 1 / bin width. The parameter a controls the bin width on the low end of the value range (typically set to 0 to represent small integers exactly), while the parameter b controls relative error on the high end by determining the number of linear subdivisions within the upper log segments. Larger values of b increase the quality of the approximation by increasing the number of bins covering the value range. When used as the basis for a histogram, the H2 encoding upper-bounds the relative error for large values. But in the context of probabilistic counters the H2 encoding does not prevent the value estimate from being wildly far away from the true value, since an unlikely series of coin flips can cause the value estimate to drift very far away. But, compared to the Morris counter, it does allow you to spend more bits to decrease the expected relative error and variance of the approximation. As a special case, when a and b are both zero, this counter is almost the same as the standard Morris counter, with the only difference being that the positive estimated values are all exact powers of two. Here’s the code for a Morris counter for comparison: // Probabilistic increment as per the Morris counter algorithm function morrisIncrement(counter) { return counter + (Math.random() < 1/Math.pow(2, counter)) } // Expected value of the Morris counter function morrisEstimatedValue(counter) { return 2 ** counter - 1 } </code></pre> References: The original paper by Robert Morris is Counting Large Numbers of Events in Small Registers</a>, published in 1977</li> The mini H2 encoding above was originally written for iopsystems/h2-histogram-js</a>.</li> The reference H2 histogram implementation is in Rust at rustcommon/histogram</a>.</li> There’s a visual explanation of the a and b histogram parameters in this notebook</a>.</li> </ul> Thanks to Sean Lynch, Brian Martin, Yao Yue, and Mihir Nanavati for a discussion that lead to these implementations. Sean made a “morris float</a>” which applies the same probabilistic approach directly to the exponent and mantissa of an f32</code>. Podcast Summarization 2025-10-20T22:23:33-04:00 Previously: Podcast Vibes Presentation</a>. Text summarization is a fundamentally a creative act, at its best preserving the essence of the original work. Writing is selection</a>, and summarization doubly so. This came to mind today while I was exploring designs for a user interface to substitute for the experience of listening to a podcast. This is a chronicle of the first hour or so of that exploration, playing around with a transcript from this recent interview</a>. At first I wanted to see whether I could simply avoid summarization altogether by avoiding line breaks and using a full-width display, but there’s quite a lot of text in a transcript, so this doesn’t work even with a relatively compact font. So I tried columns instead: Wrapping the text in columns felt a lot better since I could now move my gaze in a smaller region of space to read a complete thought. But there’s still just too much text. So I experimented with asking an LLM to take individual sentences and omit unnecessary words. Here it is for the whole transcript. Not very useful! But interesting… I used an LLM to split the transcript into topics, and then give each topic a title and description. I was initially resisting using an LLM for topic identification because it felt like handing over a lot of power to the system, but the result is much more easily scannable. You can look across the top row to get a sense of conversation, then dive in by scanning down. A fun idea to evaluate how easily auditable an AI-powered user interface is might be to use adversarial prompts that generate misleading summaries with some probability, in order to see whether I would notice. Can the cognitive cost of spot checks be made low? Same design with slightly wider columns: It could be interesting to identify conversational dependencies between transcript sections and use that to enable something like program slicing</a>, but for meaning. An example of this idea in a different context is Flowistry</a> – imagine being able to click on a topic or sentence to cut down the conversation to the subset of “related” discussion (presumably with some kind of soft relatendess cutoff). Future work: Use a change of color or font to differentiate LLM-generated from original source text.</li> Identify speakers and allow toggling the presence of individual conversational participants.</li> Try view-aware topic-splitting, where if a topic is too long for the current display, adaptively partition it into subtopics with the goal of maximizing the usage of screen space.</li> Rather than on topic per column, explore a grid with fixed-height scrollable cells or a masonry layout</a>.</li> How could this generalize to visualizing a set of thematically overlapping podcasts with the same guests (eg. 3-10 podcasts of the same author on a podcast media tour)?</li> Are there principles from story structure that we could use to structure the presentation of the text?</li> This project is forcing me to confront squarely the question of what exactly it is that one gets from a podcast. It’s clearly not just factual information: there’s also emotion, allusion, and subtext. What matters?</li> </ul> Further reading: An AI-Resilient Text Rendering Technique for Reading and Skimming Documents</a></li> AI-Resilient Interfaces</a></li> </ul> Widows, Orphans, and Runts 2025-10-17T17:45:39-04:00 In redesigning this site I wanted to prevent widows</a> in headlines, ensuring that individual overflowing words don’t wrap onto a new line. Fortunately modern CSS has a way to indicate that you want the browser to trade a bit more time for improved text layout: text-wrap: pretty</code></a>. From MDN: text-wrap: pretty</code> results in the same behavior as wrap, except that the user agent will use a slower algorithm that favors better layout over speed. This is intended for body copy where good typography is favored over performance (for example, when the number of orphans should be kept to a minimum). </blockquote> Amusingly the Wikipedia page</a> for widows and orphans states that, at least when it comes to typography, all three of these terms are used somewhat interchangeably: Widow (sometimes called orphan)</li> Orphan (sometimes called widow)</li> Runt (sometimes called widow or orphan)</li> </ul> As of late 2026 Can I Use</a> reports around 75% of global browser support for text-wrap: pretty</code>, with the bulk of the holdouts being versions of iOS before iOS 26. Data Visualizations are Data Structures 2025-10-17T16:57:48-04:00 Information is independent of its representation, and any particular data representation makes some things easy and other things hard. The art of data structures — and data visualizations — is choosing the right structure for your circumstances, and tuning it well. Concepts from the analysis and engineering of data structures, like big O</a> notation and locality in space and time, are also useful in understanding and improving visualizations and visualization systems. The “computer” that a data visualization runs on is the human mind, which has its own performance properties based on the biology of the human perceptual system and the desired information access patterns. There’s more to say on this analogy, but it will have to wait for a future post. Great Singers on Great Singing 2025-10-14T10:56:14-04:00 Years ago I read a book</a> where the author interviewed 40 famous opera singers, including Pavarotti, on their technique. I remember being struck by the differences in the ways the singers thought about their craft. While the interviewer didn’t exactly ask everyone the same questions, there was enough overlap that the tensions between them were clear. What I was most curious about – how different singers answered the same questions about their technique – was not easy to see because of the structure of the book: I wanted the data organized by topic, but the book was organized by interview. So I spent some time yesterday playing around with seeing if Claude</a> could do a first pass at this kind of digital-humanities-style data analysis by taking the knowledge from the book and restructuring it into a form where previously hidden connections would be revealed. I initially thought I would need to do a lot of manual preprocessing, and spent some time figuring out how to segment the body text away from the annotations that appear on each page. Here’s a visualization I made to help me evaluate how a particular value for the decision boundary would look when applied across more pages. After some more unsatisfactory futzing around I realized that if all I was interested in was an approximate answer, the raw messiness is okay and given the forgiving nature of LLMs, this level of attention to detail was probably just a waste of time. So I decided to just chuck all the text into Claude to see how it would do. Here’s Claude’s response to a simple prompt asking it to find and catalog the most striking juxtapositions of disagreement areas between two singers on the same subject. I asked it to reorder the topics so that the topics that are more relatable to non-singers come first, and within each topic, to reorder the items so that they maximally contrast with the items to either side of them. Topic</th> Singer</th> Position</th> Brief Supporting Quote</th></tr></thead> NATURAL VS. TECHNICAL</td> Rise Stevens</td> Less technical thinking is better</td> “When I did not know about the larynx I was singing better… Some people think so much about all of these technical things that they really mess themselves up”</td></tr> </td> Luciano Pavarotti</td> Years of intense technical work required</td> “It took me six years of study… never change ideas… the first five or six months it is very depressing”</td></tr> </td> John Alexander</td> Singing is natural like speaking</td> “Singing is simply sustained speech… I feel almost all vocal problems can be solved by shaping the singing technique to conform to the speaking technique”</td></tr> MENTAL FOCUS</td> Birgit Nilsson</td> Think low for high notes</td> “The higher the note goes, the lower the support. The support is as low as possible”</td></tr> </td> Martina Arroyo</td> Stay mentally high throughout</td> “You should stay mentally high throughout the entire scale”</td></tr> </td> Luciano Pavarotti</td> Position never changes, always high</td> “For me the position never changes… it is high, even when I sing a low note”</td></tr> PLACEMENT</td> Cornell MacNeil</td> Sing backward, placement is nonsense</td> “Placement is nonsense… utter, complete crap. You sing it backward”</td></tr> </td> Anna Moffo</td> Extreme forward placement</td> “The place is the most forward, focused sound, complete with overtones”</td></tr> </td> James McCracken</td> Warns against over-focusing on mask</td> “If somebody decides he’s going to sing in the mask before he knows how to sing… it’s thin. There’s not enough body resonance”</td></tr> </td> Placido Domingo</td> Think way out in front</td> “My thinking of singing is over there… Way out in front of me… I’m not thinking of placing the voice in the mask”</td></tr> </td> Paul Plishka</td> Everything in the nose</td> “To me, everything happens here in the nose”</td></tr> SUPPORT/APPOGGIO</td> Luciano Pavarotti</td> Push down, compress against cords</td> “Appoggiare means… push down the voice… in a way that it doesn’t move anymore…”</td></tr> </td> Cornell MacNeil</td> Floating ribs, diaphragm overrated</td> “I think the diaphragm is enormously overrated… I think it comes more from the floating ribs in the back”</td></tr> </td> Rosa Ponselle</td> Lean on diaphragm, push out</td> “Appoggiare… lean on it… Always push out with the stomach and abdomen during the phrase, not in”</td></tr> </td> Beverly Sills</td> Both diaphragm AND larynx</td> “How does one exist without the other?… I use that ‘sitting on the diaphragm’ to produce the feeling that I want in my throat”</td></tr> THROAT SPACE</td> John Alexander</td> Same space throughout range</td> “I try to keep as much space as possible, but always the same throughout the entire range of my voice”</td></tr> </td> Sherrill Milnes</td> More space going higher</td> “More spacing… More lift of the soft palate as you’re going higher, and… more opening in the throat”</td></tr> </td> Luciano Pavarotti</td> Less space in passaggio</td> “I think I give less space when I go through the passaggio, and then more space after I’ve left it”</td></tr> </td> Placido Domingo</td> Horizontal to vertical shift</td> “You are having a horizontal sound… to think vertical, in a way… up and down at the same time”</td></tr> OPEN THROAT</td> Cornell MacNeil</td> Meaningless concept</td> “It doesn’t mean anything to me… I think it’s nonsense!”</td></tr> </td> Paul Plishka</td> No sensation in throat at all</td> “I don’t know. I know the vocal cords are in the throat, but when I’m singing I have no sensation in my throat at all”</td></tr> </td> Magda Olivero</td> Don’t force larynx down</td> “If you teach a young student to push down the larynx… goodbye! That he should do automatically”</td></tr> LARYNX POSITION</td> Bonaldo Giaiotti</td> Firmly lower it all the way</td> “He demonstrated… breathing deeply, which caused his Adam’s apple to descend firmly all the way down”</td></tr> </td> Dr. Leo Reckford</td> Minimal movement only (millimeters)</td> “The larynx should make minimal excursions up or down… measured in millimeters… It’s definitely wrong!”</td></tr> </td> Louis Quilico</td> Must bring it down independently</td> “To bring down the larynx is not a natural thing… Yes!” (demonstrated independent up/down jiggling)</td></tr> </td> Pablo Elvira</td> Never pull it down</td> “The larynx should never be pulled down… You should never feel a down pull”</td></tr> </td> Sherrill Milnes</td> Low larynx equals open throat</td> “Open throat… low larynx… Those are in a sense the same thing. If your throat is open your larynx is low”</td></tr> </td> Cornell MacNeil</td> Don’t think about it at all</td> “As soon as you pull something somewhere, you’ve already created a tension”</td></tr> </td> James McCracken</td> Larynx slightly down</td> “Do we agree that an open throat has to do with the larynx being slightly down? Yep”</td></tr> </td> Régine Crespin</td> Beginning of yawn sensation</td> “The beginning of the yawn… Be suddenly, quickly astonished… Start breathing by the nose, and be astonished”</td></tr> PASSAGGIO</td> Luciano Pavarotti</td> Squeeze intensely for years</td> “It took me six years… you become cyanotic, red in the face… you must really make the voice more squeezed”</td></tr> </td> John Alexander</td> Don’t think about it</td> “The less we think about passaggio, the better off we are”</td></tr> </td> Marilyn Horne</td> Squeeze</td> “We really tightened up the middle and especially the passaggio. We squeezed on that passaggio”</td></tr> </tbody></table> I spot-checked some entries and confirmed that the quotes I looked at were accurately sourced from the underlying interviews. (I’ve also fixed up some entries manually, and see other problems that I’m not going to bother fixing right now since this is just a quick post.) Here are some specific ways in which the analysis above, and others like it, can be incomplete: Non-exhaustive: There may be correlated omissions due to bias in the combination of LLM and prompt.</li> Incorrect data: LLMs generate data that can be hard to verify and the effort required to verify a claim often exceeds the effort required to generate the claim. As Jonathan Swift said, “Falsehood flies, and the Truth comes limping after it.” This can be a big deal</a>.</li> “Wrong” topics: Topics are proxies for the underlying data and ideally they “carve nature at its joints”. But the appropriateness of topics depends on the task at hand – for example, if you’re interested in the underlying factors that drive agreement or disagreement, the LLM-chosen topics might make it harder or easier to see those factors.</li> </ul> Nonetheless, the level of synthesis here is pretty impressive to me. While the result isn’t of particularly high quality, it nonetheless captures some real patterns from the underlying text that are worth investigating further. A real analysis would take orders of magnitude more time and this exploration at least shows that a table like the above is an interesting way to look at the data. I think there’s a huge amount of potential in this kind of ad-hoc, task-specific dimensionality reduction, particularly for books and other rich source material, where the same knowledge can be projected into many different structures from which it is possible to much more easily make surprising connections. Appendix: Amusing quotes</h3> It’s like Sir Thomas Beecham, who said, ‘Most sopranos sound like they’ve been living on seaweed.’ </blockquote> Jerome Hines: “Take away my extraordinary size and physique, take away my handsome face and my glorious voice, and what have you got?” Jan Peerce: “Yeah, what have you got?” Hines: “Jan Peerce.” </blockquote> Louis sang to calm 149 cows and 18 chickens on a cattle boat crossing the Atlantic. When he finished his recital, “there was a breathless hush… not the sound of a single handclap. Cows don’t clap.” </blockquote> LLMs for Nominative Determinism 2025-10-12T20:10:26-04:00 As a long-time fan of names</a> and nominative determinism</a>, I wanted to know how well language models could assess the fit between a person’s name and their chosen profession. To investigate, I gathered a list of 1,000 scientists and asked Gemini</a> to score each one on how strongly their name connected with their chosen field of study. Gemini came up with some obvious names such as Jules Angst</a> the mood researcher, Storm Field</a> the television meteorologist, and Russell Brain, 1st Baron Brain</a>, who studied brains. But it also claimed some amusing and more puzzling connections, insisting that Paul Mockapetris</a>’s name was directly connected to his invention of DNS</a>: The name ‘Mockapetris’ has a phonetic similarity to ‘name’, particularly the ‘mock’ sound, which aligns directly with his invention of the Domain Name System (DNS). </blockquote> His name, ‘Mockapetris’, contains ‘petris’ which sounds similar to ‘petrify’ or ‘petrifies’, evoking a sense of making something permanent or foundational, which aligns with his creation of a core internet system. </blockquote> The name ‘Mockapetris’ has a phonetic resemblance to ‘mocked petris’, which could be interpreted as a playful or abstract reference to ‘mocking’ or manipulating ‘petris’ (perhaps stones or structures, analogous to data structures). </blockquote> Then I realized I’d been querying Google’s smallest model, Gemini Flash Lite, rather than their premier model, Gemini Pro. Oops. A few more amusing ones: Gian Francesco Malfatti</a>, mathematician: The surname ‘Malfatti’ translates from Italian to ‘badly made’ or ‘ill-made’. His most famous work is the ‘Malfatti problem,’ in which he posed the problem of packing three circles of maximal area into a triangle. He conjectured a specific, aesthetically pleasing arrangement of three mutually tangent circles. However, his conjecture was later proven to be incorrect; it is not always the optimal solution. Therefore, Malfatti’s own proposed solution to his namesake problem was, quite literally, ‘badly made’, creating an exceptionally strong and ironic case of nominative determinism. </blockquote> Brian E. Dalrymple</a>, inventor of techniques for fingerprint detection: The surname ‘Dalrymple’ bears a striking phonetic resemblance to the phrase ‘dull rimple’. ‘Rimple’ is an archaic but valid term for a wrinkle or fold, which is an excellent description of the ridges that constitute a fingerprint. Brian Dalrymple’s primary contribution to forensic science was developing techniques to detect latent fingerprints, which are faint, indistinct, or ‘dull’ by nature. Therefore, his name, ‘Dalrymple’, can be interpreted as describing a ‘dull rimple’ or a faint fingerprint, the very subject of his groundbreaking work. </blockquote> Benoit B. Mandelbrot</a> was given a high score, dubiously justified: The name Mandelbrot is of German/Yiddish origin, where ‘Mandel’ means ‘almond’ and ‘brot’ means ‘bread’. The primary object of study for Benoit Mandelbrot was the Mandelbrot set, the most famous of all fractals. The main body of the Mandelbrot set is a cardioid which is distinctly almond-shaped. Therefore, Mandelbrot, the father of fractal geometry, is most famous for studying an ‘almond bread’ set that has an ‘almond’ shape. </blockquote> I’d say instead that Benoit Mandelbrot is a great example of reverse nominative determinism: following his discovery of fractals, he gave himself a recursive middle initial: Benoît B. Mandelbrot (he added the middle initial himself, though it does not stand for a middle name) was born on Nov. 20, 1924, to a Lithuanian Jewish family in Warsaw. — New York Times</a>, October 16, 2010 </blockquote> Tech Notes on Podcast Vibes 2025-03-15T19:24:39-04:00 I’m winding down my podcast project and wanted to write a note about some of the things it taught me. In short, all of the major tech choices in this project worked out really well, and I’ll probably use this toolkit to prototype future LLM-based processing pipelines. Go</h2> Go</a> is a great choice for rapid prototyping. I’ve found it particularly useful for side projects since its simplicity makes it easy to context-switch from the other languages I use regularly. Go was a particularly good fit for this project since processing audio with APIs involved making lots of network requests and interfacing the database without doing much compute. Why not Rust? I’ve done side projects in Rust before and while it is a very powerful tool, it has a lot of mental overhead compared Go, and the type system inhibits certain forms of exploratory programming that I find very valuable. Postgres, PGX, and Postico</h2> I used Postgres</a> for the database with PGX</a> as the driver. PGX Top to Bottom</a> is a talk by the creator of PGX which explains the layered structure of the library, and taught me about a few useful API functions like CollectRows</code></a>, BeginFunc</code></a>, and CopyFrom</code></a>. This was also my first time trying out Postico</a>, a user interface to Postgres, which I found to be very well-designed. It made it easy to visually inspect the data in my tables and interactively edit schema definitions. River</h2> One lesson from my years at startups is to never build your own job execution engine if you can avoid it. After looking at various options I decided to try River</a>, an open-source job queueing library, and am very happy I did. It is made by experts with good taste and comes with nice documentation and a well-designed API that exposes convenient abstractions for defining and scheduling jobs. River also comes with a web-based user interface that let me easily monitor execution and inspect job-specific error logs, which made tracking down processing bugs a lot easier since the UI presents the error in the context of other job information, such as its input arguments, total runtime, and number of retries. DeepGram</h2> I used Deepgram</a> to transcribe podcasts. They give you $200 of free credits and the transcriptions are surprisingly good. There are cheaper options, but Deepgram is very convenient. Rather than uploading the audio to Deepgram myself I send them a link to the podcast audio to avoid the extra round-trip. The API endpint I used for this was /listen</a>: https://api.deepgram.com/v1/listen?punctuate=true&paragraphs=true&utterances=true&diarize=true&language=en&summarize=v2&topics=true&model=nova-2 By default, DeepGram will train on any audio you upload in return for a pricing discount. This was not a problem for me since the podcasts I was transcribing are already public, but might be a poor tradeoff for other use cases like personal voice notes. If you want, you can opt out</a> by adding a specific query parameter to your request. Claude</h2> Claude did an excellent job at extracting emotionally-laden subjects from the podcast transcripts. The fact that this kind of AI processing is now possible and affordable opens up a lot of opportunities for fun analytics projects. A few caveats: While Claude was able to accurately identify subject-emotion pairs, one thing I don’t have as good a sense of is how many subjects were omitted from the extraction since I only did a limited number of spot checks and never created a manually-constructed dataset for comparison, relying instead on my intuitive sense of whether the extracted data was reasonable. I noticed was that Claude would produce a similar number of results regardless of the length of the provided transcript snippet. I worked around this by feeding it the transcript in small overlapping chunks and de-duplicated the results using a vector similarity heuristic. Claude’s citation feature</a> launched while I was working on this project and turned out to be a great fit for connecting the emotional inferences back to the supporting text. Previous posts in this series: Note: Podcast Vibes</a></li> Podcast Vibes Prototyping</a></li> Podcast Vibes Presentation</a></li> </ul> Podcast Vibes Presentation 2025-02-27T21:21:18-05:00 I gave a talk today about my podcast analysis project (previously: Note: Podcast Vibes</a>, Podcast Vibes Prototyping</a>), connecting it to some visualization design work from over a decade ago which explored ways to visualize and organize text corpora with vector-based embedding models. I’ve put annotated slides up here</a>. This was in 2013 when I was working at an small machine learning startup spun out of the MIT Media Lab, and AI research was just beginning to be taken over by deep learning. I remember being amused when a team member referred to this phenomenon as deep lemming. Our main visualization was called a concept cloud and it used semantic vectors to make a more meaningful word cloud visualization, in which not only the size but also position & color of the words was used to convey the structure of conceptual relationships in the underlying text. This work was done in collaboration with many people on the team, including Elia Robyn Lake</a>, Jason Alonso, Ken Arnold</a>, Avril Kenney, Christina Laverentz, Alice Kaanta, and Andrew Lin. Writing about this brought back memories of the infamous Ass Headache Problem, and the time when our stemming pipeline thought that “Emily” was an adverb and “Coca Cola” was the plural of a singular “Coca Colon”… Language modeling has come a long way. XOR Doubly-Linked List 2025-02-06T23:35:13-05:00 A neat idea I learned today: You can make a doubly-linked list that only needs to store a single pointer – the xor</code> of the prev</code> and next</code> pointers – to allow traversal of the list from either direction. When moving forward we xor</code> current ptr with prev</code>. Moving backward xor</code> with next</code> to get prev</code>. That it. How do we get prev</code>/next</code> for xor</code>? The only way to get in the middle of the list is using iterator. Which can hold one extra pointer. For the first/last nodes extra pointer is NULL</code>. — numba888 on Hacker News</a> </blockquote> This concept has a Wikipedia page</a>. Podcast Vibes Prototyping 2025-02-02T21:39:36-05:00 I’m considering turning my little research note on podcast vibes</a> into a bit more of a product. The idea would be to track the daily pulse of the podcast ecosystem in terms of the subjects it discusses and the emotions it puts out. This weekend I played around with a few visualization ideas for looking at the emotional subjects for a single episode at a time. The picture below shows all of the subjects discussed on a recent podcast interview with a bean enthusiast</a>, with subjects discussed positively colored green and those discussed negatively colored red. The annotations are picked at regular samples across the emotional spectrum so that the labels don’t overlap. I think it would be better to instead draw attention to the most salient discussion points. Here’s how the same kind of picture looks for a longer podcast episode: And here’s one for a podcast with a much more negative slant. Amusingly, the most positive segments of this one are the ads. While looking at the transcript DeepSeek R1 complimented the host on his ability to smoothly and undetectably transition from the interview into an advertising segment. </h1> Here’s how it might look to lay out the dots on a horizontal timeline, with negatives segmented from positives. Anyway, just some early explorations. Still getting used to publishing this stuff in such an early state, but here we go. Note: Podcast Vibes 2025-01-26T23:02:05-05:00 Here’s a research note</a> on my recent explorations into using LLMs to analyze the emotional content of podcasts. I’m amazed that we have technology that can do this so well. This project supports my view that one of the most useful features of language models is that they can function as transducers from messy reality to a structured format that can then be dealt with using standard tools that are more efficient, controllable, and well-understood. It’s also a reminder of just how open the design space is for projects like this, how valuable good tooling for data pipelines is, and the importance of taste. The Structure of the Wavelet Tree 2025-01-21T15:43:37-05:00 This is a follow-up to Wavelet Matrix Construction</a>. (I’m working on a larger post about the wavelet matrix and writing and publishing extremely rough drafts as I go.) To understand the wavelet matrix it can be useful to understand the wavelet tree first, so here’s a quick primer. As I’ve mentioned before, I think of the wavelet tree (and matrix) as representing the process of sorting a sequence of integer symbols. Let’s take an example. Imagine that our integer symbols are picked from a universe of size 2³, consisting of the numbers from 0 to 7. The wavelet tree represents a quicksort</a> whose pivots are all powers of two. At each level of recursion, the value space is partitioned into two halves based on the universe size, ie. into a lower half containing the numbers 0 to 3, and an upper half containing the numbers 4 to 7. After this first step, each half is recursively sorted using the same approach, terminating when the halves contain a single integer each. You can see this in the diagram below, where each node is labeled with the range of the as-yet-unsorted integers inside it. Each level of the diagram corresponds to the next stage of the sorting process. The full sequence of numbers starts in its original order on the top level, made up of symbols from 0 to 7. On the next level those numbers are partitioned into two sublists, with the one on the left containing all of the numbers from 0 to 3 in their original order, and the one on the right containing all numbers from 4 to 7, in their original order. Then the same process happens again, with the left (0 to 3) node further partitioned into two child nodes, one from 0 to 1 and another from 2 to 3. Then we do this again and arrive at the fully-sorted sequence on the bottom level. The diagram above shows the process abstractly without any particular sequence of numbers, but let’s see how this would actually work in practice. Here’s a visualization that shows the incremental sorting process for the sequence 5 2 1 3 7 2 6 1 5 2</code>: We start off with the full sequence in its original order on the top layer, and end up with those same numbers sorted in ascending order on the bottom. In between, you see the process of recursive partitioning, where the numbers “slide” down either the left or the right arrow, depending on whether they belong to the bottom or the top half of the universe. Now, because the entire sequence is represented in full on each level of the sort, you might think this is a very expensive way to represent the data. But in fact, this is the beautiful trick: the wavelet matrix stores not the boxes, but the arrows! Here’s the same diagram again, this time with the numbers in parentheses notating the data that actually gets written into the wavelet matrix as individual 0- and 1-bits. The drawing tool</a> I’m using does not allow me to color the individual zeros and ones red or blue but you should think of the the zeros as colored blue, indicating that the number goes down and to the left along the blue arrow, and ones as red, indicating that the number goes down and to the right following the red arrow. The data in the wavelet matrix consists of a bit vector per level containing the bits in parentheses, With the one small tweak that, rather than writing the bits down from left to right as they’re shown here, we first write down the bits for all the left nodes, followed by the bits for all the right nodes. If those bits are all you’ve got it might look like we’ve scrambled the numbers beyond all possibility of repair. But we will soon see that it’s easy to reconstruct the original data from this new representation. The reason for this is that at each level we’ve actually written out bits from the numbers themselves. For example, the number 5 is represented in binary as 101</code>, and those are the exact bits written out for each 5 in the sequence, in that order – a 1-bit on the top layer, a 0-bit on the middle layer, and another 1-bit on the bottom layer. (I’d like to make an animated version of this diagram later, but for now I’m happy since this is already quite an improvement from my last post.) More to come. Appendix: Scraps</h3> Here are some scraps of visualizations I’m working on that are quick sketches to see whether an idea works. This first one has some promise: This tree view is an interesting idea, too, and inspired the boxy diagrams below: The next few are a bit misleading because it is not the case that the bottom half of each node sorts to the left: I experimented with showing the flow from the full parent node, but that seems even less enlightening, and I think the arrows in the main post are better illustrations of the idea. Wavelet Matrix Construction 2025-01-20T18:32:05-05:00 This is a follow-up post to Enter the Wavelet Matrix</a>. Compared to my previous entries this one is much more of a stream-of-consciousness that’s not really designed for anyone else to read (though hopefully on the way to a much more coherent and comprehensive explanation!) Sidenote: This is very much an early draft, and not really understandable on its own, but I’m practicing the art of forging ahead with partial work. The wavelet matrix can be constructed using an extremely simple algorithm, though explaining the meaning of the resulting structure will take a little bit more work. The wavelet matrix is a way to represent a sequence of integers, which we will call symbols, by representing a sorting process – the symbols are incrementally sorted by their successive bits, from high to low, using a stable bucket sort, and the incremental results are written down into the wavelet matrix structure, and later used for reconstituting the elements or performing other algorithms such as ranged quantile queries. Here’s an image illustrating the construction of a wavelet matrix for the sequence 1 4 2 3 7</code>. The sequence starts out in its original order on the top layer and moves down incrementally through a series of bitwise bucket sorts that sort the elements into ‘left’ and ’right buckets based on whether the element has a 0-bit or 1-bit on that level. The The big bold numbers are the symbols</li> The gray numbers underneath are their bit encodings</li> This diagram represents three successive applications of a bucket sort, from high bits to low bits. The unsorted elements on the first row rearrange themselves until they’re in their final positions on the bottom row.</li> This visualization will need to be improved in many ways, but I’m trying to get better at publishing drafts, so here we are. Don’t re-use colors between the numbers and their bits, and the links which represent the data encoded by the matrix</li> Consider introducing a version of this that sorts according to the wavelet tree rule</li> </ul> </li> </ul> The data stored in the wavelet matrix is represented by the “pipes” in between the numbers rather than the numbers themselves. This is what I mean when I say the wavelet matrix represents the process of sorting. It records whether each element “went left” or “went right” in the successive applications of the bucket sort. The idea behind this picture is that all elements that go left appear before the elements that go right. So for example, the numbers 1, 2, and 3 have a zero for their top bit, and therefore appear on the left of the second row. The numbers 4 and 7 have a 1 in that top bit, and therefore go right and appear at the end. The numbers that went left are indicated by gray pipes, and the ones right are indicated by black pipes. The next sorting is based purely on the second bit. The numbers 2 and 3 have a 1 in that position, so they go right. The same process occurs again on the third layer. Note how 7 was the maximal element on each row indicated and therefore went right on each level. This is my sixth attempt to try to explain this process, and clearly it’s not working very well, but I will forge ahead, because I think this section does capture something about the practical algorithm I use for large-alphabet wavelet matrices. I’m beginning to think it’s probably best to start with an introduction to the wavelet tree where the effects of the sort are much more salient, since the bottom row ends up with the symbols sorted by value. In the wavelet matrix, the symbols end up sorted by the bit-reversal of their value. Actually… Here’s a visualization of the wavelet tree representation of the sequence 7 6 5 4 3 2 1 </code></pre> The wavelet tree has a nice hierarchical structure where all of the children stay underneath their parent node, though this is not obvious from these matrix-style visualizations that lose the node borders. And here’s the wavelet matrix version: Here we can see that each layer is sorted independently of the previous – ie. other than the stability of the sort, whether the elements on each layer go either left or right purely based on the bit at that level, rather than taking into account the previous levels too (which is what gives the preceding picture its tree structure). As a side note, the reason the wavelet matrix is an improvement over the wavelet tree is because the wavelet tree requires you to store the node offsets at each level, which scales linearly with the size of the maximum symbol. Without this information, it becomes impossible to navigate from one level of the wavelet tree to the next. The wavelet matrix is based on this ingenious idea of ordering all of the left children before all of the right children on each level, rather than keeping the child nodes “underneath” the parent. This helps because when you’re on a particular level you can compute the number of left children that precede you and the number of right children that precede you with a simple rank query on the bit vector holding that level. Therefore, navigating to a left child involves looking at the number of zeros to your left, and jumping to that location on the next level. Navigating to the right child involves looking at the number of ones to your left and jumping to that location on the next level (plus the total number of zeros, ie. left children, since all the left children precede the right children). It’s very clever but hard to explain. Hopefully I can find a nice way to present all of this. References</h2> See Translating Between Wavelet Tree and Wavelet Matrix Construction</a> for more details on the relationship between the wavelet tree and wavelet matrix, including a discussion of role bit reversal plays in the translation.</li> See Practical wavelet tree construction</a> (2021) for description of the bucket sort algorithm visualized above (page 31).</li> </ul> Enter the Wavelet Matrix 2025-01-15T20:32:08-05:00 Imagine you have an array of integers and want to preprocess it so that you can answer quantile queries, such as computing the median or the 99th percentile, in constant time. One possible solution is to sort the array and then answer the quantile queries by looking up the relevant entries in the array. But what if, instead, I asked you to preprocess the array for range quantile queries, where rather than computing the quantile over the entire array, you want to compute quantiles for any sub-slice arr[i..j]</code> in constant time? Computing range quantiles involves combining information from both the original order and the sorted order, so sorting the entire sequence in advance no longer suffices. And the quickselect</a> algorithm, which is often used for computing specific quantiles directly from an unsorted array, does not help us either, since rather than being constant-time its runtime depends on the length of the slice of the array. Enter the wavelet matrix, a data structure that represents the process of sorting. The wavelet matrix reorganizes the data in the array in a way that allows answering range quantile queries, and a lot more, in constant time: You can access any element of the original sequence in its original order in constant time.</li> You can access any element of any slice of the array in sorted order in constant time.</li> You can count the number of times any element occurs in any slice of the array in constant time, which is known as a rank query.</li> You can count the number of times all elements occur in any slice of the array in constant time with respect to the length of the slice, with the runtime scaling with the number of unique elements that are counted.</li> You can find the location of any occurrence of any element of the array in close-to-constant time, which is known as a select query.</li> You can find all of the elements that occur more than X% of the time, in any slice of the array, in an amount of time that depends only on X and not the length of the slice. This is known as a k-majority query.</li> </ul> This all sounds very attractive, but there are some trade-offs. The main downsides are that the data representation has some space overhead, on the order of 10-20% of the space required to store the bits of the integers themselves, and a high constant time factor due to the distributed nature of the data representation. Internally, the wavelet matrix is composed of an array of bit vectors, each representing the bit planes of the original integers, which means that the bits of any specific element are scattered throughout memory and need to be reassembled as queries are executed, which hurts performance due to a lack of data locality and somewhat unpredictable access patterns, which can be somewhat alleviated by batching. I’d like to explain more about the mechanics of how this thing works in the future, but for now there’s a great paper written by Gonzalo Navarro, one of the co-discoverers of the wavelet matrix, which serves as a great introduction of the basic idea along with a whirlwind of the literature around it. The article is called Wavelet Trees for All</a> because the wavelet matrix is actually an optimized data representation for a conceptual tree structure called the wavelet tree. The word wavelet hints at the fact that the integers are decomposed into something like their frequency components, but no actual wavelets are used in the process of making a wavelet tree. I’ll leave you with a picture I made a while back of an algorithm that constructs a wavelet tree through a series of incremental stable sorts. Further reading</h2> A nice description of wavelet trees</a> by Alex Bowe</li> Original wavelet matrix paper</a> by Francisco Claude and Gonzalo Navarro</li> Paper: Practical Wavelet Tree Construction</a></li> Paper: New algorithms on wavelet trees and applications to information retrieval</a></li> Paper: Range Quantile Queries: Another Virtue of Wavelet Trees</a></li> </ul> Gilbert Curve 2025-01-10T17:03:28-05:00 The Hilbert Curve</a> is space-filling fractal that, in the limit, will fill any power-of-two-sized square with an increasingly dense squiggle that fills the plane. The Gilbert Curve</a> is a generalization of the Hilbert curve to rectangular spaces. The main code is in Python but there are also implementations in JavaScript</a> and C</a>. Related: A very fast C++ Hilbert curve implementation</a> in 2D and 3D (details here</a> and here</a>). LLMs Make Context More Valuable 2024-12-31T16:49:48-05:00 Reader-side LLMs enable information filtering and synthesis to be done on behalf of the reader. This increases the value of curated contextual information, which can now be sifted through efficiently, and automatically integrated with the main text. LLMs have issues with provenance and reliability, though, which is why I’m interested in author-provided context, which not only serves as a trust signal but also tames hallucinations, making catastrophic failure less likely. Existing media formats have lightweight interaction mechanisms such as footnotes and annotations which augment a text with additional information. However, these interactions are usually limited to a small set of pre-determined annotations and footnotes, and the author likely had ideas they decided were not worth incorporating into the final work. Similarly, commented code only contains a fragment of its relevant context and rarely explains every design decision. Some design decisions can be inferred from the surrounding context of the code itself, but other information cannot be inferred, and has to come from knowledge of the broader system and its environment. I wonder what might be a good testing ground for these ideas. One thought is that large programming projects often have public design discussions about the broader tradeoffs involved in particular architectural or feature decisions. And as far as other kinds of writing, the Bible is probably one of the most analyzed and annotated</a> pieces of text in human history, though that one’s a bit out of my area of expertise… See also: Advanced Essay</a> Splash Color Format 2024-12-25T17:00:51-05:00 Splash</a> is a color format that reduces the space of choices in a playful way and lends itself to interesting UI design possibilities. Each color is specified as a three-digit number with one digit per color channel, for a total of one thousand possible colors. For example, 407</code> stands for 4/9 units of red, 0/9 units of green, and 7/9 units of blue. Voice Note Browser 2024-12-24T19:37:05-05:00 I like thinking through ideas on a walk and sometimes record voice notes to myself as I go. The result is a collection of related notes, and I wanted a way to see their transcripts together on one page. But to my surprise, couldn’t find any apps that show more than a single transcript at a time. So I built a little web app to do this using the newly-added transcripts</a> in Apple’s Voice Memos, showing the transcripts from the couple of days’ worth of notes in an simple list. I used an Apple Shortcut to copy out the most recent week of voice memos. I originally wanted to use rsync</code>, but ran into permission issues and did not feel sufficiently bold to give my terminal app “Full Disk Access”.</li> </ul> </li> I used SvelteKit to make the app. The transcript is stored as JSON alongside the audio in a metadata section. Here’s how I parsed it, treating the audio file contents as a utf-8 string:const jsonStart = text.indexOf(`{"attributedString"`); if (jsonStart > -1) { const jsonEnd = text.indexOf("\x00", jsonStart); const slice = text.slice(jsonStart, jsonEnd); const transcript = JSON.parse(slice); } </code></pre> </li> I learned that SvelteKit can stream promises</a>, making it easy to load individual voice notes asynchronously and incrementally. The index page returns an array of Promise</code>s representing individual notes, and the frontend shows data as it arrives. This was very cool.</li> </ul> </li> </ul> This came together quickly and I’m really happy with it – it’s nice when something so simple can be so useful. A few ideas for future work: Provenance: Make it easy to listen to the underlying audio for the cases when the transcription doesn’t get things right. For example, making a highlight on the page should pop up a little tooltip that lets you listen to the audio corresponding to that part of the transcript.</li> </ul> </li> Structured extraction: Use LLMs to extract to-dos so that the ideas are easier for me to act upon. It would be nice if there was a sidebar with a bunch of to-do items next to each note.</li> There might be other kinds of structured extraction that would also be useful. I could imagine a set of local language models making parallel passes over each note with each focused on extracting a particular kind of structure.</li> </ul> </li> </ul> Hyparquet 2024-12-18T21:22:19-05:00 I came across a nice dependency-free library today, hyparquet</a>, for parsing Parquet files in JavaScript. It supports many compression formats</a> including Snappy and Zstd, and reads Parquet files in chunks using range requests</a> rather than loading the entire file. You can read just the metadata, or particular rows and columns. It’s a good companion to Jeff Heer’s flechette</a>, a lightweight library for reading and writing Arrow files. (API reference</a>) See also: fzstd</a>, a small library for decompressing zstd files. Advanced Essay 2024-12-15T22:07:32-05:00 In the 1970s, decades before computers began to compete at the highest levels of chess, a new variant of the game was invented, called Advanced Chess</a>. The idea was that instead of a human player competing against a computer, the human and machine would play together on the same side. Rust-analyzer can expand macros 2024-12-13T13:10:12-05:00 If you use an editor with Rust’s LSP integration</a>, you can put your cursor on a particular derive</code> macro, such as Clone</code> in #[derive(Clone)]</code>, or an the name of a macro application, such as matches!</code> in matches(foo, bar)</code>, and select “Expand Macros Recursively”, which will open up a side buffer showing the expanded code from that macro. This is extraordinarily useful when trying to understand what code is generated from a derive, or when debugging an issue with your own macros. I discovered this through a comment made by David Barsky</a> in the Rust Zulip</a>. Formatting Code with a Git Hook 2024-12-11T18:21:38-05:00 Here’s a git pre-commit hook</a> to auto-format code on commit. It’s useful if you’re working with code whose formatting guidelines differ from the ones you’ve configured in your code editor. #!/bin/bash # .git/hooks/pre-commit # Store list of staged files that match your target pattern. # The `git diff` command returns a list of staged files. files=$(git diff --cached --name-only | grep -E '\.(js|ts|svelte)$') if [ -n "$files" ]; then pnpm format # Add all of the previously staged files back to the staging area # to check in the formatted files. git add $files fi # Exit with an error when there are no longer any staged changes, # since otherwise git will create an empty commit. if git diff --cached --quiet; then echo "Error: After formatting, there are no longer any files are staged for commit" >&2 exit 1 fi </code></pre> Quicker Netlify Deploys 2024-12-08T11:16:20-05:00 I use Netlify</a> to host small static sites like this one. It does its job well and is very convenient to use. But I noticed that since the standard way of setting things up is to link Netlify directly with your Git repo, deploys could take a while. When you push your code, Netlify builds the site on their own infrastructure before deploying it. I recently came across</a> a very neat alternative where you can build your site locally and deploy it to Netlify directly with a single HTTP request. This is really convenient when you don’t need the overhead that comes with more careful deploy management. The link above does a good job of describing the process, but the upshot is that the following command is all you need to have a deploy running live just a few seconds later. curl -H "Content-Type: application/zip" \ -H "Authorization: Bearer PERSONAL_ACCESS_TOKEN_FROM_STEP_1" \ --data-binary "@FILE_NAME.zip" \ https://api.netlify.com/api/v1/sites/SITE_NAME.netlify.com/deploys </code></pre> Right now I use it like this (these build commands go in a justfile</code></a>) to ensure that there’s a corresponding commit for every deployed version: pub: no-uncommitted-changes build @rm -f dist.zip @zip -q -r dist.zip dist; @curl -H "Content-Type: application/zip" \ -H "Authorization: Bearer $(cat .netlify-token)" \ --data-binary "@dist.zip" \ https://api.netlify.com/api/v1/sites/what.yuri.is/deploys | jq @rm -f dist.zip @echo "" @echo "Deployed!" # Halts with an error if the repository contains uncommitted changes no-uncommitted-changes: @ git diff --exit-code > /dev/null || (echo "Please commit changes to the following files before proceeding:" && git status --short) build: zola build </code></pre> Pareto Frontier 2024-10-25T22:21:38-04:00 Status: This was written in a hurry. I should find some way to label these with how much effort went into them / how much attention should be put on them… The Pareto frontier can help you find a subset of your data points that are the ones that represent trade-offs between maximizing the various dimensions. Imagine you’re looking at data about thousands of cars, and for each car you know its cost, mileage, and top speed. Maybe you’re trying to get a general sense of which cars perform the best, but this can be difficult when there are multiple attributes along which they can be compared. All else being equal, though, you want a lower cost, higher mileage, and higher top speed, and it could be the case that some cars are strictly better than others on all three of these aspects. That set of those points is called the Pareto frontier</a>. The Pareto frontier will reveal the points that represent interesting trade-offs: no point on the frontier is strictly better than any other. Of course, there may be important real-world dimensions not captured in your data – perhaps you also care about the size of the car, for example – and in that case, looking at only the frontier of cost, mileage, and top speed can be a dangerous thing since it would cause you to ignore potentially relevant points along those other dimensions or skew your data (since size is correlated with all three dimensions). But if you can quantify these “hidden” aspects and add them to your data, the new higher-dimensional frontier might prove to be a source of insight. Implementation</h2> Here’s a simple JavaScript implementation of a Pareto frontier algorithm from the paper Fast Linear Expected-Time Algorithms for Computing Maxima and Convex Hulls.</a>, coauthored by Jon Bentley</a> of Programming Pearls fame. It takes an array of points, each represented by an array, and returns the indices of the points that are on the “max” Pareto frontier (intuitively, these points are generally “strictly larger” than the others when considering all dimensions). // Return the indices of points on the max Pareto frontier (maxima of a point set) using algorithm // M3 from "Fast Linear Expected-Time Algorithms for Computing Maxima and Convex Hulls". // Note that if multiple equal points are potentially on the frontier, only one will be returned. function paretoFrontier(points) { const n = points.length; if (n === 0) return []; let topMax = 0; const max = [0]; for (let i = 1; i < n; i++) { let j = 0; while (j <= topMax) { if (dominates(points[max[j]], points[i])) { // move max[j] to front of max[1..j] const [move] = max.splice(j, 1); max.unshift(move); j = topMax + 2; } else if (dominates(points[i], points[max[j]])) { // move max[j] to max[topMax] const [move] = max.splice(j, 1); max.splice(topMax, 0, move); topMax -= 1; } else if (equals(points[i], points[max[j]])) { j = topMax + 2; } else { // pts i, max[k] incomparable j += 1; } } if (j === topMax + 1) { topMax += 1; max[topMax] = i; } } return max.slice(0, topMax + 1); } function equals(p1, p2) { const k = p1.length; for (let i = 0; i < k; i++) { if (p1[i] !== p2[i]) return false; } return true; } // p1 dominates p2 if it is greater than it in some components and equal // in the remainder (incomparable in no components). Allows for dimensions // s. t. neither p1 nor p2 is less than the other, eg. if the values // are distinct Objects. function dominates(p1, p2) { const k = p1.length; let i = 0; while (i < k && p1[i] === p2[i]) i++; if (i === k) return false; // equal in all dimensions for (; i < k; i++) { // if p2 is greater than p1 in this dimension, then p1 does not dominate p2. if (p2[i] > p1[i]) return false; // if p2 is incomparable to p1 in this dimension, the p1 does not dominate p2. if (!(p2[i] < p1[i] || p2[i] === p1[i])) return false; } return true; // p1 dominates p2 } </code></pre> Cheerio for Web Scraping 2024-10-13T19:22:53-04:00 A great JavaScript library has recently reached 1.0: Cheerio</a>. It gives you a jQuery-like syntax for quickly parsing out content from an HTML string. I recently used it to parse out some data from a structured list using a nice high-level DSL that’s used by Cheerio’s extract</a> function. const $ = cheerio.load(htmlContent); // The page has many of these: // <div class="ts-segment"> // Jimmy Wales // <a href="https://youtube.com/watch?v=XXXX&t=5112">(01:23:45)</a> // Hmm? // </div> const result = $.extract({ segments: [{ selector: '.ts-segment', value: { name: '.ts-name', timestamp: { selector: '.ts-timestamp', }, href: { selector: '.ts-timestamp a', value: 'href' }, text: '.ts-text', }, }] }); </code></pre> The input HTML content contains many snippets like the one in the comment above. One nice thing about Cheerio’s extract function is that if parts of a composite value are missing, then the entries in each segment will remain undefined while the existing values are extracted. For example, if the name</code> is missing from a particular segment and its selector fails to match, then its timestamp</code>, href</code>, and text</code> will still be extracted. Running the extraction will return a JavaScript object with the key segments</code>, whose value is an array of objects each with a name</code>, timestamp</code>, href</code>, and text</code>, mirroring the shape of the input. One nice thing about this interface is its flexibility, which allows you to extract arbitrary properties at multiple levels of a nested tree-like query. The docs</a> for the function aren’t very informative as to what can be extracted but this tutorial</a> explains many of the possibilities, including using a function as the extractor. Visualizing Sentences with Finite State Automata 2024-10-07T13:10:38-04:00 The National Electronic Injury Surveillance System</a> in the US collects data on product-related injuries. It has a dataset of emergency room visits, with granular information on over 8.5 million visits across 100 hospitals going back over 20 years. An interesting feature of the data is that each ER visit comes with a narrative story. For example: A 50YOF DROPPED A CRYSTAL BALL ON FOOT, SPRAINED FOOT </blockquote> PATIENT ROLLED OFF BED, HIT EAR ON DRESSER OR ON WOODEN KNOB SHAPED LIKE AN ELEPHANT HEAD AT HOME; EAR LAC. </blockquote> 2 NIGHTS AGO A FRIEND MICROWAVED A MARSHMALLOW AND PUT IT ON PT’S NECKPT HAS BURN TO NECK WITH BLISTERING </blockquote> One way to get a qualitative sense of what’s in this dataset is to do some thematic keyword searches. For example, for bicycles: PT FELL OFF BICYCLE AND HIT HEAD </blockquote> PT RIDING BICYCLE AND FELL AND LAC ARM </blockquote> 13YOBF FELL OFF BICYCLE;RADIAL HEAD FX </blockquote> Avocados: PT CUTTING AN AVOCADO AND LAC HAND ON KNIFE </blockquote> LAC FINGER ON KNIFE CUTTING AVOCADO </blockquote> CUT WITH KNIFE TO FINGER WHILE CUTTING AVOCADOSDX: LAC FINGER PT </blockquote> Cowboy boots: PT DANCING IN COWBOY BOOTS STRAINED RIGHT ANKLE </blockquote> PT. HAS FOOT DISFIGUREMENT FROM CHRONIC COWBOY BOOT WEARING. DX:FOOT DISFIGUREMENT </blockquote> PATIENT WAS ACCIDENTALLY KICKED IN RIBS PLAYING WITH FRIENDS WHOWORE COWBOY BOOTS. DX. RIGHT RIB CAGE STERNAL SOFT TISSUE. </blockquote> Banana peels: WALKING DOWN HANDICAP RAMP CARRYING BOXES AND STEPPED ON A BANANA PEELAND FELL STRAINING KNEE </blockquote> 32 YO FEMALE SLIPPED ON BANANA PEEL AND FELL TO FLOORKNEE STRAIN </blockquote> PT SLIPPED ON BANANA PEEL AND FELL ON FLOOR IN STORE </blockquote> I wanted to visually summarize the variation within sets of similar stories like these, and after tinkering a bit realized that this problem was very similar to a collaboration</a> I was a part of a while ago that focused on visualizing variations across a set of lede sentences in newspaper articles. One of the goals there was to reveal authorial/media bias, which led us to stay close to the surface texts since phrase ordering and word choice are part of how bias is expressed. In the ER visit stories, though, the texts are written in medicalese and the event semantics matter more than surface syntax, so I think it would be fine to dramatically restructure the text if doing so reveals interesting higher-level patterns. It’s also possible that visualizing something like the temporal sequence of events and their consequences directly will wind up being a good idea here. The story so far</h2> After getting the dataset into shape I spent some time learning about minimal finite state acceptors, which I thought could be a mechanism for merging similar sentence fragments together. FSAs compactly represent a set of strings by reusing suffixes wherever possible: Input: Yuri accidentally went to the store Ruby deliberately went to the store Output: Yuri accidentally ⤵ went to the store Ruby deliberately ⤴ </code></pre> This post</a> by Andrew Gallant has a great overview of FSA construction and usage. In the section on FSA construction</a> Andrew visualizes the algorithm for constructing a minimum finite-state acceptor for a given set of strings. I implemented the algorithm described in the post, which comes from a paper called Incremental Construction of Minimal Acyclic Finite State Automata</a> , and tried it out on some of the stories from the emergency room data. My plan was to see whether FSAs could be used to capture the variation in a curated set of sentences based on a search term, such AVOCADO or BOOTS. An issue that came up is that, since each node in an FSA represents a fixed set of strings, finite state automata will only compress strings if they share a suffix. For example, the following two sentences cannot be compressed using an FSA because they differ in their last word: Yuri accidentally went to the bar Ruby deliberately went to the store </code></pre> One way around this is to focus the compression on a local neighborhood around the word of interest (eg. WENT above) and to build two automata: one for the prefix leading up to the word, and another for the reversed suffix of the word. The reversal of the suffix makes it so that the “end” of the FSA is closest to the word of interest, which often leads to a more compact automaton. For example, here’s a pair of FSAs for BOOTS, visualizing local neighborhoods across 10 sentences: Here’s BOOTS with 20 sentences: And here’s BICYCLE with 25 sentences: It feels like there’s a glimmer of promise here, but I need to do some more thinking about how to proceed. One idea is to use LLMs to conform the sentences to a more standard structure, then build automata for each “semantic fragment” of the sentence, and visualize the result using something like a parallel coordinates plot with ordinal Y axes. Another thing I want to try is to identify collocations (multi-word phrases like “christmas tree”) and use those as the elements over which the FSA is built, rather than always simple-mindedly splitting on word boundaries. There’s also a lot of work to do on visual presentation. The current diagrams are just the raw output from Mermaid</a> and are hard to read. Perhaps there’s a layout algorithm that is a better match for this family of graphs. My suspicion is that, in the end, the visualization should look significantly less graph-like, with the FSA graph serving as more of an inner structure to guide layout decisions. Related</a>: First there is old work by Book and Chandra, with the following abstract. Summary. It is shown that for every finite-state automaton there exists an equivalent nondeterministic automaton with a planar state graph. However there exist finite-state automata with no equivalent deterministic automaton with a planar state graph. </blockquote> </blockquote> I wonder if it would be reasonable to “planarize” nonplanar FSA graphs by duplicating nodes and edges. I’d like to make it easier to read these from left to right without so much eye movement, and being able to lay sentences out without intersection seems like it would help. A few things this project reminds me of: The data is similar to that underlying some visualizations I worked on for large collections of distributed traces. For that effort we aggregated individual traces into a prefix tree, with edges representing service dependencies. (FSAs are compressed representations of prefix trees). One of the powerful ideas in that project was that we built service-centric views that merged together all of the prefix tree nodes for the target service in postprocessing, which is similar to what I did above in building the prefix trees for the prefix/suffix of a local window around the target word.</li> Both forms of merging result in ambiguous graphs. I think this is fine so long as there is a way to disambiguate interactively – it could let us scale to much larger sentence sets.</li> </ul> </li> This post on dominator trees</a>.</li> The Gyllenhaal Experiment</a>, which visualizes people’s attempts to spell a famous person’s name:</li> </ul> Creating a Visualization Library in Mathematica 2024-10-03T21:24:26-04:00 Status: First draft; written in a hurry. Publishing since I want to get better at publishing incremental work and nobody reads these things anyways. I recently came back to using Mathematica</a> as my tool of choice for early-stage prototyping and data exploration. Unlike many other language ecosystems, Mathematica comes with a lot built in, including many data visualization functions. However, plots are scattered throughout the landscape without a high-level system unifying them: every new chart lives in its own world, and there’s nothing resembling The Grammar of Graphics</a> unifying them. So I decided to see how much work it would take to create a more uniform and flexible plotting system like Plot</a>, my current tool of choice for browser-based visualizations, in Mathematica. It’s been a few days and I’ve made some progress and have the following picture to show for it: Which was made by this code: plot[ dot[ "x" -> data[[All, 1]], "y" -> data[[All, 2]], "stroke" -> data[[All, 5]]]] </code></pre> The library works along very similar lines to Observable Plot, which is the existing library I’m most familiar with, and directly inspired this effort. Technical notes</h2> The following was written in something of a hurry, and I plan to refine it later. The basic idea is that a plot is a collection of marks overlaid atop each other. (This plot has one mark, dot</code>.) A plot encodes meaning through visual channels, such as x or y or color. Each mark provides data to some number of channels. Based on this data, together with preferences provided by the user, the plot function comes up with an encoding for each channel that maps from the data domain to the visual domain (eg. taking a category label such as “a” and turning it into a color, or taking a numeric data value and turning it into a position along the x-axis. The usability of the library depends strongly on the amount of deduction the library can make for you, and its ability to add explanatory marks to the plot, such as legends and axes, so that the resulting picture can be made sense of. An interesting point that I had not sufficiently appreciated is the importance of representing scales in a way that allows you to make these explanatory marks. For example, the library needs to be able to tell when a log scale is used so that it can generate logarithmic axis ticks. Another example is that in Plot, the “shorthand” field specification syntax is what allows the library to know how to label your axes. A possibility in Mathematica is to use controlled evaluation for the same purpose. This also raises some questions about evaluation order. I was hoping to use regular marks to build the axis legends, but had to figure out a solution to the time-ordering problem: Normal marks determine the set of inputs for each channel, from which scales for each channel are inferred, and ticks generated. But axis marks need to know the tick values. My current solution is to allow a mark to be a function which will be invoked with scale information. Any points plotted by such marks will not be used to determine the domain of the plot. Another Plot design point that I hadn’t fully appreciated is that one reason it uses a z channel to allow eg. the line mark to make multiple lines is because sometimes one data column wants to be plotted as multiple curves, eg. with categorical time series, where you don’t want the user to need to reshape their data to calling your mark function. My implementation so far is about 250 lines of code, and doesn’t yet include the ability to make bar charts, does not include faceting, or support for something like Plot’s z channel. I’m also not yet sure about the best way to support interactive selection, highlighting, and animated transitions. But you can use continuous, ordinal, and categorical encodings to plot text, dots, and lines, with scale and tick inference and a composable mark system. We’ll see if my current design survives the addition of any of these new features. Part of the pleasure of using such a high-level programming language is that rewrites are cheap: when you know what you want to say, and if performance isn’t too much of a concern, you can express yourself very concisely. References</h2> Plot</a> – My favorite plotting library, from the folks behind d3</a>. Wolfram Videos: ggplot: A Grammar of Graphics for the Wolfram Language</a> – I came across this while writing this note. Haven’t watched it yet. Appendix: Code</h2> (* Names of all channels *) channelNames = {"x", "y", "r", "color", "opacity", "fx", "fy"}; (* Mapping from option names to channel names *) optionChannels = <|"x" -> "x", "x1" -> "x", "x2" -> "x", "y" -> "y", "y1" -> "y", "y2" -> "y", "r" -> "r", "stroke" -> "color", "fill" -> "color", "opacity" -> "opacity", "strokeOpacity" -> "opacity", "fillOpacity" -> "opacity", "fx" -> "fx", "fy" -> "fy"|>; (* Converts its argument to a list, if it isn't one already *) toList[x_] := If[ListQ[x], x, {x}] (* "Broadcasts" scalar values in an association to lists, so the \ result is tabular in structure. *) toTable[assoc_] := AssociationThread[ Keys[assoc] -> Map[toList, Transpose[Thread[Values[assoc]]]]] (* Normalization functions to rescale a domain to [0, 1] in the case \ of continuous and ordinal, or to [0, n] in the case of indexed *) continuous[domain_] := Rescale[#, MinMax[domain]] & indexed[domain_] := AssociationThread[domain -> Range[Length[domain]]] ordinal[domain_] := indexed[domain]/Length[domain] - 1/(2 Length[domain]) (* Map an association to another association: <| k->v |> -> <| \ k->f[k,v ]|> *) (* This is a bit confusing, but have a look at its usage in mark \ render functions. *) kvMap[f_, assoc___] := AssociationMap[First[#] -> f[First[#], Last[#]] &, assoc] (* Applies the appropriate scale function to each option in assocs *) scaleMap[scale_, assocs__] := kvMap[ Map[scale[optionChannels[#]]["apply"], #2] &, toTable[Association[assocs]]] (* Returns a list of k->v where k is the channel name and v is a list \ of values *) markChannelValues[mark_] := mark // List // Prepend[optionChannels] // KeyIntersection // Values // Transpose // MapApply[#1 -> toList[#2] &] // KeyDrop[Keys[mark["scales"]]] (* Dot *) render[dot, opts_, scale_, ctx_] := Module[{ m = scaleMap[scale, <|"x" -> 0, "y" -> 0|>, KeyTake[opts, {"x", "y", "r", "stroke"}]]}, MapThread[ Style[Circle[#1, #2], #3] &, {Transpose[ ctx["dims"]*{m["x"], m["y"]}], m["r"], m["stroke"]}]] dot[opts___] := Association[Options[dot], opts] Options[dot] = {"type" -> dot, "fillOpacity" -> 0.1, "strokeOpacity" -> 0.5, "r" -> 1, "stroke" -> Automatic, "scales" -> <||>}; (* Line *) render[line, opts_, scale_, ctx_] := Module[{ m = scaleMap[scale, KeyTake[opts, {"x", "y", "stroke"}]]}, (* For now we draw a single curve with a single color. *) Style[Line[Transpose[ctx["dims"]*{m["x"], m["y"]}]], First[m["stroke"]]]] line[opts___] := Association[Options[line], opts] Options[line] = {"type" -> line, "x" -> 0, "y" -> 0, "fillOpacity" -> 0.3, "stroke" -> Automatic, "scales" -> <||>}; (* Text *) render[text, opts_, scale_, ctx_] := Module[{ m = toTable[Association[ scaleMap[scale, KeyTake[opts, {"x", "y", "stroke"}]], KeyTake[opts, {"dx", "dy"}], <|"align" -> opts["align"]|>]]}, MapThread[ Style[Text[#1, #2, #3], #4] &, {opts["text"], Transpose[ctx["dims"]*{m["x"], m["y"]} + {m["dx"], m["dy"]}], m["align"], m["stroke"]}]] text[opts___] := Association[Options[text], opts] Options[text] = {"type" -> text, "x" -> 0, "y" -> 0, "dx" -> 0, "dy" -> 0, "align" -> Center, "stroke" -> Automatic, "scales" -> <||>}; (* Link *) render[links, opts_, scale_, ctx_] := Module[{ m = Association[ scaleMap[scale, KeyTake[opts, {"x1", "y1", "x2", "y2", "stroke"}]], KeyTake[opts, {"dx1", "dx2", "dy1", "dy2"}]], p1, p2, xy }, p1 = Transpose[ctx["dims"]*{m["x1"], m["y1"]} + {m["dx1"], m["dy1"]}]; p2 = Transpose[ctx["dims"]*{m["x2"], m["y2"]} + {m["dx2"], m["dy2"]}]; Thread[Style[Map[Line, Thread[List[p1, p2]]]], m["stroke"]]] Options[links] = {"type" -> links, "x1" -> 0, "y1" -> 0, "x2" -> 0, "y2" -> 0, "dx1" -> 0, "dx2" -> 0, "dy1" -> 0, "dy2" -> 0, "stroke" -> Automatic, "scales" -> <||>}; links[opts___] := Association[Options[links], opts] (* Axes are given as function marks, which accept the scales as an \ argument, and whose values do not inform the scale domains. *) (* This solves the order-of-operations problem where the domain is \ inferred from marks, and wants to be "visualized" by axis marks. *) (* We still need a way to signal to Plot to not add its own axes if \ the user provided some; maybe with some magic options. *) xAxis[] := If[KeyExistsQ[#, "x"] && Length[#["x"]["domain"]] > 0, {text["x" -> #["x"]["ticks"], "y" -> 0, "text" -> Map[NumberForm[#, {Infinity, 1}] &, #["x"]["ticks"]], "dy" -> -18, "stroke" -> Black, "align" -> Top, "scales" -> <|"y" -> <|"apply" -> Identity|>, "color" -> <|"apply" -> Identity|>|>], links["x1" -> #["x"]["ticks"], "x2" -> #["x"]["ticks"], "y1" -> 0, "y2" -> 0, "dy1" -> -12, "dy2" -> -16, "scales" -> <|"y" -> <|"apply" -> Identity|>, "color" -> <|"apply" -> (Black &)|>|>]}, {}] & yAxis[] := If[KeyExistsQ[#, "y"] && Length[#["y"]["domain"]] > 0, {text["x" -> 0, "y" -> #["y"]["ticks"], "text" -> Map[NumberForm[#, {Infinity, 1}] &, #["y"]["ticks"]], "dx" -> -20, "stroke" -> Black, "align" -> Right, "scales" -> <|"x" -> <|"apply" -> Identity|>, "color" -> <|"apply" -> Identity|>|>], links["x1" -> 0, "x2" -> 0, "y1" -> #["y"]["ticks"], "y2" -> #["y"]["ticks"], "dx1" -> -12, "dx2" -> -16, "scales" -> <|"x" -> <|"apply" -> Identity|>, "color" -> <|"apply" -> (Black &)|>|>]}, {}] & (* Default color schemes *) indexedColorScheme := Module[{colors = Map[Interpreter["StructuredColor"], {"#1f77b4", "#ff7f0e", "#2ca02c", "#d62728", "#9467bd", "#8c564b", "#e377c2", "#7f7f7f", "#bcbd22", "#17becf"}]}, colors[[#]] &] ordinalColorScheme := Blend[Map[ Interpreter["StructuredColor"], {"#f7f4f9", "#e7e1ef", "#d4b9da", "#c994c7", "#df65b0", "#e7298a", "#ce1256", "#980043", "#67001f"}], #] & (* Scale inference fills in the subcomponents of a channel scale \ based on the channel, whether the values are numeric, and \ already-specified subcomponents. *) (* Base case *) inferScale[channelName_, isNumeric_, {dom_, norm_, out_}] := {dom, norm, out} (* Default "out" for the radius channel *) inferScale["r", isNumeric_, {dom_, norm_, Automatic}] := {dom, norm, (2 + 5 #) &} (* Default domain for the "opacity" channel *) inferScale["opacity", isNumeric_, {Automatic, norm_, out_}] := {{0, 1} &, norm, out} (* Default domain and normalization for channels with all numeric \ values *) inferScale[channelName_, True, {Automatic, Automatic, out_}] := {MinMax, continuous, out} (* Default domain and normalization for channels with nonnumeric \ values *) inferScale[channelName_, False, {Automatic, Automatic, out_}] := {Union, ordinal, out} inferScale["color", False, {Automatic, Automatic, out_}] := {Union, indexed, out} (* Default continuous|ordinal color scheme *) inferScale["color", isNumeric_, {dom_, norm : continuous | ordinal, Automatic}] := {dom, norm, ordinalColorScheme} (* Default indexed color scheme *) inferScale["color", isNumeric_, {dom_, norm : indexed, Automatic}] := {dom, norm, indexedColorScheme} (* Default normalization is continuous *) inferScale[channelName_, isNumeric_, {dom_, Automatic, out_}] := {dom, continuous, out} (* Default out transform is Identity *) inferScale[channelName_, isNumeric_, {dom_, norm_, Automatic}] := {dom, norm, Identity} (* Tick inference figures out the ticks for a scale based on the \ properties of its subcomponents. *) inferTicks[in_, norm : ordinal | indexed, domain_] := domain inferTicks[in : Log10, norm : continuous, domain_] := Range[Floor[in[First[domain]]], Ceiling[in[Last[domain]]], 1/4.] inferTicks[in_, norm : continuous, {lo_, hi_}] := Range[lo, hi, (hi - lo)/4.] (* Remove Null elements from a list, since it's a common mistake to \ call plot[mark1, mark2, ] with a trailing comma. *) removeNulls[xs_] := xs /. Null -> Sequence[] (* The plot function accepts a list of marks, and returns a graphics \ object. *) Options[plot] = {"dims" -> {100, 100}, "margins" -> {{50, 10}, {50, 10}}, "metaMarks" -> ({} &)}; plot[marks___, OptionsPattern[]] := Module[{ dims = OptionValue["dims"], margins = OptionValue["margins"], markList = Join[{xAxis[], yAxis[]}, removeNulls[Flatten[{marks}]]], markTypes, channelValues, channelNumericQ, scales, markScales, filledMarks, renderedMarks }, channelValues = Merge[Map[markChannelValues, Select[markList, AssociationQ]], Catenate]; channelNumericQ = Map[AllTrue[NumericQ], channelValues]; scales = Association[ <|"x" -> <|"domain" -> {}, "ticks" -> {}, "apply" -> (0 &)|>, "y" -> <|"domain" -> {}, "ticks" -> {}, "apply" -> (0 &)|>|>, channelValues // kvMap[Module[{ channel = #, values = #2, in = Identity, dom, norm, out, domain}, {dom, norm, out} = ConstantArray[Automatic, 3] //. opts_ :> inferScale[channel, channelNumericQ[channel], opts]; domain = dom[Map[in, values]]; <|"domain" -> domain, "ticks" -> inferTicks[in, norm, domain], "apply" -> Composition[out, norm[domain], in]|> ] &] ]; markList = FixedPoint[Replace[Flatten[#], f_Function :> f[scales], {1}] &, markList]; markScales = markList // Map[Merge[{scales, #scales}, Merge[Last]] &]; markTypes = Comap[markList, "type"]; renderedMarks = Thread[render[markTypes, markList, markScales, <|"dims" -> dims|>]]; Graphics[ Translate[renderedMarks, Map[First, margins]], ImageSize -> dims + Map[Total, margins], PlotRange -> Transpose[{{0, 0}, dims + Map[Total, margins]}]]] raw = ExampleData[{"MachineLearning", "FisherIris"}, "Data"]; data = raw // Map[Append[First[#], Last[#]] &]; plot[ dot[ "x" -> data[[All, 1]], "y" -> data[[All, 2]], "stroke" -> data[[All, 5]]]] </code></pre> It depends what race you’re running 2024-09-20T20:04:25-04:00 Imagine there are three teams of runners who are about to compete against each other in a team-vs-team race. The first team has some very fast runners, but also some that are very slow. Despite the slowpokes, this team has highest average speed. 🯅 🯅 🯅 🯅 🯅 🯅 🯅 ----------------------------- Slow Fast </code></pre> The second team is made up entirely of unremarkable runners, all of whom run at roughly the same middling pace. 🯅🯅🯅🯅🯅🯅🯅 ----------------------------- Slow Fast </code></pre> The third team consists of runners who are far below average. But it has one incredible athlete – a world-class talent who’s easily the fastest runner on any of the teams. 🯅🯅🯅🯅🯅🯅 🯅 ----------------------------- Slow Fast </code></pre> Which of these teams is most likely to win the race? Well, it depends entirely on what kind of race it is. It might be a relay race, in which the total time taken by a team is the sum of the times taken by the individual runners. In this scenario, the first team would be most likely to win since it has the highest average speed.</li> Or maybe the race is one in which all of the runners start at the same time and teams are scored by how long it takes everyone on the team to finish the race. In this case, the time for a team depends exclusively on its slowest runner, which would give the second team the advantage since the other two teams both have slower runners.</li> Or maybe teams are scored not by their slowest runner, but by their fastest. Then the third team would be the favorite since it has the fastest individual runner.</li> </ol> This analogy is an attempt to explain some simple but deep ideas in software performance that I came to relatively recently, despite writing software for more than a decade: It’s useful to think of the execution time of a program not as a number, but as a distribution.</li> Which aspects of this distribution are worth optimizing depends not on the program, but on the context in which the program is invoked.</li> </ul> In the analogy, each team represents a computer program, and each runner represents a single execution of that program. The differences in the runner’s “run times” captures the idea of performance variation across multiple executions of the same program. The relay race is like a plain for</code> loop. Executions of the program occur serially, and the total time taken is the sum of the times of the individual executions. To win this race, it makes sense to optimize the mean execution time, which is just a scaled version of the sum.</li> The second race, in which all of the runners have to cross the finish line, is like invoking a program many times in parallel, then waiting for all of them to complete, as one might in a map-reduce</a> computation. For example, imagine you want to search a partitioned dataset. Individual partitions can be searched in parallel, and then the results combined, with the total run time depending on the maximum time it takes to conduct an individual search (since you need to wait for the slowest search to finish before returning the combined results).</li> The third race, in which the team with the fastest runner wins, is like issuing a bunch of parallel calls and but only waiting around for the first result. A practical example of this pattern is the idea of hedged requests</a> where, simplifying somewhat, to improve the response times from a service you send it multiple identical requests and take the response that arrives first, ie. the one with the minimum run time.</li> </ol> References</h2> The Tail at Scale</a> (summary</a>) Just as fault-tolerant computing aims to create a reliable whole out of less-reliable parts, large online services need to create a predictably responsive whole out of less-predictable parts; we refer to such systems as “latency tail-tolerant,” or simply “tail-tolerant.” Here, we outline some common causes for high-latency episodes in large online services and describe techniques that reduce their severity or mitigate their effect on whole-system performance. </blockquote> Minimum Times Tend to Mislead When Benchmarking</a> In this short post, I want to make a simple argument against using the minimum time of a benchmark as a proxy for that benchmark’s performance. </blockquote> Tail Latency Might Matter More Than You Think</a> I continue to believe that if you’re going to measure just one thing, make it the mean. However, you probably want to measure more than one thing. </blockquote> This draft is based on ideas I learned about in discussions with Yao Yue, Dan Luu, and Rebecca Isaacs over the last few years, all of whom know way more about this stuff than I do and have</a> written</a> articles</a> and</a> papers</a> that cover the same ground but in much more conceptual depth and technical detail. Also, all this assumes that you care about winning the race</a> in the first place. Indie Apps I Enjoy 2024-09-13T00:11:01-04:00 Sublime Text</a></h3> This is my text editor of choice for many years and I’m a happy paying customer. It’s fast with a very light feel, and does the basics well – editing, navigation, search – while staying responsive and almost bug-free. I’ve tried VSCode before but couldn’t get past inelegancies in the user interface or the laggy feeling of the editor. The folks behind Sublime Text also make a visual Git client, Sublime Merge</a>, which I also use. superwhisper</a></h3> This is a great little app for speech-to-text transcription. In fact, I’m writing this post with it. I like it for its relatively reliable and responsive text-to-speech functionality (press a keyboard shortcut, talk, press another one, and have it paste the transcribed text). I haven’t been interested in any of their more recent developments that add additional AI features. Little Snitch</a></h3> A nice firewall and network monitor app for Mac that comes with some nice visualizations of data rates over time, and geo-located servers that your computer is talking to. Things</a></h3> This is a to-do app that does the basics well and has an extremely pleasant user interface. I mostly use it to add tasks on the go using a shortcut on my home screen. These tasks go into an Inbox area that I can triage later. It’s mostly helpful during the times of my life when things get busy and I need some external memory to stay organized. Bike</a></h3> Bike is a text editor and outlining tool. I like its approach to text editing and use it for some of my freeform notes because it makes it easy to create a hierarchy and jump around a big outline. There are some features that I have consistently missed (I want to be able to paste in images & files), so I’m in the process of moving on to other apps for my writing, but I still like many of its ideas, including a novel caret affinity</a> system that solves a common issue when editing rich text. Weather strip</a></h3> This app has a really nice visual way of presenting the weather. I’m always excited about clear and clever data visualizations. Capo</a></h3> This is a fun app that helps you learn how to play a song by doing some helpful signal processing and showing you an annotated spectrogram with chord changes, and gives you the ability to slow down and loop particular sections of a song. Others</h3> This is getting a bit long so I’ll end with a few other indie apps (all created by independent developers or small teams) that I either use or spiritually approve of: Bear</a>, Overcast</a>, Halide</a>, Puzzmo</a>, Datasette</a>, and Kirby</a>. And on a related note, there are a few individuals who make fonts I like a lot: MB Type</a> and Fabrizio Schiavi</a>. Setting up an auto-publishing workflow 2024-09-12T23:08:24-04:00 A quick note on how I’m syncing Bear with this site, since I haven’t done this before. I’m using launchd</code></a> to update this site the with the most recent content from Bear</a> every minute. Right now, there’s hardly any content here so the inefficiency of re-generating and re-deploying everything is not a problem. The launchd</code> bit was a bit prickly to figure out, so I figured I’d write it down here. ChatGPT helped with a lot of the basics. Here are the instructions I ended up with (just pub</code> is a recipe that uses the excellent Just</a> command runner): Save this to ~/Library/LaunchAgents/is.yuri.pubcommand.plist</code>: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>is.yuri.pubcommand</string> <key>StartInterval</key> <integer>60</integer> <key>ProgramArguments</key> <array> <string>/bin/zsh</string> <string>-c</string> <string>just pub</string> </array> <key>WorkingDirectory</key> <string>/Users/yurivish/Dropbox/Projects/what</string> <key>StandardOutPath</key> <string>/tmp/what_yuri_is_output.log</string> <key>StandardErrorPath</key> <string>/tmp/what_yuri_is_error.log</string> <key>EnvironmentVariables</key> <dict> <key>PATH</key> <string>/Users/yurivish/.cargo/bin/:/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string> </dict> <key>RunAtLoad</key> <true/> </dict> </plist> </code></pre> Then run: launchctl unload ~/Library/LaunchAgents/is.yuri.pubcommand.plist launchctl load ~/Library/LaunchAgents/is.yuri.pubcommand.plist </code></pre> This will automatically run the publication command every minute, publishing the #pub</code> notes from Bear. To temporarily disable the scheduled job, you can unload the plist using launchctl: launchctl unload ~/Library/LaunchAgents/com.user.pubcommand.plist </code></pre> This will stop the job from running without deleting the plist file. To re-enable it later, simply load it again: launchctl load ~/Library/LaunchAgents/com.user.pubcommand.plist </code></pre> You can see logs by tail</code>ing the paths specified in the plist file under StandardOutPath</code> and StandardErrorPath</code>. Trying something new. 2024-09-12T20:52:36-04:00 This is my place for rough ideas and half-baked thoughts. I’ve set it up to automatically synchronize with a new note app that I’m trying out, Bear</a>. I hope this workflow will lower the barrier to writing online. I’ve wanted to be a writer for a long time but have never been very comfortable with the process, and I often feel like my taste handily exceeds my skills. This seems fairly common</a> and I’m finally going to do something about it. In the spirit of making things easy I’m going to try another experiment, which is to draft my notes with voice using superwhisper</a>. This will either increase the length of my posts or improve my speaking concision… The main goal is to get comfortable publishing messy notes and unrefined ideas. I’m curious to see how it goes. Euclidean Distance and Pearson Correlation 2021-12-28T00:00:00+00:00 I recently learned there’s a useful connection between Euclidean distance</a> and Pearson correlation</a>: the squared Euclidean distance between two z-normalized</a> vectors equals their Pearson distance</a>, up to a scale factor. This short note walks through a derivation. Z-normalization</h3> Let’s say we have two z-normalized vectors $x$ and $y$ in $N$ dimensions. Z-normalizing</a> a vector involves subtracting its mean then dividing by its standard deviation, so we know that the means $\mu_x$ and $\mu_y$ of our vectors $x$ and $y$ are zero: $$ \mu_x = \mu_y = 0 $$ and their standard deviations $\sigma_x$ and $\sigma_y$ are one: $$ \sigma_x = \sigma_y = 1 $$ We’re going to introduce some statistical concepts and see how they simplify under these constraints. Expressions in color are simplifications valid only under z-normalization. Summations implicity sum over all terms: $\small{\sum{x_i} = \sum_i^Nx_i}$ Covariance</h3> Since the means $\mu_x$ and $\mu_y$ are both zero, the numerator reduces to a dot product: $$ \operatorname{cov}(x, y) = {\sum{(x_i - \mu_{x})(y_i - \mu_{y})} \over N} $$ $$ \htmlClass{special}{\operatorname{cov}(x, y) = {{\sum{x_iy_i}} \over N}} $$ Standard deviation</h3> The standard deviations $\sigma_x$ and $\sigma_y$ both simplify to one by the definition of z-normalization: $$ \sigma_x = \sqrt{cov(x, x)} $$ $$ \htmlClass{special}{\sigma_x = \sqrt{\sum{x_i^2} \over N} = 1} $$ Pearson correlation</h3> The denominator reduces to one, leaving us with just the covariance: $$ \operatorname{P}(x, y) = {\operatorname{cov}(x, y) \over \sigma_x\sigma_y} $$ $$ \htmlClass{special}{\operatorname{P}(x, y) = \operatorname{cov}(x, y) = {{\sum{x_iy_i}} \over N}} $$ Euclidean distance</h3> Now let’s look at squared Euclidean distance</a> between z-normalized vectors. $$ \operatorname{dist^2}(x, y) = \sum{(x_i - y_i)^2} $$ We can expand out the squared differences inside the sum: $$ {\operatorname{dist^2}(x,y) =}\sum{(x_i^2 - 2x_iy_i + y_i^2)} $$ then separate out the terms into their own sums so we can tackle them one by one: $$ {\operatorname{dist^2}(x,y) =} \sum{x_i^2} - 2\sum{x_iy_i} + \sum{y_i^2} $$ Let’s examine this. The two sum-of-squares terms, $\sum{x_i^2}$ and $\sum{y_i^2}$, are both $N$ times the squared standard deviation, which reduces to $N$: $$ \underbrace{\sum{x_i^2}}_{N \text{ times } \sigma^2_x ~=~ N} \htmlClass{light}{ - 2\sum{x_iy_i} +} \underbrace{\sum{y_i^2}}_{N \text{ times } \sigma^2_y ~=~ N} $$ </div> Meanwhile, the middle term is $2N$ times $\operatorname{P}(x, y)$: $$ \htmlClass{light}{\sum{x_i^2} -} \underbrace{2\sum{x_iy_i}}_{2N \text { times } \operatorname{P}(x, y)} \htmlClass{light}{+ \sum{y_i^2}} $$ Putting these pieces together, our formula becomes: $$ \htmlClass{special}{\operatorname{dist^2}(x, y) = 2N - 2N \cdot \operatorname{P}(x, y)} $$ which can be simplified to: $$ \htmlClass{special}{\operatorname{dist^2}(x, y) = 2N(1 - \operatorname{P}(x, y))} $$ showing that the squared Euclidean distance between two z-normalized vectors is $2N$ times their Pearson distance</a>. Why does this matter?</h3> A common trick to improve Euclidean nearest-neighbor searches is to z-normalize the vectors that you’re searching over. The connection to Pearson correlation gives us an intuition for why this works so well in practice. Additionally, representing correlation as a distance metric allows us to use it with algorithms that rely on properties of a distance metric, such as the triangle inequality</a>. Despite the name, Pearson distance is not actually a distance</a>. References</h3> On Clustering Time Series Using Euclidean Distance and Pearson Correlation</a>: For time series comparisons, it has often been observed that z-score normalized Euclidean distances far outperform the unnormalized variant. In this paper we show that a z-score normalized, squared Euclidean Distance is, in fact, equal to a distance based on Pearson Correlation. </blockquote>

yuri.is

LLM Experiments #5: Design by API

LLM Experiments #4: Intermezzo

LLM Experiments #3: Noodling

LLM Experiments #2: Eval Planning

LLM Experiments #1: Shorter Sentences

Abraham Lincoln & English as She Is Spoke

H2 Morris Counter

Podcast Summarization

Widows, Orphans, and Runts

Data Visualizations are Data Structures

Great Singers on Great Singing

LLMs for Nominative Determinism

Tech Notes on Podcast Vibes

Podcast Vibes Presentation

XOR Doubly-Linked List

Podcast Vibes Prototyping

Note: Podcast Vibes

The Structure of the Wavelet Tree

Wavelet Matrix Construction

Enter the Wavelet Matrix

Gilbert Curve

LLMs Make Context More Valuable

Splash Color Format

Voice Note Browser

Hyparquet

Advanced Essay

Rust-analyzer can expand macros

Formatting Code with a Git Hook

Quicker Netlify Deploys

Pareto Frontier

Cheerio for Web Scraping

Visualizing Sentences with Finite State Automata

Creating a Visualization Library in Mathematica

It depends what race you’re running

Indie Apps I Enjoy

Setting up an auto-publishing workflow

Trying something new.

Euclidean Distance and Pearson Correlation

Standard deviation</h3> The standard deviations $\sigma_x$ and $\sigma_y$ both simplify to one by the definition of z-normalization:</p> $$ \sigma_x = \sqrt{cov(x, x)} $$</p> $$ \htmlClass{special}{\sigma_x = \sqrt{\sum{x_i^2} \over N} = 1} $$</p>

Pearson correlation</h3> The denominator reduces to one, leaving us with just the covariance:</p> $$ \operatorname{P}(x, y) = {\operatorname{cov}(x, y) \over \sigma_x\sigma_y} $$</p> $$ \htmlClass{special}{\operatorname{P}(x, y) = \operatorname{cov}(x, y) = {{\sum{x_iy_i}} \over N}} $$</p>

Standard deviation</h3>
The standard deviations $\sigma_x$ and $\sigma_y$ both simplify to one by the definition of z-normalization:</p>
$$ \sigma_x = \sqrt{cov(x, x)} $$</p>
$$ \htmlClass{special}{\sigma_x = \sqrt{\sum{x_i^2} \over N} = 1} $$</p>

Pearson correlation</h3>
The denominator reduces to one, leaving us with just the covariance:</p>
$$ \operatorname{P}(x, y) = {\operatorname{cov}(x, y) \over \sigma_x\sigma_y} $$</p>
$$ \htmlClass{special}{\operatorname{P}(x, y) = \operatorname{cov}(x, y) = {{\sum{x_iy_i}} \over N}} $$</p>