cl-tidyr is a Common Lisp library for tidy data reshaping, part of the cl-tidyverse ecosystem. It provides a set of tools to change the shape (layout) of your data, handle missing values, and manage nested structures.
In Data Science, "Tidy Data" is a standard way of mapping the meaning of a dataset to its structure. In a tidy dataset:
- Every column is a variable.
- Every row is an observation.
- Every cell is a single value.
Messy data often stores variables as column headers or combines multiple observations into one row. cl-tidyr helps you transform this "messy" data into "tidy" data, making it much easier to analyze with libraries like cl-dplyr and cl-tibble.
If you are coming from R, cl-tidyr will feel very familiar. Here is a lookup table for the most common functions:
| R tidyr | cl-tidyr | Purpose |
|---|---|---|
pivot_longer() |
pivot-longer |
Wide to Long (Gather) |
pivot_wider() |
pivot-wider |
Long to Wide (Spread) |
separate() |
separate |
Split one column into many |
unite() |
unite |
Combine many columns into one |
drop_na() |
drop-na |
Remove rows with missing values |
replace_na() |
replace-na |
Fill missing values with specific data |
fill() |
fill |
Propagate values (last observation carried forward) |
nest() |
nest |
Create list-columns of tibbles |
unnest() |
unnest |
Flatten list-columns |
expand() |
expand |
Generate all combinations of variables |
complete() |
complete |
Fill in missing rows/combinations |
This tutorial is for Common Lisp developers who want to perform data cleaning and preparation.
Imagine you have survey results where dates are column headers. This is "Wide" data.
(defparameter *survey-wide*
(cl-tibble:tibble :student '("Alice" "Bob")
:2025-01-01 '(90 85)
:2025-01-02 '(92 88)))
;; To analyze this over time, we want it "Long":
(pivot-longer *survey-wide* '(:2025-01-01 :2025-01-02)
:names-to "date"
:values-to "score")
;; => Result:
;; #<TIBBLE 4x3>
;; STUDENT DATE SCORE
;; "Alice" "2025-01-01" 90
;; "Alice" "2025-01-02" 92
;; "Bob" "2025-01-01" 85
;; "Bob" "2025-01-02" 88cl-tidyr uses a consistent standard for missing data via cl-vctrs-lite:*na*.
(defparameter *messy-data*
(cl-tibble:tibble :x (vector 1 *na* 3)
:y (vector "a" "b" nil)))
;; Drop rows with any NA
(drop-na *messy-data*)
;; Replace NA with a default
(replace-na *messy-data* '(:x 0 :y "unknown"))
;; Fill missing values (Last Observation Carried Forward)
(fill *messy-data* :x :direction :down)Using the tidy macro, you can chain operations together elegantly:
(tidy *survey-wide*
(pivot-longer '(:2025-01-01 :2025-01-02) :names-to "date" :values-to "score")
(separate "date" '("year" "month" "day") :sep "-")
(drop-na :score))cl-tidyr is built on cl-vctrs-lite. To ensure robust performance:
- Detection: Use
(is-missing-p val)to check for mission values. It correctly identifies both Lispniland the ecosystem-specific*na*. - Production: Functions like
completeandpivot-widerwill produce*na*by default for missing cells.
cl-tidyr prioritizes semantic correctness and flexibility during the cleaning phase. This has implications for Common Lisp's specialized vectors (e.g., (vector double-float)):
- Type Promotion: To accommodate
*na*,cl-tidyrpromotes specialized numeric vectors to element-typeT. Specialized vectors cannot store*na*without a type error. - The Recommended Workflow:
- Clean: Use
cl-tidyrverbs to reshape, drop, or fill missing data. The resulting tibble columns will typically be of typeT. - Optimize: Once the data is "tidy" and all NAs are removed or replaced with numeric defaults, coerce the columns back to specialized types for high-performance numeric processing.
- Clean: Use
;; Example: Transitioning from Tidy to Optimized
(let* ((clean-df (drop-na messy-df :score))
(scores (cl-tibble:tbl-col clean-df :score))
(specialized-scores (coerce scores '(simple-array double-float (*)))))
;; High-performance processing here...
)(ql:quickload :cl-tidyr)Requires: cl-tibble, cl-dplyr, cl-vctrs-lite.
MIT