Shared Memory for R Objects
→ share() writes an R object into shared memory and returns a shared
version
→ Compact ALTREP serialization — shared objects move through
serialize() and mirai() as small references
→ Lazy access and automatic cleanup — data is read on demand and freed by R’s garbage collector
→ OS-level shared memory (POSIX / Win32) — pure C, no external dependencies
install.packages("mori")Parallel computing multiplies memory. When 8 workers each need the same 200 MB dataset, that is 1.6 GB of serialization, transfer, and deserialization. RAM holds 8 separate copies.
share() writes the data into shared memory once. Each worker then maps
the same physical pages. Per-worker copies become per-worker references.
library(mori)
library(mirai)
library(lobstr)
daemons(8)
# 200 MB data frame — 5 columns × 5M rows
df <- as.data.frame(matrix(rnorm(25e6), ncol = 5))
shared_df <- share(df)Without mori, each worker holds the full data frame. With mori, each worker holds a small reference into the shared memory region:
mirai_map(1:8, \(i, data) format(lobstr::obj_size(data)),
.args = list(data = df))[.flat] |> unique()
#> [1] "200.00 MB"
mirai_map(1:8, \(i, data) format(lobstr::obj_size(data)),
.args = list(data = shared_df))[.flat] |> unique()
#> [1] "824 B"The workers also skip 8 × 200 MB of serialization and deserialization, which gives a significant runtime saving:
boot_mean <- \(i, data) colMeans(data[sample(nrow(data), replace = TRUE), ])
# Without mori — each daemon deserializes a full copy
mirai_map(1:8, boot_mean, .args = list(data = df))[] |> system.time()
#> user system elapsed
#> 0.613 11.691 8.504
# With mori — each daemon maps the same shared memory
mirai_map(1:8, boot_mean, .args = list(data = shared_df))[] |> system.time()
#> user system elapsed
#> 0.001 0.002 4.389
daemons(0)Workers must run on the same machine, because mori shares physical RAM.
shared_name() returns the shared memory name of a shared object.
map_shared() opens a region by this name. This passes a reference
between processes without serialization:
x <- share(rnorm(1e6))
shared_name(x)
#> [1] "/mori_615b_f436100d"# Another process can map the region by name
y <- map_shared(shared_name(x))
identical(x[], y[])
#> [1] TRUEThe ALTREP serialization hooks emit the same identifier on the wire. The serialized form is a few bytes, regardless of the data size:
length(serialize(x, NULL))
#> [1] 131This is transparent to any R serialization pathway. mirai, parallel,
callr, and base R serialize() all carry shared objects as
references, not copies.
Sub-elements of a shared list serialize as references too. In this case, each element travels as a path into the parent shared region, not as the full data:
daemons(3)
# Share a list — all 3 vectors in a single shared region
lst <- share(list(a = rnorm(1e6), b = rnorm(1e6), c = rnorm(1e6)))
# Each element arrives on the worker as a zero-copy reference
mirai_map(lst, \(v) format(lobstr::obj_size(v)))[.flat] |> unique()
#> [1] "904 B"
daemons(0)share() writes all atomic vector types, lists, and data frames
directly into shared memory. Attributes are preserved end-to-end.
Pairlists become lists. The returned ALTREP wrappers point into the
shared memory region. There is no deserialization and no per-process
memory allocation.
share() returns all other R objects (environments, closures, language
objects) unchanged. It creates no shared memory region for them.
A data frame uses a single shared region. Workers read columns on demand. A worker that needs 3 of 100 columns loads only 3. Character strings load on demand, one element at a time.
df <- share(as.data.frame(matrix(rnorm(1e7), ncol = 100)))
shared_name(df) # one region for all 100 columns
#> [1] "/mori_615b_f436100f"
shared_name(df[[50]]) # sub-path into the same region
#> [1] "/mori_615b_f436100f[50]"R’s garbage collector manages the shared memory. A region stays alive
while R holds a reference to any object backed by it. The reference can
be the original from share(), or a column or sub-list extracted from
it. This reference can be in the original or another process. When no
references remain, or when the session exits cleanly, R frees the shared
memory automatically.
Important: Make sure that R does not garbage-collect the return
value of share() before a consumer maps its shared memory.
If a process dies before cleanup runs (a crash, SIGKILL, or the OOM
killer), its region can be left behind. prune_shared() reclaims these
orphans. It removes only regions whose creating process is no longer
running.
Consumers map the shared data read-only. This prevents corruption of the shared region. Changes are always local. Copy-on-write makes sure that other processes continue to read the original shared data:
- Structural changes to a list or data frame (add, remove, or reorder elements) produce a regular R list. The shared region stays unchanged.
- Modifying values of a shared vector (for example,
X[1] <- 0) materializes only that vector into a private copy. Other vectors in the same shared region stay zero-copy.
mizu builds a complete async and parallel framework over the same shared-memory core.
mori-shared objects travel through mizu tasks as references. Workers map the same physical pages — nothing is copied onto the pool, at any data size:
library(mizu)
p <- mizu_pool(4)
# The 200 MB data frame shared above reaches the pool as a reference:
t <- mizu_submit(p, format(lobstr::obj_size(shared_df)), shared_df = shared_df)
mizu_collect(t)
#> [1] "824 B"
# A parallel map over its columns reads the same shared pages on each worker:
m <- mizu_map(p, shared_df, \(col) mean(col), .template = numeric(1))
mizu_collect(m)
#> V1 V2 V3 V4 V5
#> 0.0007697908 -0.0007094785 -0.0007802216 0.0003363780 -0.0003906289
mizu_pool_stop(p)Use mizu when you want the framework: task pools with sub-microsecond
round trips, mizu_map(), channels, promises integration for Shiny,
reproducible parallel RNG, and Python workers.
mori remains the right choice when:
- You have a parallel stack already. A mori shared object works for any
serialize()pathway — mirai, parallel, callr, future. - Data must outlive a pool. A mori region is named and lives until R’s garbage collector frees the last reference. Any process on the machine can map it by name.
–
The mori project is released with a Contributor Code of Conduct. When you contribute to this project, you agree to obey its terms.