8000 GitHub - klehmann/mindoodb-data-import · GitHub
[go: up one dir, main page]

Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

mindoodb-data-import

A small, generic command-line example that imports JSON documents into local MindooDB databases and pushes them to a MindooDB server.

Its distinguishing feature: each imported record carries a _createdAt timestamp, which is replayed as the document's real creation time. This lets you seed a database as if the documents had been created over the past months or years, so the MindooDB Time Travel feature can reconstruct historical states.

Install

npm install

This installs mindoodb plus the CLI dependencies. The commands run directly from TypeScript via tsx — no build step required.

Data layout

Everything lives under data/<tenantId>/:

data/
  importtesttenant1/
    user.identity.json        # the (non-admin) user identity used to author docs
    admin.identity.json       # the tenant admin identity (public keys + push auth)
    user.keybagkeys.json      # the user's default + $publicinfos doc keys
    contacts/
      importdata.json         # documents to import into the "contacts" database
    <otherDb>/
      importdata.json
  • Each subdirectory that contains an importdata.json becomes one MindooDB database, named after the directory but lowercased: a database id is itself a directory name in the store layout, and Windows/macOS file systems cannot keep Sales and sales apart. Two source folders differing only in case abort the run instead of merging into one database.
  • If data/ holds exactly one tenant directory it is auto-selected; otherwise pass the tenant id as the command argument.

importdata.json

A JSON array of records. _id and _createdAt are mandatory; every other field becomes the document body.

[
  {
    "_id": "112233",
    "_createdAt": "2026-01-01T00:00:00.000Z",
    "firstname": "John",
    "lastname": "Doe"
  }
]
  • _createdAt may be an ISO-8601 string or epoch milliseconds.
  • Fields whose names start with _ (other than _id/_createdAt) are ignored: MindooDB reserves the underscore namespace for its own bookkeeping.

Server-first flow

Every command consults the server before touching local stores. It prompts for the server URL (enter demo as a shortcut for MindooDB's public demo server, https://demo-server1.mindoodb.com, the same one Haven offers), shows the server's signing-key fingerprint (SHA-256, first 8 bytes as aa:bb:cc:…) for you to confirm — exactly like the Haven UI — and then checks whether the tenant already exists (/.well-known/mindoodb-tenants/…):

  • Tenant exists (the expected case) — the server holds the authoritative directory database with the user's access grant. The tool pulls that directory into the local store (authenticating as the user), so the local access check passes without fabricating anything.
  • Tenant does not exist — a brand-new tenant. The tool registers the user once in the local directory (admin-signed, reusing the imported tenant keys), and push later publishes that directory to the server.

This ordering avoids creating a second grantaccess document for a tenant that the server already knows about.

Commands

dry-run [tenantId]

Runs the server-first handshake above, then imports into an in-memory MindooDB and prints per-database counts and any validation issues. Nothing is written to disk. Because it pulls the real directory grant, this command now requires network access to the server.

npm run dry-run
# or: npx tsx src/cli.ts dry-run importtesttenant1

import [tenantId]

Runs the server-first handshake, then imports into on-disk stores under out/<tenantId>/stores/<dbId>/…, mirroring how a MindooDB server stores the same tenant. If out/ is not empty you are asked to confirm clearing it first.

npm run import
# or: npx tsx src/cli.ts import importtesttenant1

push [tenantId]

Pushes the locally-imported databases to a MindooDB server:

  1. Runs the server-first handshake (URL, fingerprint confirmation, existence check).
  2. If the tenant does not exist yet, publishes/registers it using the admin identity and pushes the directory database so the server receives the user's access grant.
  3. Syncs each database (documents and attachments) with live console progress, authenticating as the tenant admin.
npm run push
# or: npx tsx src/cli.ts push importtesttenant1

Run import before push; the push reads the on-disk stores from out/.

Passwords

Passwords are prompted interactively with hidden input. Up to three distinct 6F04 secrets may be involved:

Secret Unlocks Needed by
User-identity password user.identity.json private keys dry-run, import, push
Keybag password the key entries in user.keybagkeys.json (export password) dry-run, import, push
Admin password admin.identity.json private signing key dry-run, import, push

The admin password is prompted for every command. It is used to sign the local directory grant when a brand-new tenant is registered from scratch, and to authenticate the tenant publish and pushes. When the tenant already exists on the server, the directory grant is pulled rather than fabricated, but the admin password is still collected up front so the flow is uniform.

The user-identity password and the keybag password are treated as independent (you are prompted for each); they may or may not be the same string depending on how the keybag file was exported.

Document ids

MindooDB custom ids must match ^[a-z][a-z0-9_]*$ — lowercase only, because ids end up in on-disk filenames and those live on case-insensitive file systems. Ids that do not comply (for example the purely numeric "112233" or the mixed-case "Contact42") are normalized:

  • the id is lowercased,
  • non-conforming characters are replaced with _, and
  • the id is prefixed with --id-prefix (default doc), producing e.g. doc_112233 and doc_contact42.

When an id is normalized, the original value is preserved in the document body under the sourceId field. Ids that are already valid are kept verbatim.

Because lowercasing can map two source ids onto one document id (Contact42 and contact42), the import skips the second record and reports it as an issue rather than merging the two silently — a custom-id create is idempotent, so the collision would otherwise pass unnoticed. The same guard catches outright duplicate _id values in the source file.

npx tsx src/cli.ts import --id-prefix rec

How historical timestamps work

MindooDB stamps every store entry's createdAt from a single internal "semantic clock". This tool pins that clock to each record's _createdAt while the document is created, then restores the real clock — so the persisted entry (and therefore the document's creation time) is backdated. Records are imported one at a time because each has its own timestamp.

Access-control checks on a server continue to use the server's own receivedAt time, so backdated createdAt values only affect the historical view, not trust decisions.

Output

  • out/<tenantId>/stores/<dbId>/docs/… — document store
  • out/<tenantId>/stores/<dbId>/attachments/… — attachment store

The out/ directory is git-ignored.

License

Licensed under the Apache License, Version 2.0. Copyright 2026 Mindoo GmbH.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0