Owner: Will Salt
Date: 2025‑09‑20 Status: Draft for review
This document outlines the design of a Mini‑Ledger, a double‑entry financial ledger targeting 100k+ TPS and 99.999% availability, built with Go services, CockroachDB for strongly‑consistent, multi‑region storage, and Redpanda for durable event streaming and fan‑out. The design uses CQRS + Event Sourcing, a transactional outbox via CockroachDB changefeeds to Redpanda, and idempotent, exactly‑once write semantics from clients to the canonical ledger log. We operate with GitOps (Flux) pulling OCI‑packaged Kubernetes resources per cluster, a SLSA Level 4‑compliant CI/CD pipeline that signs and attests all artifacts, an Istio ambient service mesh (mTLS mesh‑wide via ztunnel; optional L7 via waypoint proxies), and observability via the Grafana LGTM stack deployed per region with a global Grafana for federated views. Whilst the design is multi‑region, and some data is replicated cross‑region, for regulatory reasons some data will stay within designated regions.
- Financial correctness via strict double‑entry and immutable ledger log.
- 100k+ TPS sustained write throughput with p99 end‑to‑end < 150 ms (in‑region).
- Five‑nines availability across multi‑AZ / multi‑region.
- Strong consistency for transaction commit order; tunable consistency for read views.
- Full auditability and deterministic replay (event‑sourced).
- FX pricing/valuation, risk engines, AML screening (integrations only).
- Full core‑bank replacement—focus is the ledger and adjacent primitives.
- Language: Go
- Primary DB: CockroachDB
- Streaming: Redpanda
- Infra: Kubernetes (K8s), mTLS service mesh (Istio ambient mode), Gateway API for ingress
- API Gateway (Istio Gateway API per region/cluster: ingress, authn/z, rate limiting, idempotency keys)
- Transaction Service (validation, command orchestration, conflict checks)
- Ledger Writer (append to canonical ledger log; enforces double‑entry invariants)
- Outbox / Changefeed (CRDB changefeed → Redpanda topics)
- Projectors (materialize read models: account balances, statements, aggregates)
- Query Service (CQRS read side; low‑latency caches)
- Reconciliation & Repair (periodic and on‑demand checks)
- Ops Plane (admin UI, schema & feature flags, DR tooling)
- Includes GitOps pipeline: build artifacts → publish OCI bundles of K8s resources → Flux pull‑based reconcile per cluster.
- Client calls POST /transactions with Idempotency‑Key and legs.
- Transaction Service validates schema; checks auth; resolves accounts & currency; reserves idempotency key.
- Ledger Writer performs single SQL transaction: append immutable
ledger_entries(legs) +transactionsheader +outbox_eventsrow. - CockroachDB commits (consensus quorum).
- Changefeed emits
outbox_eventsto Redpanda → consumers (Projectors, external systems). - Projectors update read models (
balances,account_summaries) idempotently. - Client receives 201 Created with committed
transaction_idand status.
- Write side: linearizable per shard (via CRDB) ensuring a total order per account‑range; global order is the DB commit order.
- Read side: eventually consistent projections; staleness bounds via changefeed lag metrics.
- Integer minor units (e.g., cents) using
INT8(up to ±9.22e18). currency_code CHAR(3)(ISO‑4217) or extended table for digital assets (scale, symbol).- All arithmetic performed with 64‑bit integers; prohibit float/decimal in app layer.
-- Accounts (immutable identity; mutable metadata stored versioned)
CREATE TABLE accounts (
account_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
ledger_id UUID NOT NULL, -- multi-ledger tenancy
currency_code CHAR(3) NOT NULL,
status STRING NOT NULL DEFAULT 'ACTIVE',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
closed_at TIMESTAMPTZ,
metadata JSONB NOT NULL DEFAULT '{}'
) PARTITION BY LIST (ledger_id); -- align with CRDB locality constraints
-- Transactions (header)
CREATE TABLE transactions (
transaction_id UUID PRIMARY KEY,
ledger_id UUID NOT NULL,
client_id STRING NOT NULL, -- tenant/app
idempotency_key STRING NOT NULL,
total_amount_abs INT8 NOT NULL, -- sum of debit legs == sum of credit legs
currency_code CHAR(3) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
committed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
status STRING NOT NULL DEFAULT 'COMMITTED', -- or REVERSED
attributes JSONB NOT NULL DEFAULT '{}',
UNIQUE (client_id, idempotency_key)
);
-- Ledger entries (immutable legs)
CREATE TABLE ledger_entries (
entry_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
transaction_id UUID NOT NULL REFERENCES transactions(transaction_id),
account_id UUID NOT NULL REFERENCES accounts(account_id),
direction STRING NOT NULL CHECK (direction IN ('DEBIT','CREDIT')),
amount_minor INT8 NOT NULL CHECK (amount_minor > 0),
currency_code CHAR(3) NOT NULL,
effective_at TIMESTAMPTZ NOT NULL, -- posting date
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
INDEX (account_id, effective_at),
INDEX (transaction_id)
);
-- Outbox for changefeed → Redpanda
CREATE TABLE outbox_events (
event_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
aggregate_type STRING NOT NULL, -- 'transaction','account'
aggregate_id UUID NOT NULL,
event_type STRING NOT NULL, -- 'TransactionCommitted','TransactionReversed'
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);-- Monotonic balance snapshots (per account)
CREATE TABLE balances (
account_id UUID PRIMARY KEY,
balance_minor INT8 NOT NULL,
updated_at TIMESTAMPTZ NOT NULL,
version INT8 NOT NULL -- projector sequence number
);
-- Statement lines for fast queries
CREATE TABLE account_statements (
account_id UUID,
entry_id UUID,
transaction_id UUID,
direction STRING,
amount_minor INT8,
effective_at TIMESTAMPTZ,
PRIMARY KEY (account_id, effective_at, entry_id)
);Notes:
- Projections can be rebuilt by replaying
ledger_entries(or outbox stream). - For multi‑region, we’ll use CRDB regional by row to place accounts and entries near their primary region.
- Double‑entry: sum(credits) == sum(debits) per transaction and per currency.
- Atomicity: all legs commit in a single DB transaction.
- Idempotency:
(client_id, idempotency_key)unique—retries return the original result. - Immutability: ledger entries are append‑only; reversals are new compensating transactions.
- Ordering: for any given account, entries are observed in commit order via index
(account_id, effective_at, entry_id)plus commit time.
POST /v1/transactions Request:
{
"idempotency_key": "e5c1-...",
"ledger_id": "...",
"currency": "USD",
"description": "ACH payout",
"legs": [
{"account_id": "A1", "direction": "DEBIT", "amount_minor": 1250},
{"account_id": "A2", "direction": "CREDIT", "amount_minor": 1250}
],
"effective_at": "2025-09-20T10:12:00Z",
"attributes": {"merchant_id": "m_123"}
}Response 201:
{"transaction_id": "T123", "status": "COMMITTED", "committed_at": "..."}GET /v1/accounts/{id}/balance → {"balance_minor": 12345, "asof": "..."}
POST /v1/transactions/{id}:reverse → creates compensating legs.
message MoneyLeg {
string account_id = 1;
enum Direction { DEBIT = 0; CREDIT = 1; }
Direction direction = 2;
int64 amount_minor = 3;
}
message CreateTransactionRequest {
string idempotency_key = 1;
string ledger_id = 2;
string currency = 3;
repeated MoneyLeg legs = 4;
google.protobuf.Timestamp effective_at = 5;
map<string,string> attributes = 6;
}
message CreateTransactionResponse {
string transaction_id = 1;
string status = 2; // COMMITTED
google.protobuf.Timestamp committed_at = 3;
}ledger.transactions.v1— emitted fromoutbox_eventschangefeed; key =transaction_id.ledger.accounts.v1— account lifecycle events.ledger.projections.balance.apply.v1— internal bus for projector work‑queue (optional if consumers read primary stream).
- Protobuf envelopes (schema‑versioned); Confluent Schema Registry compatible.
- Keys use stable UUIDs for partition affinity.
- Producer config: enable idempotence;
acks=all;max.in.flight.requests.per.connection=1for ordering.
- Use CRDB
CREATE CHANGEFEED FOR TABLE outbox_events INTO 'kafka://redpanda-broker:9092'with ordering and resolved timestamps. - App writes outbox rows in the same SQL transaction as the ledger commit; changefeed guarantees at‑least‑once to Redpanda; consumers are idempotent (use event_id).
- Batching writes per SQL transaction: commit N legs (typically 2) + 1 outbox row.
- Connection pooling (pgx) with prepared statements; avoid ORM overhead.
- Hot‑path Go optimizations: object pooling for small structs, zero‑alloc JSON/proto encoders, bounded goroutine pools.
- Sharding by ledger_id/account hash to distribute ranges across CRDB nodes.
- Async projections (don’t block client on read‑side updates).
- In‑region p50 commit: 8–15 ms (DB), p99 < 50 ms; end‑to‑end p99 < 150 ms with TLS & Redpanda enqueue decoupled.
- Backpressure: API Gateway returns 429 with
Retry‑Afterwhen queue depth exceeds thresholds.
- k6 to create a synthetic generator with real schemas; idempotent retries; 70/30 read/write mix.
- Observe CRDB metrics: raft propose latency, LSM compaction, changefeed lag; Redpanda metrics: ISR, end‑to‑end lag.
- Multi‑region deployment with survival goal = REGION (or ZONE for lower latency); 5‑node clusters per region; 3 regions for 5‑nines target.
- Regional by row locality on
accountsandledger_entriesto keep writes close to owners and observe data locality regulations. - Follower reads for queries that can tolerate staleness.
- Deploy a dedicated cluster per region for low latency and high throughput; bridge topics cross‑region asynchronously when needed.
- Topic replication factor RF=3; min ISR 2; rack awareness (availability zone).
- Client → API: Client provides Idempotency‑Key which is used to deduplicate requests.
- DB commit is the source of truth.
- Outbox ensures no lost messages; consumers dedupe by
event_id.
- CRDB incremental backups to object storage hourly; full weekly; encrypted with KMS.
- Periodic restore‑replay drills; RPO ≤ 5 minutes; RTO ≤ 30 minutes (target).
- mTLS everywhere (SPIFFE/SPIRE identities).
- AuthN: OIDC/JWT at gateway; AuthZ: fine‑grained RBAC + ABAC on ledger_id/account scope.
- Secrets & keys: HashiCorp Vault + HSM for signing keys.
- PII controls: field‑level encryption; tokenization for external exports.
- Audit: append‑only admin audit log in CRDB; signed Redpanda audit stream.
- Compliance: SOC2 controls; CIS benchmarks; data retention & GDPR delete via tombstone projections (canonical log immutable).
- Tracing: OpenTelemetry (OTLP) spans across API → DB → Redpanda; trace‑ID stuffed into outbox payload for cross‑plane correlation; Istio ambient telemetry for L7 metrics and mesh traces.
- Metrics: RED (Rate, Errors, Duration) for each service; ledger invariants counters (total debits == credits).
- Logging: structured JSON; sampling on high‑vol paths; PII redaction.
- LGTM Stack: Grafana LGTM (Loki, Grafana, Tempo, Mimir) deployed per region, backed by object store (S3/GCS); global Grafana with multiple data sources for cross‑region visibility. Dual‑write to a single store for a global view is optional.
- Alerting: Services generate their own Prometheus alert rules and package them alongside their code; centralized alert routing via AlertManager with regional and global notification channels.
- Unit tests: invariants (sum debits == sum credits), idempotency, currency mismatch rejections.
- Property‑based tests: randomized leg sets maintaining invariants.
- Integration tests: CRDB transactional semantics; changefeed → local Redpanda; projector idempotency.
- Chaos & fault injection: kill CRDB nodes, partition network; ensure no orphaned transactions; verify recovery by replay.
- Deterministic replay tests: rebuild projections from scratch and diff against live.
- GitHub Actions for all CI/CD workflows with SLSA Level 4 compliance for build provenance and security.
- Multi-stage builds: Docker containers for services, OCI artifacts for Kubernetes manifests and Helm charts.
- Artifact publishing: Push to regional container registries (ECR/GCR) and OCI registries (Harbor/GitHub Container Registry).
- Security scanning: Supply chain security with SBOM generation, vulnerability scanning, and signature verification.
- Build attestations: SLSA provenance attestations for all artifacts with cryptographic signatures following the v1.2 RC1 specification.
- Container images: Go services (API Gateway, Transaction Service, Ledger Writer, Projectors) with distroless base images.
- OCI artifacts: Kubernetes manifests, Helm charts, Flux configurations, and Istio configurations.
- Security artifacts: SBOMs (SPDX format), vulnerability reports, and SLSA build attestations following v1.2 RC1 standards.
- Per‑service SemVer: Each service follows Semantic Versioning (MAJOR.MINOR.PATCH). Backward‑incompatible API or contract changes increment MAJOR; backward‑compatible features increment MINOR; bug fixes increment PATCH.
- Product SemVer: The overall Mini‑Ledger product has its own SemVer that represents a tested, interoperable set of service versions (a bill‑of‑materials).
- Conventional Commits → SemVer: Version bumps are computed from Conventional Commits:
feat→ MINOR,fix/perf/refactor(non‑breaking) → PATCH, and commits withBREAKING CHANGE:or!→ MAJOR. Changelogs and release notes are auto‑generated per service and for the product BOM. - Tagging & provenance: Git tags drive both per‑service and product releases; Docker and OCI artifacts are tagged with both
service@semverandproduct@semverlabels/annotations, plus build digest and SLSA provenance references. - API compatibility policy: Public APIs (REST/gRPC) use additive changes within a MAJOR; removals/behavioral breaking changes require MAJOR bump. Deprecations carry at least one MINOR cycle with warnings.
- Schema/version gates: Database migrations are backward‑compatible within a product MAJOR using “expand → migrate → contract” workflow. Migrations are versioned and tied to service/product releases to ensure safe rollouts/rollbacks.
- Kubernetes with HPA/VPA on API & Writer; PDBs and Pod anti‑affinity.
- Progressive delivery: Flagger integrated with Istio for automated canary/blue‑green rollouts (metric checks, traffic shifting, rollback); CRDB rolling upgrades per docs (no more than one node per range).
- Schema migrations:
golang-migratewith backward‑compatible steps; feature flags gate writes to new columns. - Service mesh: Istio ambient mode deployed per cluster; mTLS enforced mesh‑wide with ztunnel and waypoint proxies. Optional multi‑cluster mesh federation for failover.
- Ingress: Gateway API (Istio implementation) per region/cluster with regional API endpoints for simplicity; optional multi‑cluster mesh federation for failover.
- GitOps: Gitless GitOps with Kubernetes manifests and Helm charts stored as OCI artifacts; Flux controllers run per cluster to reconcile from OCI (using
OCIRepository+Kustomization). - Observability: LGTM stack (Loki, Grafana, Tempo, Mimir) per region with object store backing; global Grafana instance with federated datasources for cross‑region dashboards and alerting.
- Flagger default SLO checks for rollouts: p99 latency, error rate, and HTTP success rate sourced from Prometheus/Mimir; per‑service thresholds aligned to API SLOs; automatic rollback on breach.
- Changefeed lag spikes: write continues; projections fall behind; alert on lag; readers can request strong reads directly from canonical tables for critical paths.
- Redpanda partition outage: outbox persists; re‑delivery on recovery; consumer offsets stored in Redpanda; idempotent apply by
event_id. - Region failure: CRDB survives with quorum; traffic steered to nearest region; balances served from follower reads with bounded staleness until leader re‑election.
- Hot account contention (high‑fan‑in): introduce escrow sub‑accounts or per‑account sequencer; apply batching transfers via internal settlement windows.
- DB: FoundationDB (excellent) vs CockroachDB; chose CRDB for SQL ergonomics + built‑in changefeeds + Postgres protocol.
- Messaging: Pulsar vs Redpanda; Redpanda performance & simplicity preferred.
- Architectural: direct Redpanda first (log‑centric) then DB sink; rejected due to correctness complexity and cross‑system XA.
- Define exact SLOs per endpoint (latency, availability).
- Settle on regional placement policy for cross‑currency ledgers.
- Pick schema registry and governance process.
- Produce capacity plan (node counts, sizing) and run load tests.