8000 With lua-parallel-scripts on, KEYS/SCAN still take the command gate alone and stall every connection (after #757) · Issue #772 · kividbio/kividb · GitHub
[go: up one dir, main page]

Skip to content

With lua-parallel-scripts on, KEYS/SCAN still take the command gate alone and stall every connection (after #757) #772

Closed

Description

@murtazayusuf

Follow-up to #754 / #757. With lua-parallel-scripts yes (Lua builds), keyspace walks (KEYS, SCAN, DBSIZE, RANDOMKEY, the flushes) take the command gate alone (tag_locks::sees_the_whole_keyspace → takes_the_gate_alone), so a walk can never see a parallel script half done. As a result, #757's fix (other connections keep being served during a long KEYS/SCAN) does not apply there: a KEYS over 2M keys still stalls every connection for its whole duration (~280 ms in CI).

Options to evaluate:

  • walks take each hash-tag's lock (or every tag stripe) shard by shard instead of the gate, so a script holding tag X only blocks the walk while it's in X's shard;
  • walks read through a consistent per-shard snapshot, and only wait for scripts touching that shard;
  • accept it and document that with parallel scripts on, KEYS/SCAN are exclusive (Redis's KEYS is exclusive too, but it's one thread for everything anyway).

Measure the probe-PING worst case and script throughput (BullMQ bench) for each.

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceThroughput and latency

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0