Repository navigation
[Bug]: NEAR ChainTracker stalls on skipped block heights despite fast RPC responses #2342
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
During investigation of NEAR provider performance, we captured a ChainTracker stall caused by a skipped block height.
Height 220979153 does not exist. Block 220979154 references 220979152 through both prev_height and prev_hash. We independently confirmed this on our primary node, backup node, and https://rpc.mainnet.near.org.
The provider's tracker repeatedly requested the missing height and failed with:
could not get block data in Chain Tracker GET_BLOCK_BY_NUM failed sending chainMessage UNKNOWN_BLOCK DB Not Found Error: BLOCK HEIGHT: 220979153
Observed timeline on October 21, 2026, UTC:
- 18:02:13.375815 — Tracker published height 220979150.
- 18:02:16.497772 — A block(finality=final) response containing height 220979154 had fully arrived at the provider.
- 18:02:20.612521 — Tracker finally published height 220979159.`
The tracker failed to publish a newer height for approximately 4.115 seconds after a newer finalized block had already been received. It logged seven hash-fetch failures for the missing height.
A passive packet capture showed 253 completed RPC requests during the overall 7.237-second publication gap. The slowest took approximately 21 ms, and the capture reported zero dropped packets. This particular stall therefore cannot be explained by slow HTTP responses.
In ChainTracker.readHashes, the tracker requests consecutive numeric heights using latestBlock - idx. An error on a skipped height aborts the update before publication, followed by retry backoff.
Expected behavior: handle NEAR's skipped heights without repeatedly blocking tracker updates, while preserving correct finalization and proof semantics.
Lava Version
lavap v6.2.9
How to reproduce?
- Run a NEAR provider connected to a synchronized mainnet node with debug logging enabled.
- Observe a skipped height within the range of blocks the tracker needs to fetch.
- Verify the missing height and its successor:
{"jsonrpc":"2.0","id":1,"method":"block","params":{"block_id":220979153}}
Expected response: UNKNOWN_BLOCK.
{"jsonrpc":"2.0","id":1,"method":"block","params":{"block_id":220979154}}
Relevant response fields:
{
"height": 220979154,
"prev_height": 220979152
}- Observe repeated hash-fetch failures and retry backoff while newer finalized blocks are available.
- Compare the arrival of block(finality=final) responses with ChainTracker callback triggered - new blocks detected logs.
For a deterministic regression test, use a mock fetcher that advances across a skipped height and returns UNKNOWN_BLOCK for that height. Verify that tracker publication does not repeatedly stall and that finalization/proof data remains valid.
The historical heights above document the captured incident; reproducing the live stall requires the tracker to encounter a skipped height during advancement.
Activity