Repository navigation
[Bug]: lavap payment claims require tx indexing on --node; otherwise paid sessions are retried and fail with "all relays rejected" #2340
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
Summary
lavap rpcprovider confirms a relay-payment claim only by looking the tx up by hash (Client.Tx) on the node given in --node. If that node runs CometBFT with [tx_index] indexer = "null", the lookup can never succeed ("transaction indexing is disabled"). After 5 minutes every claim is treated as failed, even though it was included with code 0 and paid.
The provider then re-submits the same, already paid sessions. The chain rejects them as double spends, simulation fails with all relays rejected, and after 3 attempts the sessions are dropped.
Nothing in the provider docs says that --node must index transactions.
Environment
- lavap v6.2.5, chain
lava-mainnet-1(lavad v5.5.1) --nodeis our own full node with[tx_index] indexer = "null"
What we see (every epoch)
INF Sent Transaction Hash=<h>
... 5 minutes later ...
ERR failed sending tx, wasn't found after timeout hash=<h>
ERR Failed getting a new tx factory error=
ERR relay_payment - sending Tx Failed error="Failed getting a new tx factory ErrMsg: : "
ERR failed sending rewards claim error="..."
WRN Timeout waiting for transaction hash="<raw bytes>"
... next reward cycles ...
INF Simulation failed reason:="... relay payment failed ErrMsg: all relays rejected {provider:...,description:...}"
INF Rewards for session are being removed due to surpassing the maximum allowed retries for payment requests. maxRetriesAllowed=3 sessionIds=...
We checked this against /block and /block_results from our node over about 42 hours (174 claim txs):
- every claim that lavap reported as "wasn't found" is on chain with code 0;
- all 2,114 sessions that lavap later dropped had already been paid;
- no session was paid twice.
So no money was lost here. The problem is that the retry path cannot work while the lookup is impossible.
Root cause
- lavap only confirms by hash lookup.
waitForTxCommitpolls onlyclientCtx.Client.Tx(ctx, hash, false)(L271) and gives up after 5 minutes (L293-L295). The CometBFT error ("transaction indexing is disabled") is only logged at debug level (L278). - The logged error is misleading. On that timeout an empty
TxResultDatais returned. Its emptyRawLoggoes intoparseTxErrorsAndTryGettingANewFactory, which returnsfmt.Errorf("%s", ""). That becomesFailed getting a new tx factory error=with an empty error (L111-L121). - Already-paid sessions are queued for retry.
updatePaymentRequestAttempt(rewards, false)puts the sessions intofailedRewardsPaymentRequestswithout checking whether their payments were already seen.PaymentHandlerprocessed those payment events from block results about 5 minutes earlier. - One rejected relay fails the whole message (chain side, v5.5.1 and
main). A re-submitted session is rejected asdouble spending detected(L154-L161). The whole message then fails if any relay was rejected:(L312-L318). The comment, and the intent of CNS-776: Don't fail relay payment on one bad relay #1054 ("Don't fail relay payment on one bad relay"), say all, but the code says any. As a result the partial-acceptance branch (// if all relays failed, fail the TX if rejectedRelaysNum != 0 {
rejected_cuevent,RejectedRelaysin the response, L320-L325, L338) can never run.
Impact
- ERR lines every epoch that look like lost payments.
- Hidden risk of loss:
- While the lookup is impossible, a claim that really fails (node unreachable during broadcast, mempool eviction) is retried in the same message as already-paid sessions. The whole message is rejected, and those unpaid sessions are dropped after 3 attempts.
- Because of (4), this also happens with tx indexing enabled whenever one relay in a retry batch is permanently invalid.
- Every timeout blocks a claim goroutine for 5 minutes.
Suggestions
- Document that the
--nodeused by lavap must have tx indexing (indexer = "kv"), or check/status→tx_indexat startup and warn. - In
waitForTxCommit, treat "transaction indexing is disabled" as unknown rather than failed. Confirm the claim another way, for example from the payment eventsPaymentHandleralready receives, or from the account sequence. - When recording a failed attempt, skip sessions whose payment
PaymentHandlerhas already processed (for example, keep a short-lived set of paid(epoch, sessionId)). Deleting from the map insidePaymentHandlerwould not help, because it runs before the sessions are inserted. - Log the real error instead of an empty
Failed getting a new tx factory error=. Print the hash as hex inTimeout waiting for transaction. - Chain: either make the
rejectedRelaysNumcheck match its comment, or fix the comment and document the any-rejects-all behavior.
Workaround: enable [tx_index] indexer = "kv" on the node lavap uses, or point --node at a node that has tx indexing.
Lava Version
lavap v6.2.5, lavad v5.5.1 (lava-mainnet-1). main still has the same code in tx_sender.go and msg_server_relay_payment.go.
How to reproduce?
- Run
lavap rpcproviderwith--nodepointing at a Lava node whoseconfig.tomlhas[tx_index] indexer = "null". - Serve some relays and wait for a reward claim:
INF Sent Transaction Hash=<h>. - Five minutes later lavap logs
failed sending tx, wasn't found after timeout hash=<h>. The claim is on chain with code 0; check it on any node that has tx indexing. - In the next reward cycles the same sessions are retried, fail simulation with
all relays rejected, and are dropped after 3 attempts.
Activity