FFFF Shards with no live consumers · Issue #14 · patrobinson/gokini · GitHub
[go: up one dir, main page]

Skip to content

Shards with no live consumers #14

Open

Description

@relistan

We're running Gokini with the Benthos integration you wrote (kinesis_balanced). Most of the time everything works fine. After awhile, however, we keep ending up with shards that are not being consumed. It appears to be something in the GetLease() code. We're still trying to track down the issue, but I wanted to get something on the project radar.

At first we thought it was a problem of more than one consumer process stepping on each other's toes, but it happens even with only a single consumer.

The symptom is shards in DynamoDB marked as Closed: false but where the LeaseTimeout is quite awhile in the past. See attached screenshot.

Screen Shot 2020-08-19 at 9 54 43 AM

Activity

  1. patrobinson commented on Aug 20, 2020

    @patrobinson
    Owner

    Can you please export the dynamodb table and send it to me?

    If you restart benthos does it fix the problem or do the shards remain locked but not processing?

  2. relistan commented on Aug 20, 2020

    @relistan
    Author

    @patrobinson Sorry, I can't export the table since this is for real and we had to get it back up and running. Yes, restarting it works. This has happened a few times, so it's not a one-off. The only remaining columns were the sequence IDs if I recall properly, and I also screenshotted those at the time:

    Screen Shot 2020-08-19 at 9 54 49 AM

  3. patrobinson commented on Aug 30, 2021

    @patrobinson
    Owner

    Hi @relistan

    Have you observed any errors in the logs such as Error renewing lease?

    I've got some time now I can try and replicate this myself

  4. patrobinson commented on Aug 30, 2021

    @patrobinson
    Owner

    reading through the code I think this could happen if GetRecords returns an unrecoverable error.

    https://github.com/patrobinson/gokini/blob/master/consumer.go#L364-L376

    At which point I think we should panic rather than leave ourselves in a bad state.

    I've released a beta version with this fix https://github.com/patrobinson/gokini/releases/tag/v0.2.0-beta and I'll give it a whirl later this week

  5. relistan commented on Aug 30, 2021

    @relistan
    Author

    @patrobinson we stopped running it shortly after I opened this. We switched to the other Benthos Kinesis consumer and hard-pinned worker to shards. That's less than ideal but worked. So, unfortunately, I have no better info for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0