[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task mixing #95

Closed
pronvis opened this issue Feb 3, 2021 · 17 comments · Fixed by #97
Closed

Task mixing #95

pronvis opened this issue Feb 3, 2021 · 17 comments · Fixed by #97

Comments

@pronvis
Copy link
pronvis commented Feb 3, 2021

Copypasted of issue that I create in tokio: tokio-rs/tokio#3500

Version

    │   ├── tokio v1.1.0
    │   │   └── tokio-macros v1.0.0
    │   ├── tokio-util v0.6.2
    │   │   ├── tokio v1.1.0 (*)
    │   │   └── tokio-stream v0.1.2
    │   │       └── tokio v1.1.0 (*)
    │   ├── tokio v1.1.0 (*)
    ├── tokio v1.1.0 (*)
    ├── tokio-stream v0.1.2 (*)
    ├── tokio-util v0.6.2 (*)
    │   ├── tokio v1.1.0 (*)
    │   ├── tokio-stream v0.1.2 (*)
│   └── tokio v1.1.0 (*)
│       │   └── tokio v1.1.0 (*)
│       ├── tokio v1.1.0 (*)
│       ├── tokio-util v0.6.2 (*)
├── tokio v1.1.0 (*)
├── tokio-util v0.6.2 (*)

cargo tree | grep redis:

├── bb8-redis v0.8.0
│   └── redis v0.19.0

cargo tree | grep bb8:

├── bb8 v0.7.0
├── bb8-redis v0.8.0
│   ├── bb8 v0.7.0 (*)

Platform
Linux videotest01.dev.wb.ru 4.19.0-13-cloud-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux

Description
After moving to Tokio v1.0.1 our "reddis-proxy" service start to fail with "magic" errors:
Response was of incompatible type: "Response type not hashmap compatible" (response was status("PONG"))
This error happens when we call let response: redis::RedisResult<collections::HashMap<String, String>> = conn.hgetall(key).await;
Looks like async tasks mixed somehow and hgetall get response from PING request. Do not know where to dig into, will try with newer version of tokio...


We have async fn:

async fn ask_redis(
    redis_pool: Pool<RedisConnectionManager>,
) -> Result<_> {
    let mut conn = redis_pool.get().await?;
    let key = todo!();

    let response: redis::RedisResult<collections::HashMap<String, String>> =
        conn.hgetall(key).await;
    
    ...
}

That called ~10 times per second. And sometimes (in 10-20% cases) this line says: Response was of incompatible type: "Response type not hashmap compatible" (response was status("PONG")).

Redis-pool have logic inside, that sends PING requests to validate connection (I think it do that before each request in my case). So, looks like response on that requests goes in wrong future task... it is the only way (as I think) why hgetall can receive PONG string as a response. We do not have those values in redis.
And all works fine before we moved to tokio 1.x

@djc
Copy link
Owner
djc commented Feb 3, 2021

That's pretty surprising! It does indeed seem like there's an issue here with the <RedisConnectionManager as ManageConnection>::is_valid(), but I'm pretty confused that the call to PING and awaiting its result wouldn't take care of the PONG response. Can you produce a minimal reproduction of the issue?

@djc
Copy link
Owner
djc commented Feb 3, 2021

I noticed this in the Redis docs:

If the client is subscribed to a channel or a pattern, it will instead return a multi-bulk with a "pong" in the first position and an empty bulk in the second position, unless an argument is provided in which case it returns a copy of the argument.

Is your client subscribed to a channel or pattern?

@djc
Copy link
Owner
djc commented Feb 3, 2021

(Note that you don't need to bump the tokio dependency to 1.1 in the bb8 dependencies explicitly, since 1.1 is semver-compatible with 1.0. You can just use tokio 1.1 in your downstream dependency/application, and bb8 should pick it up.)

@HoldMyBeer1111
Copy link

Any chance it may be connected with redis-rs/redis-rs#308 ?

@HoldMyBeer1111
Copy link

Also we can have a look how it is implemented in mobc: https://github.com/importcjj/mobc/blob/87ae0152a51c0526caee9bd87455acf20cc778cb/mobc-redis/src/lib.rs#L30

@pronvis
Copy link
Author
pronvis commented Feb 3, 2021

Is your client subscribed to a channel or pattern?

no, we don't

@djc
Copy link
Owner
djc commented Feb 4, 2021

@invis87 maybe you can try with the explicit return type as suggested in redis-rs/redis-rs#308?

@pronvis
Copy link
Author
pronvis commented Feb 4, 2021

@djc I already have it: let response: redis::RedisResult<collections::HashMap<String, String>> = conn.hgetall(key).await;

@djc
Copy link
Owner
djc commented Feb 4, 2021

I meant in the ConnectionManager::is_valid() code.

@pronvis
Copy link
Author
pronvis commented Feb 4, 2021

I don't know how to reproduce that bug. Sometimes service "moved to buggy state" and then there are a lot of Response was of incompatible type: "Response type not hashmap compatible" (response was status("PONG")) until restart, but usually it works fine.
Looks like I need to wait for that state to appear and then trying to catch the problem with gdb.

Buuut, I will try to fix ConnectionManager::is_valid() anyway :)

This was referenced Feb 9, 2021
@djc djc closed this as completed in #97 Feb 9, 2021
@pronvis
Copy link
Author
pronvis commented Feb 11, 2021

Hmmm, interesting!
With those dependencies:

bb8 = "0.7.0"
bb8-redis = "0.8.1"

Service still have response was status("PONG") errors - much less then with version without fix, but still.

On the other hand, with deps:

bb8 = { git = "https://github.com/invis87/bb8.git", branch = "fix_ping_request" }
bb8-redis = { git = "https://github.com/invis87/bb8.git", branch = "fix_ping_request" }

Number if errors is zero.

@djc
Copy link
Owner
djc commented Feb 11, 2021

That is surprising. Did you correct for other differences in the Cargo.lock file?

@pronvis
Copy link
Author
pronvis commented Feb 12, 2021

Hmmm, after night response was status("PONG") appears on mine version too... I think it is less of them, but still.

Here is the diff:

 [[package]]
 name = "bb8"
-version = "0.7.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dae93eccab998c4b8703e3a6bbaa1714c38e445ebacb4bede25d0408521e293c"
+version = "0.8.0"
+source = "git+https://github.com/invis87/bb8.git?branch=fix_ping_request#5a38f008d11f2467b44d469d69e0a31a7fdf618d"
 dependencies = [
  "async-trait",
  "futures-channel",
@@ -100,9 +99,8 @@ dependencies = [

 [[package]]
 name = "bb8-redis"
-version = "0.8.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8a77bf264e0442cbf0644595e64f1f8e15f1e5fe99c9d0959072e54ffbd2b9d5"
+version = "0.9.0"
+source = "git+https://github.com/invis87/bb8.git?branch=fix_ping_request#5a38f008d11f2467b44d469d69e0a31a7fdf618d"
 dependencies = [
  "async-trait",
  "bb8",

Am I the only one who use hgetall with bb8-redis? Problem appears only here:

    let response: redis::RedisResult<collections::HashMap<String, String>> =
        conn.hgetall(key).await;

@pronvis
Copy link
Author
pronvis commented Feb 12, 2021

@djc look what Ive found: importcjj/mobc#28

@djc
Copy link
Owner
djc commented Feb 15, 2021

Yeah, I'm guessing this is actually due to redis-rs/redis-rs#325 and bb8 is actually not causing this.

@Barre
Copy link
Barre commented Oct 18, 2021

@djc A workaround would be using get_multiplexed_async_connection instead of get_async_connection, would you accept a PR for this?

As the semantics are not the same maybe this should be done through a feature flag or an other adapter?

@djc
Copy link
Owner
djc commented Oct 25, 2021

@Barre I think that would make sense. Please submit it without a feature flag first, and explain a bit more in the PR about any changes in the semantics that you expect/are documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants