8000 Question about KV Cache quantization · Issue #23 · efeslab/Atom · GitHub
[go: up one dir, main page]

Skip to content

Question about KV Cache quantization #23

Open

Description

@SherrySwift

Hi, thanks for your great work!
I have a small question about KV Cache quantization. Did you use pagedattention to accelerate KV Cache 4-bit quantization? If so, where is the corresponding cuda kernel code? Thank you.

Activity

  1. happierpig commented on Sep 6, 2024

    @happierpig
    Collaborator

    Hi @SherrySwift ,

    Yes, we use page attention with low-bit quantization to speedup memory loading time. INT4 tokens are loaded and decoded into FP16, which still uses FP16 arithmetic calculations. Check quantization code here: https://github.com/efeslab/Atom/blob/main/kernels/include/flashinfer/quantization.cuh#L10

  2. SherrySwift commented on Sep 7, 2024

    @SherrySwift
    Author

    Hi @happierpig ,
    Thanks for your quick reply, I have another question.
    According to

    constexpr size_t PACK_NUM = 8;
    , 8×4bit data are packed into one int32 data.
    But in , it seems that 4bit KV Cache are stored in int8 type rather than int32 type.
    Feel a little bit confused about this.

  3. happierpig commented on Sep 7, 2024

    @happierpig
    Collaborator

    Sure! We use int8 data type to pack 2 int4 elements since the minimum addressable data size on GPU is byte.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0