Repository navigation
build error from tokenizers dependency #340
Description
I'm seeing the same rust build error reported here huggingface/tokenizers#1485, which was fixed in huggingface/tokenizers#1355 (tokenizers-0.14.1). Running pip install --dry-run transformers==4.34.0 shows that the version is the one including the fix. Is it possible to update transformers to a more recent version?
Otherwise, there is this workaround suggested: RUSTUP_TOOLCHAIN=1.72.0 pip install tokenizers==0.13.2, but it another error
error: package `clap v4.5.26` cannot be built because it requires rustc 1.74 or newer, while the currently active rustc version is 1.72.0```.
Activity
joprice commentedon Jan 18, 2025
I was able to build by setting
RUSTFLAGS="-A invalid_reference_casting"to ignore the warning, but seems like since it can lead to UB, it should be addressed eventually.