💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Python 3.11 support (Python only modification)
Python 3.11 support (Python only modification)
Decoder is now a composable trait, but without being backward incompatibleProcessor is now a composable trait, but without being backward incompatibleBoth trait changes warrant a "major" number since, despite best efforts to not break backward compatibility, the code is different enough that we cannot be exactly sure.
unstable_wasm feature to support building on Wasm (it's unstable !)Decoder is now a composable trait, but without being backward incompatibleProcessor is now a composable trait, but without being backward incompatibleBoth trait changes warrant a "major" number since, despite best efforts to not break backward compatibility, the code is different enough that we cannot be exactly sure.