Tokenicer v0.0.12

What's Changed

• Fix padding side proxy by @Qubitium in #55

Full Changelog: v0.0.11...v0.0.12

TokeNicer v0.0.11

What's Changed

• pad token decoding should use literal decode and not decode which is … by @Qubitium in #54

Full Changelog: v0.0.10...v0.0.11

TokeNicer v0.0.10

What's Changed

• Add explicit model_config support to Tokenicer.load by @Qubitium in #53

Full Changelog: v0.0.9...v0.0.10

TokeNicer v0.0.9

What's Changed

• qwen 2 omni (thinker) compat by @Qubitium in #50
• Qwen2 omini thinker compat by @Qubitium in #51
• Update pyproject.toml by @Qubitium in #52

Full Changelog: v0.0.8...v0.0.9

Tokenicer v0.0.8

What's Changed

• add pyproject.toml & fix warnings by @CSY-ModelCloud in #47
• fix attr may not exist by @CSY-ModelCloud in #48
• bump version to 0.0.8 by @CSY-ModelCloud in #49

Full Changelog: v0.0.7...v0.0.8

Tokenicer v0.0.7

What's Changed

• [CI] use new docker image by @CSY-ModelCloud in #44
• fix AttributeError: 'Qwen3_5MoeConfig' object has no 'bos_token_id' by @ZX-ModelCloud in #45
• Update setup.py by @Qubitium in #46

Full Changelog: v0.0.6...v0.0.7

Tokenicer v0.0.6

What's Changed

• [FIX] avoid proxying call to inner tokenizer (ChatGLMTokenizer compatibility) by @ZX-ModelCloud in #42

New Contributors

• @ZX-ModelCloud made their first contribution in #42

Full Changelog: v0.0.5...v0.0.6

Toke(n)icer v0.0.5

What's Changed

• Fix pad_token for longcat by @LRL-ModelCloud in #36
• Update test_pad_token.py by @LRL-ModelCloud in #38
• [CI] user docker image by @CSY-ModelCloud in #40
• [CI] add gemma3 test by @CL-ModelCloud in #35

New Contributors

• @LRL-ModelCloud made their first contribution in #36

Full Changelog: v0.0.4...v0.0.5

Toke(n)icer v0.0.4

What's Changed

⚡ Now tokenicer instance dynamically inherits the native tokenizer.__class__ of tokenizer passed in or loaded via our Tokenicer.load() api.
⚡ CI now tests tokenizers from 64 models

• fix mpt pad token bug by @CL-ModelCloud in #24
• fix model_config bugs by @CL-ModelCloud in #25
• test code clean up by @CL-ModelCloud in #26
• Inherits PretrainedTokenizer by @Qubitium in #28
• loop & test all models by @CSY-ModelCloud in #30

Full Changelog: v0.0.2...v0.0.4

Toke(n)icer v0.0.3

What's Changed

Now tokenicer instance dynamically inherits the native tokenizer.__class__ of tokenizer passed in or loaded via our Tokenicer.load() api.

• fix mpt pad token bug by @CL-ModelCloud in #24
• fix model_config bugs by @CL-ModelCloud in #25
• test code clean up by @CL-ModelCloud in #26
• Inherits PretrainedTokenizer by @Qubitium in #28

Full Changelog: v0.0.2...v0.0.3