pcuenq
/

Hunyuan-7B-Instruct-tokenizer

Text Generation

Model card Files Files and versions

pcuenq HF Staff commited on Jul 7

Commit

915ca05

·

verified ·

1 Parent(s): 9d9f5b8

Update README.md

Files changed (1) hide show

README.md +2 -0

README.md CHANGED Viewed

@@ -9,6 +9,8 @@ This is a transformers fast tokenizer for [mlx-community/Hunyuan-7B-Instruct-3bi
 ## Conversion
 ```py
 from huggingface_hub import snapshot_download
 from tokenization_hy import *

 ## Conversion
+We used this code to convert the tokenizer from the original `tiktoken` format:
 ```py
 from huggingface_hub import snapshot_download
 from tokenization_hy import *