起始问题
提问原文
What is the tokenizor in the below code?
from mlx_lm import generate, load
from mlx_lm.models.cache import load_prompt_cache, make_prompt_cache, save_prompt_cache
model, tokenizer = load("mlx-community/Qwen2.5-32B-Instruct-4bit")
What is the tokenizor in the below code?
from mlx_lm import generate, load
from mlx_lm.models.cache import load_prompt_cache, make_prompt_cache, save_prompt_cache
model, tokenizer = load("mlx-community/Qwen2.5-32B-Instruct-4bit")