tinker_cookbook.tokenizer_utils
| Function | Description |
|---|---|
register_tokenizer() |
Register a custom tokenizer factory. |
get_registered_tokenizer_names() |
Return a list of all registered custom tokenizer names. |
is_tokenizer_registered() |
Check if a tokenizer name is registered. |
unregister_tokenizer() |
Unregister a custom tokenizer factory. |
get_tokenizer() |
Get a tokenizer by name. |