Skip to content

tinker_cookbook.tokenizer_utils.register_tokenizer

tinker_cookbook.tokenizer_utils.register_tokenizer(name, factory)

Register a custom tokenizer factory.

Once registered, get_tokenizer will call factory() instead of loading from HuggingFace when the given name is requested.

Parameters:

  • name (str) – The tokenizer name (typically a HuggingFace model ID like "Foo/foo_tokenizer").
  • factory (Callable[[], Tokenizer]) – A callable that takes no arguments and returns a Tokenizer instance.

Returns: None

def my_tokenizer_factory():
return MyCustomTokenizer()
register_tokenizer("Foo/foo_tokenizer", my_tokenizer_factory)