Skip to content

tinker_cookbook.hyperparam_utils.get_lora_lr_multiplier

tinker_cookbook.hyperparam_utils.get_lora_lr_multiplier(model_name)

Get a model-specific multiplier for the LR, when training with LoRA.

Given two models A and B, and learning rate LR_A that's known to be optimal for A, we can guess an optimal learning rate for B as LR_B = LR_A * get_lora_lr_multiplier(B) / get_lora_lr_multiplier(A)

Parameters:

  • model_name (str) – HuggingFace model identifier.

Returns: The LoRA LR multiplier combining full-finetune scaling and LoRA factor.