Skip to content

tinker_cookbook.hyperparam_utils.get_lora_lr_over_full_finetune_lr

tinker_cookbook.hyperparam_utils.get_lora_lr_over_full_finetune_lr(model_name, lora_alpha)

Return the factor that you should scale the full fine-tuning learning rate by to get the equivalent LoRA learning rate. Previously we had a more complicated formula, but the factor of 10 was more accurate empirically. See Lora Without Regret (https://thinkingmachines.ai/blog/lora/) for more details.

Parameters:

  • model_name (str) – HuggingFace model identifier (currently unused but kept for API consistency).
  • lora_alpha (int) – LoRA alpha scaling parameter (currently unused; multiplier is fixed at 10).

Returns: float