tinker_cookbook.eval.TrainingClientEvaluator

class tinker_cookbook.eval.TrainingClientEvaluator()

An evaluator that uses a TrainingClient to compute metrics (e.g., loss).

class NLLEval(TrainingClientEvaluator):
async def __call__(self, training_client):
loss = await compute_nll(training_client, self.eval_data)
return {"eval/nll": loss}

__call__(training_client)

Run evaluation and return a dict of metric names to values.

Parameters:

• training_client (tinker.TrainingClient) – The Tinker training client to evaluate with.

Returns: A dict mapping metric names to float values.