tinker_cookbook.eval.SamplingClientEvaluator

class tinker_cookbook.eval.SamplingClientEvaluator()

An evaluator that uses a SamplingClient to compute metrics (e.g., accuracy).

class AccuracyEval(SamplingClientEvaluator):
async def __call__(self, sampling_client):
acc = await measure_accuracy(sampling_client, self.test_set)
return {"eval/accuracy": acc}

__call__(sampling_client)

Run evaluation and return a dict of metric names to values.

Parameters:

• sampling_client (tinker.SamplingClient) – The Tinker sampling client to evaluate with.

Returns: A dict mapping metric names to float values.