SamplingClient
class tinker.SamplingClient(holder, sampling_session_id, shadow=False, retry_config=None, subprocess_sampling=None)
Client for text generation and inference from trained or base models.
The SamplingClient lets you generate text tokens from either a base model or from weights
you've saved using a TrainingClient. You typically get one by calling
service_client.create_sampling_client() or training_client.save_weights_and_get_sampling_client().
Key methods:
- sample() - generate text completions with customizable parameters
- compute_logprobs() - get log probabilities for prompt tokens
Create method parameters:
model_path: Path to saved model weights (starts with 'tinker://')base_model: Name of base model to use for inference (e.g., 'Qwen/Qwen3-8B')retry_config: Configuration for retrying failed requests
sampling_client = service_client.create_sampling_client(base_model="Qwen/Qwen3-8B")
prompt = types.ModelInput.from_ints(tokenizer.encode("The weather today is"))
params = types.SamplingParams(max_tokens=20, temperature=0.7)
future = sampling_client.sample(prompt=prompt, sampling_params=params, num_samples=1)
result = future.result()
Parameters:
- holder (InternalClientHolder)
- sampling_session_id (str)
- shadow (bool, default:
False) - retry_config (RetryConfig | None, default:
None) - subprocess_sampling (bool | None, default:
None)
sample(prompt, num_samples, sampling_params, include_prompt_logprobs=False, topk_prompt_logprobs=0)
Generate text completions from the model.
Parameters:
- prompt (types.ModelInput) – The input tokens as ModelInput
- num_samples (int) – Number of independent samples to generate
- sampling_params (types.SamplingParams) – Parameters controlling generation (temperature, max_tokens, etc.)
- include_prompt_logprobs (bool, default:
False) – Whether to include log probabilities for prompt tokens - topk_prompt_logprobs (int, default:
0) – Number of top token log probabilities to return per position
Returns: A Future containing the SampleResponse with generated text
prompt = types.ModelInput.from_ints(tokenizer.encode("The weather today is"))
params = types.SamplingParams(max_tokens=20, temperature=0.7)
future = sampling_client.sample(prompt=prompt, sampling_params=params, num_samples=1)
result = future.result()
for sample in result.samples:
print(tokenizer.decode(sample.tokens))
Async variant: sample_async()
compute_logprobs(prompt)
Compute log probabilities for prompt tokens.
Parameters:
- prompt (types.ModelInput) – The input tokens as ModelInput
Returns: A Future containing a list of log probabilities for each token in the prompt. None values indicate tokens where log probabilities couldn't be computed.
prompt = types.ModelInput.from_ints(tokenizer.encode("Hello world"))
future = sampling_client.compute_logprobs(prompt)
logprobs = future.result()
for i, logprob in enumerate(logprobs):
if logprob is not None:
print(f"Token {i}: logprob = {logprob:.4f}")
Async variant: compute_logprobs_async()
get_tokenizer()
Get the tokenizer for the current model.
Returns: PreTrainedTokenizer compatible with the model
get_base_model()
Get the base model name for the current sampling session.
Returns: str
Async variant: get_base_model_async()