Choose a verifier, a task, and a base model. Download a ready-to-run training loop.

Reinforcement: Build Your Own Reward Model

Input

You get

Runnable preference loopLoop: Preferred ↑ / Rejected ↓ ➔ Preference loss ➔ Step

Loss forward_backward_custom / DPO

Task: Mapping verification logic directly onto out-of-sample forecasts.

Base model: deepseek-ai/DeepSeek-V3.1

File forecasting-deepseek-v3-1-dpo.py

Build

Reward: MAX

You just built this

`forecasting-deepseek-v3-1-dpo.py`

Runnable loop for Calibrated Forecasting on DeepSeek-V3.1.

✓ Preference-pair reward
✓ DeepSeek-V3.1
✓ LoRA rank 32
✓ 50 training steps
✓ Script compiled
✓ Four Tinker primitives in the file
✓ API key never stored

LoRA r=32, 50 steps, Pairwise preference loss.

LoRA rank

Steps

View code

forecasting-deepseek-v3-1-dpo.pysample · forward_backward · optim_step · save_state

"""Reinforcement.tech compiled Tinker loop

H1: Reinforcement: Build Your Own Reward Model

INPUT
  Signal : Preference pairs (Pairwise preference)
  In     : Preferred / rejected pairs from real traces.
  Task   : Calibrated Forecasting — Scored predictions over time
  Model  : deepseek-ai/DeepSeek-V3.1 (MOE, large MoE)

OUTPUT
  Loop         : Runnable preference loop.
  Loss         : pairwise preference (no reference policy)
  LoRA rank    : 32
  Steps        : 50

Tinker primitives used in this file:
  sample                   preview the adapter
  forward_backward_custom  pairwise preference loss
  optim_step               Adam update on the adapter
  save_state               checkpoint weights + optimizer

Requires:
  uv pip install tinker torch
  export TINKER_API_KEY=...
  Docs: https://tinker-docs.thinkingmachines.ai/tinker/quickstart/
"""

from __future__ import annotations

import asyncio
import os
from pathlib import Path

import tinker
from tinker import types

BASE_MODEL = "deepseek-ai/DeepSeek-V3.1"
LORA_RANK = 32
LEARNING_RATE = 1e-5
STEPS = 50
SAVE_EVERY = 10
RUN_NAME = Path(__file__).stem


def require_key() -> None:
    if not os.environ.get("TINKER_API_KEY"):
        raise SystemExit("Set TINKER_API_KEY before running this loop.")


async def connect():
    service = tinker.ServiceClient()
    training = await service.create_lora_training_client_async(
        base_model=BASE_MODEL,
        rank=LORA_RANK,
        user_metadata={"product": "reinforcement.tech", "run": RUN_NAME},
    )
    tokenizer = training.get_tokenizer()
    return service, training, tokenizer


async def sampling_client(training):
    """Ephemeral on-policy sampler. Do not pass name= — it is deprecated and ignored."""
    return await training.save_weights_and_get_sampling_client_async()


async def checkpoint(training, step: int) -> None:
    if step % SAVE_EVERY != 0 and step != STEPS - 1:
        return
    saved = await training.save_state_async(name=f"{RUN_NAME}-step-{step}")
    await saved.result_async()

BETA = 0.1
# Replace PREFERRED / REJECTED with your pairs. This file trains on one example.
PREFERRED = "0.31 — base rate plus one independent signal; do not round to 50%."
REJECTED = "Hard to say. Maybe 50-50. The world is uncertain."
PROMPT = "Complete the task. Prefer the trace a senior builder would keep."


def as_sft_datum(tokenizer, prompt: str, completion: str) -> types.Datum:
    prompt_tokens = tokenizer.encode(prompt)
    completion_tokens = tokenizer.encode(completion)
    full = prompt_tokens + completion_tokens
    n_prefix = max(len(prompt_tokens) - 1, 0)
    return types.Datum(
        model_input=types.ModelInput.from_ints(tokens=full[:-1]),
        loss_fn_inputs=dict(
            target_tokens=full[1:],
            weights=[0.0] * n_prefix + [1.0] * len(completion_tokens),
        ),
    )


def dpo_loss(data, logprobs_list):
    """Pairwise logistic preference. No frozen reference policy (closer to SLiC than textbook DPO)."""
    import torch
    import torch.nn.functional as F

    preferred_lp = logprobs_list[0].sum()
    rejected_lp = logprobs_list[1].sum()
    loss = -F.logsigmoid(BETA * (preferred_lp - rejected_lp))
    return loss, {"dpo_loss": float(loss.detach())}


async def train() -> None:
    require_key()
    _service, training, tokenizer = await connect()

    preferred = as_sft_datum(tokenizer, PROMPT, PREFERRED)
    rejected = as_sft_datum(tokenizer, PROMPT, REJECTED)

    for step in range(STEPS):
        fwdbwd = await training.forward_backward_custom_async(
            [preferred, rejected],
            dpo_loss,
        )
        optim = await training.optim_step_async(types.AdamParams(learning_rate=LEARNING_RATE))
        result = await fwdbwd.result_async()
        await optim.result_async()
        await checkpoint(training, step)
        print(f"step {step:03d}  {result.metrics}")

    sampling = await sampling_client(training)
    prompt = types.ModelInput.from_ints(tokenizer.encode(PROMPT))
    preview = await sampling.sample_async(
        prompt=prompt,
        num_samples=1,
        sampling_params=types.SamplingParams(max_tokens=128, temperature=0.4),
    )
    print("sample:", tokenizer.decode(preview.sequences[0].tokens))


if __name__ == "__main__":
    asyncio.run(train())

Loading the builder…

What people train

Waldo

Tool agent

Reward cited tool calls and thrift.

Source: Waldo on Tinker Open recipe

AxiomProver

Reasoning model

Reward proofs that verify.

Source: AxiomProver on Tinker Open recipe

Judgmental forecast

Forecasting model

Reward calibrated probabilities.

Source: Mantic on Tinker Open recipe