Reinforcement tuning

SeekrFlow supports reinforcement tuning using group relative policy optimization (GRPO). Reinforcement tuning trains a model to generate higher-quality outputs by scoring candidates against reference answers using one or more graders. To train a model with reinforcement tuning, follow the same process as standard fine-tuning with a few modifications. First, ensure your dataset has a reference_answer field containing the correct answer for each problem. Include a system prompt instructing the model to use the reasoning format:

{
    "messages":[
        {
            "role": "system",
            "content": "A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>"
        },
        {
            "role": "user",
            "content": "Find the smallest positive $a$ such that $a$ is a multiple of $4$ and $a$ is a multiple of $14.$"
        }
    ],
    "reference_answer": "28"
}

Upload this dataset with the purpose reinforcement-fine-tune. See Upload file for the full schema reference. Set fine_tune_type and define a reward function using reward_components in your TrainingConfig:

from seekrai import SeekrFlow
from seekrai.types import TrainingConfig, InfrastructureConfig
from seekrai.types.finetune import (
    FineTuneType, Grader, GraderType, RewardComponents,
    StringOperation, TextSimilarityOperation
)

client = SeekrFlow()

training_config = TrainingConfig(
    training_files=['<your-reinforcement-fine-tuning-file-id>'],
    model="meta-llama/Llama-3.2-1B",
    n_epochs=1,
    n_checkpoints=1,
    batch_size=4,
    learning_rate=1e-6, # lower learning rate is typical for reinforcement tuning
    experiment_name="helperbot_grpo_v1",
    fine_tune_type=FineTuneType.REINFORCEMENT,
    reward_components=RewardComponents(
        graders=[Grader(type=GraderType.MATH_ACCURACY)]
    )
)

Create the fine-tuning job using the standard workflow. See Create a fine-tuning job for the full process.

LoRA can be used with reinforcement tuning to reduce memory requirements. See LoRA for configuration details.

Reward functions

A reward function defines how model outputs are scored during training. In SeekrFlow, reward functions are built from one or more graders — individual scoring operations that each evaluate a specific quality of the output.

Grader types

Type	Enum value	Description	Operations
Numerical accuracy	`GraderType.MATH_ACCURACY`	Returns `1` if the model output is numerically equal to the reference answer, else `0`.	None
String check	`GraderType.STRING_CHECK`	Returns `1` if the model output matches the reference based on the selected operation.	`equals`, `not_equals`, `contains`, `case_insensitive_contains`
Text similarity	`GraderType.TEXT_SIMILARITY`	Returns a similarity score between the model output and the reference answer.	`bleu`, `rouge`
LLM (LLM-as-a-judge)	`GraderType.LLM`	Uses an LLM to score how closely the model output matches the reference answer in meaning and quality.	None

Create graders

# Numerical accuracy — no operation needed
math_grader = Grader(type=GraderType.MATH_ACCURACY)

# String check — requires an operation
keyword_grader = Grader(
    type=GraderType.STRING_CHECK,
    operation=StringOperation.CONTAINS
)

# Text similarity — requires an operation
similarity_grader = Grader(
    type=GraderType.TEXT_SIMILARITY,
    operation=TextSimilarityOperation.BLEU
)

# LLM-as-a-judge — no operation; optional generation parameters
judge_grader = Grader(type=GraderType.LLM)

LLM grader

The LLM grader (LLM-as-a-judge) scores each candidate response with an LLM instead of a deterministic metric. The judge compares the model output to the reference_answer and rates how well it matches in meaning and quality. Use it for open-ended or subjective outputs, such as semantic correctness, helpfulness, or tone, where string check and text similarity are too rigid. The LLM grader takes no operation. It accepts optional generation parameters that control how the judge produces its scores:

Parameter	Type	Default	Description
`temperature`	float, 0.0–1.5	`0.0`	Sampling temperature for the judge. Keep it near `0` for consistent, reproducible scores.
`top_p`	float, 0.0–1.0	None	Nucleus sampling cutoff for the judge.
`seed`	integer ≥ 0	None	Fixes the judge’s sampling so scores are reproducible across runs.
`max_completion_tokens`	integer ≥ 1	`512`	Maximum number of tokens the judge can generate while scoring each response.

The judge runs on infrastructure managed by SeekrFlow, so you don’t configure an inference endpoint or model for it.

# LLM grader with custom generation parameters
judge_grader = Grader(
    type=GraderType.LLM,
    temperature=0.0,
    max_completion_tokens=256,
    seed=42
)

The LLM grader takes a weight and combines with other graders the same way, as shown next.

Combine graders with weights

Assign weight percentages to combine multiple graders into a single reward function. Weights must sum to 1.0. If no weights are provided, graders are weighted equally.

reward = RewardComponents(
    graders=[
        Grader(type=GraderType.MATH_ACCURACY, weight=0.4),
        Grader(type=GraderType.STRING_CHECK, weight=0.3, operation=StringOperation.EQUALS),
        Grader(type=GraderType.LLM, weight=0.3, temperature=0.0)
    ]
)

Format reward weight

By default, 10% of the reward score is based on whether the model uses the correct output format (<think> and <answer> tags). You can adjust this with format_reward_weight:

reward = RewardComponents(
    format_reward_weight=0.2,
    graders=[
        Grader(type=GraderType.MATH_ACCURACY, weight=0.3),
        Grader(type=GraderType.TEXT_SIMILARITY, weight=0.5, operation=TextSimilarityOperation.BLEU)
    ]
)

When format_reward_weight is set explicitly, the sum of all weights (format + graders) must equal 1.0.

​Reward functions

​Grader types

​Create graders

​LLM grader

​Combine graders with weights

​Format reward weight

Reward functions

Grader types

Create graders

LLM grader

Combine graders with weights

Format reward weight