With SeekrFlow™, deploying to a staging environment means promoting a model for inference. However, before using the inference model in your production environment, focus on validation and verification of your model’s performance to ensure a smooth transition.

Run validation checks

Make sure your model is ready for production deployment by running comprehensive validation checks.

Prepare representative validation data

Start by curating diverse validation datasets that mirror real-world inputs, including edge cases and difficult examples your model will encounter in production.

Example: For a customer service chatbot handling clothing returns, include:

• Simple queries ("How do I return this shirt?")
• Complex scenarios ("I received the wrong size in a different color than ordered")
• Edge cases ("I started a return but the tracking shows it's still at my house")
• Multi-intent queries ("I want to exchange this and add something to my order")

Run comprehensive checks

Next, evaluate prediction quality and system performance to ensure all production requirements are satisfied.

Track critical metrics

Statistics are a critical tool for making sure your AI is trustworthy. The following are some commonly-used statistical metrics used to evaluate a model's performance:

Prediction quality metrics

In practice, there's often a trade-off between minimizing false positives and false negatives. The relative cost of each error type helps determine whether to prioritize precision or recall when optimizing a model. The F1 score is specifically designed to balance the concerns of both, by combining precision and recall into a single metric.

Going back to the clothing returns chatbot, you might prioritize F1 score when the costs of incorrectly rejecting valid returns (customer dissatisfaction) and incorrectly accepting invalid returns (financial loss) are both significant concerns that need to be balanced.

System performance metrics

Latency: Response time per prediction
Throughput: Prediction volume capacity (e.g., 1000 requests/second)

Identify areas for improvement and iterate

Conduct an error analysis

Categorize and investigate patterns in incorrect predictions to identify underlying causes.

Implement targeted improvements

Apply insights from error analysis to refine the model through iterative improvements:

• Hyperparameter tuning
• Additional training data
• Model architecture modifications

Promote a model for inference

After promoting your model to production with SeekrFlow, the next step is to use it for inference. This involves sending input data to the model and receiving predictions. Here’s a detailed guide on setting up and performing inference with your promoted model.

List fine-tuning jobs

Endpoint: GET v1/flow/fine-tunes List Fine-Tuning Jobs

To get the job ID, list all model fine-tunes that were previously created.

client.fine_tuning.list()# List all jobs
client.fine_tuning.retrieve(ft_id) # List specific job

Promote a model

Endpoint: POST v1/flow/fine-tunes/{fine_tune_id}/promote-model Promote Model

When you've found your job ID, promote it for inference:

client.fine_tuning.promote(id="ft-1234567890")

Run inference on streaming chat completions

Chat completions are a great way to test your model's task- or domain-specific performance, as well as gauge end-user experience.

Endpoint: POST v1/inference/chat/completions Create Chat Completion Request

stream = client.chat.completions.create(
    model=ft_deploy.id,
    messages=[
        {"role": "system", "content": "You are SeekrBot, a helpful AI assistant"},
        {"role": "user", "content": "who are you?"}
    ],
    stream=True,
    max_tokens=1024,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Note: For model, you can choose any of our base supported models or models that have been promoted for inference.

Configure parameters in the request body

This guide will help you understand what each parameter does and how to tweak them in the request body for best results.

Example: Input messages

"messages": [ 

    {"role": "user", "content": "How do I reset my password?"}, 

    {"role": "assistant", "content": "To reset your password, go to the account settings and click on 'Reset Password'."} 

] 

Return token log probabilities during inference

You can also return the token log probabilities, or "logprobs". Logprobs reveal the model’s certainty for each generated token.

Low-confidence predictions highlight gaps in training data. During staging, you can flag outputs with low confidence (e.g., logprobs ≪ 0) for manual review or retraining.

Unusually high logprobs for irrelevant tokens can signal hallucinations. During staging, this can help refine prompts or adjust temperature settings.

The code below follows the OpenAI convention for request formatting:

• To return the logprobs of the generated tokens, set logprobs=True.
• To additionally return the top n most likely tokens and their associated logprobs, set top_logprobs=n, where n > 0.

client = SeekrFlow(api_key=os.environ.get("SEEKR_API_KEY"))
stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Tell me about New York."}],
    stream=True,
    logprobs=True,
    top_logprobs=5,  # NOTE: Max number, m, depends on model deployment spec; n > m may throw validation error
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
    print(chunk.choices[0].logprobs)

Demote a model

Endpoint: GET v1/flow/fine-tunes/{fine_tune_id}/demote-model Demote a Model

When you're done with your model, unstage it:

client.fine_tuning.demote(id="ft-1234567890")