Question-answering bot with confidence scoring

This recipe builds a question-answering bot with SeekrFlow that pairs every answer with a confidence score, a signal for how much to trust the reply. It calls a chat model directly through SeekrFlow’s serverless inference and parses the rating out of the response, with no extra frameworks.

What you’ll build

A Q&A bot that:

Answers questions about any topic.
Returns a confidence score with each answer.
Flags low-confidence answers for follow-up.

Prerequisites

A SeekrFlow API key, set as the SEEKR_API_KEY environment variable
Python 3.8 or later
The SeekrFlow SDK: pip install seekrai

Each question you ask is billed as serverless inference, charged by the token.

Build it

Set up the client

Create qa_bot.py with the imports, configuration, and client.

import os
import re

from seekrai import SeekrFlow

API_KEY = os.environ["SEEKR_API_KEY"]
API_URL = "https://flow.seekr.com/v1/"
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"

client = SeekrFlow(api_key=API_KEY, base_url=API_URL)

Write the prompt

Write a prompt that asks the model for an answer followed by a confidence rating on its own line. Parsing depends on that Confidence: marker. Keep it in the instructions.

TEMPLATE = """Question: {question}

Answer the question above. Then, on a new line starting with 'Confidence:', rate
your confidence from 1 (complete guess) to 10 (absolutely certain), and briefly
explain the rating.
"""

Parse the response

Write a helper that splits the answer from the confidence rating and its explanation.

def parse_response(response: str) -> dict:
    """Extract the answer, confidence score, and explanation from a reply."""
    answer, confidence, explanation = response, "Not provided", ""

    if "Confidence:" in response:
        answer, _, confidence_part = response.partition("Confidence:")
        answer = answer.strip()
        confidence_part = confidence_part.strip()
        rating_match = re.search(r"(\d+)(/10)?", confidence_part)
        if rating_match:
            confidence = f"{rating_match.group(1)}/10"
            explanation = re.sub(r"^\d+(/10)?", "", confidence_part).strip()

    return {"answer": answer, "confidence": confidence, "explanation": explanation}

Ask a question

Wrap the model call and parser in a function that prints the answer, its confidence, and a follow-up hint when confidence is low. A low temperature keeps answers more deterministic.

def ask_question(question: str) -> dict:
    """Send a question to the model and print the parsed result."""
    response = client.chat.completions.create(
        model=MODEL_ID,
        messages=[{"role": "user", "content": TEMPLATE.format(question=question)}],
        temperature=0.3,
    )
    parsed = parse_response(response.choices[0].message.content)

    print(f"Question: {question}\n")
    print(f"Answer: {parsed['answer']}\n")
    print(f"Confidence: {parsed['confidence']}")
    if parsed["explanation"]:
        print(f"Explanation: {parsed['explanation']}")

    if parsed["confidence"] in ("1/10", "2/10", "3/10"):
        print("\nLow confidence. Consider rephrasing or asking for more specifics.")

    return parsed

Ask a few questions

Run a few questions of varying difficulty to see the confidence score change.

questions = [
    "What is the capital of France?",
    "How do quantum computers work?",
    "When was the book 'Trilby' by George du Maurier published?",
]

for q in questions:
    ask_question(q)
    print("-" * 50)

Run the script

Run the finished script:

python qa_bot.py

Easy, well-known questions should come back with high confidence; obscure ones should score lower and trigger the follow-up hint.

Next steps

Keep context across turns. Hold a running messages list and include prior turns in each call to answer follow-up questions in context.
Tune the threshold. Adjust which confidence scores trigger the follow-up hint to match how cautious you want the bot to be.
Ground the answers. Combine this with the New hire onboarding agent with citations so answers come from your own documents.

​What you’ll build

​Prerequisites

​Build it

​Next steps

What you’ll build

Prerequisites

Build it

Next steps