Skip to main content

How to Use Issues to Improve Your System Prompt

The System Prompt is the brain of your agent.
Issues + Conversation Analysis are your feedback loop to keep improving it.

This recipe shows how to turn bad conversations into concrete System Prompt improvements, using your existing Issues flow.

Step 1 – Find Conversations with Real Problems​

  1. Go to Conversation Analysis.
  2. Sort or filter conversations by Score, from lowest to highest.
    • Low Score = more validators fired / more problems in this conversation.
  3. (Optional) Filter by Agent if you want to focus on a specific agent.
  4. Open a low‑scoring conversation and:
    • Look at which validators triggered,
    • Identify the agent message that is clearly wrong, unsafe, off‑tone, or off‑scope.

Step 2 – Create an Issue from the Problematic Conversation​

From the problematic agent reply:

  1. Click “Create Issue” from the conversation (or from the specific message, if available).

  2. Fill the Issue fields:

    • Topic – Make it clear and short, e.g.

      • Policy – Agent promises refund against rules
      • Tone – Negative answer about Barcelona
      • Content Quality – Wrong explanation of billing plan

    • Priority –

      • High if policy/safety/brand risk,
      • Medium if incorrect but not dangerous,
      • Low if only style/tone.

    • Agent Selection –

      • Apply to all agents for global behavior (e.g., “never insult teams”),
      • Or select a specific agent (e.g., Barcelona-fan-agent) if it only applies there.

    • Issue Description – Explain what happened and why it’s a problem:

    • Additional Evidence – Add any context:

      • “Happened 3 times this week on the same agent.”
      • “User was clearly frustrated after this answer.”

    • Suggest a Possible Solution – Write in plain language:

      • “Update System Prompt to forbid promising refunds.”
      • “Add a rule to stay neutral about football teams.”
      • “Clarify that the agent must answer only HR questions.”

  3. Save the Issue.

Step 3 – Identify Patterns Across Multiple Issues​

Before touching the System Prompt, look for repeated patterns:

  1. Go to the Issues page.
  2. Filter by:
    • Agent
    • Category / Topic (Content, Policy, Tone, Translation if you use that style),
    • Priority (start with High + Medium).
  3. Look for clusters of similar issues, for example:
    • Many issues where the agent:
      • Promises refunds it shouldn't.
      • Speaks negatively about certain teams.
      • Answers out‑of‑scope HR/legal questions.
      • Mixes languages (Hebrew/English) in a confusing way.

Write down 2–3 common patterns. These are the best candidates for System Prompt updates.

Step 4 – Open the Agent’s System Prompt​

For the agent you want to fix:

  1. Go to the AI Admin Panel.
  2. Open the relevant Agent.
  3. Scroll to the Generation section:
    • Model
    • Temperature
    • Max Tokens
    • System Prompt

You will edit the System Prompt based on what you learned from Issues.

Step 5 – Translate Issues into System Prompt Rules​

Take each recurring issue pattern and turn it into a clear instruction in the System Prompt.

5.1 – Add “Must Not Do” rules​

From issues:

  • Issue examples:
    • “Agent speaks badly about Barcelona.”
    • “Agent promises refunds against policy.”

Convert into System Prompt lines:

What you must not do:

  • Do not speak negatively about any football team or player.
  • Do not promise refunds, discounts, or account changes. Instead, explain the policy and suggest contacting support or billing.

5.2 – Clarify Scope and Out-of-Scope​

From issues:

  • “HR agent answered billing questions.”
  • “Support bot is answering legal questions.”

System Prompt addition:

Scope:

  • You only answer questions about [HR / support / sports stats / etc.].
  • Do not answer questions about [billing / legal / HR / etc.].
  • If a question is out of scope, say you are not the right assistant and suggest where to go.

5.3 – Adjust Tone and Language Based on Issues​

From issues:

  • “Tone is too harsh for angry users.”
  • “Mixing English and Hebrew.”

System Prompt addition:

Tone and language:

  • Use a friendly and respectful tone, even if the user is upset.
  • Never insult users, teams, or players.
  • Answer in [Hebrew/English] by default, or in the same language as the user when possible.

Step 6 – Save and Test the Updated System Prompt​

After updating the System Prompt, save the agent and then test it in the product:

  • Go to "Chat with Agen"t or create a test in the "Test library"

Start a new test conversation that mimics the problematic case, e.g.:

  • Ask for a refund.
  • Insult a team.
  • Ask an out‑of‑scope question.
  1. Check if:
    • The agent now refuses or redirects correctly,
    • Tone and content match your new System Prompt rules.

If not, refine the System Prompt wording and test again.

Step 7 – Link the Fix Back to the Issue and Close the Loop​

For each Issue that drove a System Prompt change:

  1. Open the Issue.
  2. Add a comment like:

    “Updated System Prompt for your-agent to forbid negative comments about teams and stay neutral. Also tested in chat with agent with similar questions.”

  3. If the fix works in new conversations:
    • Change the Issue status to Resolved / Closed.
  4. If only partially fixed:
    • Keep the Issue open.
    • Note what still needs to be changed (e.g., Guardrail, Validation Rule, Translation).

Over time, you’ll build a clear history:

  • Which System Prompt changes were made,
  • Why they were made,
  • And which Issues they solved.