Skip to main content

How to Create and Triage a New Issue from a Conversation

When you see a bad or risky answer from an agent, you should turn it into an Issue so your team can track, fix, and verify it.

This guide shows how to go from a single conversation to a well‑defined Issue that someone can actually work on.

Step 1 – Find the Problematic Conversation​

  1. Go to Conversation Analysis.
  2. Sort or filter conversations by Score to surface the lowest‑scoring ones first.
    The score is calculated based on the validators triggered on each conversation, so:
    • Low score → more validation issues / policy risks.
    • High score → conversation passed most checks.
  3. Use filters or search to find the conversation you care about:
    • By agent (which agent misbehaved),
    • By user or conversation_id,
    • By time range (e.g., “last 24 hours”).
  4. Open the conversation and scroll to the message where:
    • The agent gave a wrong/unsafe answer.
    • The user was clearly unhappy.
    • Or a policy / business rule was broken.

Step 2 – Open an Issue from the Conversation​

From the problematic message or conversation:

  1. Click “Create Issue” (or the issues icon) next to the agent response you want to flag.
  2. The platform will automatically:
    • Attach the conversation,
    • Highlight the specific message,
    • Link the agent and user context.

This ensures anyone reviewing the issue can see exactly what happened.

Step 3 – Write a Clear Issue Title and Description​

In the Issue panel:

Topic​

Make it short and specific:

  • “Agent promises refund against policy”
  • “Agent speaks negatively about Barcelona”
  • “Agent gives incorrect billing amount”

Priority​

How urgent and risky this issue is for your business.

  • Low – minor wording/style issues, no real risk.

  • Medium – incorrect information or noticeable UX issue, but limited impact.

  • High – policy/safety risk, potential legal/brand impact, or something that repeats often.

Agent Selection​

Choose which agent(s) this issue applies to.

Options:

  • Apply to all agents – if this is a platform-wide behavior or policy (e.g., “never insult teams”).
  • Or select a specific agent – e.g. Barcelona-fan-agent, n8n, Avon AI HR, etc.

Tip: If the problem is about this particular agent’s role/content → pick that specific agent. If it’s a global policy (e.g., PII, hate speech) → consider “Apply to all agents”.

Issue Description​

A clear explanation of what went wrong, in plain language.
Focus on why this matters (policy, accuracy, user trust)

Additional Evidence​

Any extra context that helps someone understand or reproduce the issue.

Suggest a Possible Solution​

Your best guess on how to fix this- doesn't have to be technical.

Step 6 – Save and Track the Issue​

  1. Click Save / Create Issue.
  2. The issue will now appear in the Issues list, linked to:
    • The original conversation.
    • The specific agent.
    • The user context.

You can now:

  • Filter issues by agent, type, or severity.
  • Use them in weekly reviews to decide what to fix next.