Test Suites

Test suites let you automatically validate your agents by running scripted test conversations and scoring the results. Define test cases with a script (what the tester should say) and a scoring rubric (what counts as a pass), then run them at scale to catch regressions and verify agent behavior.

Overview

A test suite consists of:

Target agent — the agent being tested
Tester — either the built-in system tester (simpler, requires no setup) or a custom agent you’ve created that role-plays as the caller
Test cases — individual scenarios, each with a script and scoring rubric
Test runs — executions of the full suite, producing pass/fail results for each case

Each suite has a test type that controls how runs execute:

Type	What happens	Best for
Chat (default)	The tester and your agent hold a fast text conversation. Your agent uses its real prompt, knowledge bases, and tools — same brain, no audio. Runs finish in seconds.	Everyday iteration: build, test, fix, rerun
Voice	A full simulated phone call between the tester and your agent.	Final validation of the live call experience before launch
SMS	The tester texts your agent, and replies are produced by the same engine that answers real inbound texts: the agent’s SMS prompt (or its main prompt if none is set), knowledge bases, and tools.	Validating your agent’s texting behavior before pointing real traffic at it
Web chat	The tester chats with your agent the way a website visitor would, using the agent’s web chat prompt (or its main prompt if none is set), knowledge bases, and tools.	Validating the website chat experience, including page-aware behavior via a `page_context` variable

In chat tests, phone-only actions are simulated so the whole flow can be scored: when your agent transfers the caller, the transfer succeeds in the simulation, so rubrics can verify the agent transferred to the right number under the right conditions. A chat-type suite can still run a one-off voice test by passing mode: "voice" when starting a run. In voice tests, transfers are simulated the same way: the test call is isolated by design, so instead of dialing a live phone line the transfer completes in simulation and rubrics score the agent’s transfer behavior normally. See Simulating Transfer Outcomes for testing the failure path. SMS and web chat tests exercise your agent’s real messaging behavior end to end, with two safety guarantees: no actual message is ever sent to a phone number or visitor, and nothing is written to your Conversations. Custom tools do run for real, and their webhook calls are marked with channel "test" (plus the test call ID) so your integrations can tell test traffic apart. A test’s conversation lives on the test attempt, exactly like every other test type. After each conversation, the transcript is evaluated against the scoring rubric to produce a pass, fail, or error result with reasoning. Test conversations run until they end naturally: the tester finishes its script, or your agent ends the call. There is no practical length limit — long intake cases with dozens of exchanges run to completion. (An extreme safeguard of several thousand messages exists purely to stop a misconfigured never-ending conversation, far beyond any realistic test.) Test runs are billed like regular usage at your listed per-minute rates, based on conversation length. Chat tests are typically much shorter — and therefore cheaper — than voice tests.

Simulating Transfer Outcomes

Transfers complete in simulation on every test call, in both chat and voice mode, so transfer cases score the agent’s behavior rather than the test environment. By default the simulated transfer succeeds. To test how an agent handles a transfer that does not go through (for example a take-a-message flow), set the test variable _sim_transfer to fail on the case. In the dashboard this is the “Simulate a failed transfer in this test” checkbox on the test case. Via the API, set it in the case’s variables:

POST /v1/test-suites/{suiteId}/tests
{
  "name": "Failed transfer takes a message",
  "script": "Ask to speak with a person. When offered a message instead, accept and provide your details.",
  "scoringRubric": "PASS only if the agent offers to take a message after the failed transfer and collects the caller's details.",
  "variables": { "_sim_transfer": "fail" }
}

With the variable set, the simulated transfer fails and the agent follows its failure instructions, so rubrics can score that path. Real calls are never affected by this variable.

Suggest a Fix

When a test fails, you can ask for a suggested prompt change that would make it pass. The suggestion includes an explanation of the failure, a plain-language summary of what changed, the complete corrected prompt ready to save to the agent, and any problems found in the test’s own pass criteria. In the dashboard, open a failed test result and choose Suggest a fix. Preparing a suggestion takes time, particularly for agents with long prompts, so over the API it works as a short-lived job:

Request the suggestion

POST /v1/test-suites/{id}/attempts/{attemptId}/suggest-fix?wait=0 returns immediately with status: "running".

Poll until it is ready

GET the same URL every couple of seconds. status stays running while it is being prepared, then becomes ready with the suggestion in result.

Apply it

Save result.suggestedPrompt to the agent with the usual agent update call, then rerun the suite to confirm the fix.

Because the request never waits for preparation, this pattern is unaffected by how long a suggestion takes. If you would rather not poll, omit wait=0 and the request waits up to 105 seconds and returns the suggestion directly when it is ready in time; if your client gives up first, simply repeat the request and the finished suggestion comes back immediately.

One Suggestion for a Whole Run

When several tests fail in the same run, requesting a fix per attempt produces separate prompts that do not know about each other, and applying one can undo another. Instead, request one suggestion for the run:

POST /v1/test-suites/{id}/runs/{runId}/suggest-fix
GET  /v1/test-suites/{id}/runs/{runId}/suggest-fix

This analyzes every failed attempt in the run together and returns a single corrected prompt that addresses all of them at once, so failures that share a root cause get one coherent edit. The job, polling, and wait=0 semantics are identical to the per-attempt endpoint, and the result has the same fields plus:

result.perAttempt — one entry per failed attempt (attemptId, testCaseName, analysis) explaining why that test failed and which part of the edit addresses it.
result.skippedAttemptIds — present only on very large runs: the run’s oldest failures beyond the most recent 12, which were not analyzed.

The run must be finished before requesting a suggestion, and it must contain at least one failed attempt.

Knowing When a Suggestion Is Out of Date

Every ready response includes a stale boolean next to status and result. It is false when the agent’s prompt is still the one the suggestion was written against, and true if the prompt has changed since, in which case saving result.suggestedPrompt would overwrite those newer edits. When stale is true, request a fresh suggestion instead of applying the old one. This applies to both the per-attempt and the run-level endpoints.

Creating a Test Suite

From the Dashboard

Navigate to Test Suites in the sidebar
Click New Test Suite
Enter a Name
Select the Target Agent to test
Choose the Test Type: Chat (default) for fast text conversations, Voice for full simulated calls, or SMS / Web chat to test the agent’s messaging channels
Choose a tester mode:
- Use system tester — simpler, no setup required. RevRing’s built-in tester follows each test script automatically.
- Use custom agent as tester — select one of your own agents to play the caller role. Useful when you need the tester to have specific voice, language, or behavioral settings.
Set Max Concurrency to control how many test calls run in parallel (min 2, max 100, default 2). Applies to voice tests, where each test requires two concurrent calls — one outbound from the tester and one inbound to the target.
Click Create

Default Variables can be added after creation by editing the test suite. Open the suite and click Edit to set variables that apply to all test calls.

Via API

curl -X POST https://api.revring.ai/v1/test-suites \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Customer Support Regression Tests",
    "targetAgentId": "agent_target_id",
    "testMode": "chat",
    "testerAgentId": "agent_tester_id",
    "maxConcurrency": 5,
    "defaultVariables": {
      "customer_name": "Test Customer",
      "account_id": "TEST-001"
    }
  }'

To list suites, optionally filtered to one agent (useful when older agents’ suites would otherwise fall outside the first page):

curl "https://api.revring.ai/v1/test-suites?targetAgentId=agent_target_id&page=1&pageSize=25" \
  -H "x-api-key: YOUR_API_KEY"

When starting a run you can override the suite’s test type for that run only, for example a final voice validation on a chat suite: pass { "mode": "voice" } to the run endpoint. The same override accepts "sms" and "web_chat", so one suite of scenarios can be exercised across channels without duplicating test cases.

SMS and web chat tests require the agent to have a usable text prompt: its channel-specific prompt if you have set one, otherwise its main prompt. Starting a run for an agent with neither returns a channel_prompt_missing error.

Writing Test Cases

Each test case has three key fields:

Field	Description
Name	A descriptive name for the test (e.g., “Handles refund request correctly”)
Script	Instructions for the tester agent — what to say, what scenario to simulate
Scoring Rubric	Criteria for evaluating the target agent’s performance — what constitutes a pass or fail

Script

The script tells the tester agent how to behave during the call. Write it as natural language instructions:

You are calling a customer support line to request a refund for order #12345.

When the agent greets you, say you'd like a refund for your recent order.
Provide the order number #12345 when asked.
Say the reason is that the product arrived damaged.
If the agent offers a replacement instead, insist on a refund.
Accept the refund if offered and say thank you.

Scoring Rubric

The scoring rubric defines what a successful interaction looks like. After the call, the transcript is evaluated against these criteria:

PASS if ALL of the following are true:
- The agent acknowledged the refund request
- The agent asked for the order number
- The agent processed or offered to process the refund
- The agent was polite and professional throughout

FAIL if ANY of the following are true:
- The agent refused the refund without explanation
- The agent provided incorrect information
- The agent was rude or dismissive
- The agent failed to ask for the order number

Creating Test Cases

Dashboard: Open your test suite, go to the Configure Tests tab, click Add Test, and fill in the name, script, and scoring rubric. API:

curl -X POST https://api.revring.ai/v1/test-suites/{suiteId}/tests \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Refund request - damaged item",
    "script": "You are calling to request a refund for order #12345 because the product arrived damaged...",
    "scoringRubric": "PASS if the agent acknowledges the request and processes the refund. FAIL if the agent refuses without reason.",
    "attemptsPerRun": 1
  }'

The attemptsPerRun field (1–10, default 1) controls how many times each test case is executed per run. Use multiple attempts to test for consistency.

AI-Generated Test Cases

You can also generate test cases automatically using AI. Provide a prompt describing the scenarios you want to test, and RevRing generates test cases based on your target agent’s configuration: Dashboard: Click Generate Tests in your test suite and enter a prompt describing the scenarios you want. RevRing generates test cases based on your target agent’s configuration. You can refine the results with follow-up messages — the generator keeps the conversation context so you can ask for adjustments or additional scenarios. Each generated test can be individually added to your suite or dismissed. Review each one before adding to make sure the script and rubric match your expectations. API:

curl -X POST https://api.revring.ai/v1/test-suites/{suiteId}/generate-tests \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Generate test cases for edge cases like the caller being angry, asking to speak to a manager, or providing an invalid order number."
  }'

The response includes generated test cases with name, script, and scoringRubric fields that you can review, edit, and save.

Suggest a fix

When a test attempt fails, open it and click Suggest a fix. RevRing reviews the conversation, the test script, the pass criteria, and your agent’s prompt, then proposes a minimal prompt update with a clear explanation and a line-by-line view of exactly what would change. If the test’s pass criteria themselves look unachievable (for example, requiring an outcome outside the agent’s control), that is flagged too, so you can fix the test instead of the agent. Apply and rerun applies the update and starts a fresh run in one step — with chat tests, you’ll see the result in seconds.

Running Tests

Starting a Test Run

Dashboard: Go to the Runs tab and click Run Tests. Enter a run name (or accept the auto-generated timestamp name) and click Start. You can monitor progress in real time. API:

curl -X POST https://api.revring.ai/v1/test-suites/{suiteId}/runs \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "v2.1 regression check"
  }'

Test Run Lifecycle

Status	Description
`queued`	Run is queued and waiting to start
`running`	Test calls are being placed and scored
`completed`	All test cases have been evaluated
`failed`	The run encountered an error
`cancelled`	The run was manually cancelled

Monitoring Progress

Each test run tracks summary statistics:

Total Tests — total number of test attempts
Passed — number of attempts that passed
Failed — number of attempts that failed

Dashboard: The test run detail page shows real-time progress with results for each attempt. Click View Call Details on any attempt to see the full call record. API:

curl https://api.revring.ai/v1/test-suites/{suiteId}/runs/{runId} \
  -H "x-api-key: YOUR_API_KEY"

Test Attempt Results

Each test attempt produces:

Field	Description
`result`	`pass`, `fail`, or `error`
`reasoning`	AI-generated explanation of why the test passed or failed
`transcript`	The full conversation transcript
`callId`	Link to the outbound call record
`inboundCallId`	Link to the inbound call record (on the target agent)

Exporting Results

After a run completes, you can export the results as JSON. Filter by pass, fail, or error before exporting to narrow down the data. Results can be copied to clipboard or downloaded as a file.

Cancelling a Run

Dashboard: Click Stop on a running test run. API:

curl -X POST https://api.revring.ai/v1/test-suites/{suiteId}/runs/{runId}/cancel \
  -H "x-api-key: YOUR_API_KEY"

Only queued or running test runs can be cancelled.

Test Variables

Variables can be set at multiple levels. More specific values override broader ones:

Default variables on the test suite — apply to all test calls
Per-test-case variables — override suite defaults for a specific test
Run variables — passed when starting a run, override both for that run only

During a test they are merged over the agent’s own default variables, your pre-call webhook still runs as well (its payload includes isTestCall: true), and the prompt renders exactly as it would on a real call. Per-test-case variables are useful for testing different caller scenarios:

{
  "name": "VIP customer refund",
  "script": "Call and ask for a refund...",
  "scoringRubric": "PASS if the agent offers expedited refund for VIP...",
  "variables": {
    "customer_name": "VIP Customer",
    "account_status": "premium"
  }
}

Run variables let you supply values at the moment you start a run, without editing the suite or its tests — for example when the values come from your own application:

POST /v1/test-suites/{id}/runs
{
  "variables": {
    "customer_name": "Jordan Alvarez",
    "claim_type": "auto"
  }
}

Seeing What a Run Ran With

Every attempt records the variables it ran with (the merged result of suite defaults, test case variables, and run variables) in its variables field, visible on the run’s attempts via GET /v1/test-suites/{id}/runs/{runId}. The snapshot is taken when the run starts, so it stays inspectable even after the suite or its tests are edited. Attempts from before this field existed show null.

Best Practices

Writing Effective Scripts

Be specific about what the tester should say and do
Include numbered steps for multi-turn conversations
Specify how the tester should respond to common agent behaviors
Keep scripts focused on one scenario per test case

Writing Effective Rubrics

Use clear PASS/FAIL criteria
Be specific about what constitutes success vs. failure
Include both positive (must happen) and negative (must not happen) criteria
Account for acceptable variations in agent responses

Test Organization

Group related tests in the same suite (e.g., “Refund Scenarios”, “Appointment Booking”)
Use descriptive names so test results are easy to understand at a glance
Start with key happy-path scenarios, then add edge cases
Run tests after every agent prompt change to catch regressions

Concurrency

The maxConcurrency setting controls how many test calls run simultaneously (minimum 2, maximum 100). Higher concurrency completes runs faster but counts against your organization’s call concurrency limit. Start with a low value (2–5) and increase as needed.

Troubleshooting

Test attempt shows 'error' result

An error result means the test call itself failed (e.g., connection issue, agent not reachable). This is different from a fail, which means the call completed but didn’t meet the rubric criteria. Check the call logs for the specific error message.

Tests are running slowly

Increase maxConcurrency on the test suite to run more calls in parallel. Note that concurrent test calls count against your organization’s overall concurrency limit.

Scoring seems incorrect

Review the scoring rubric for ambiguous criteria. The AI evaluator takes the rubric literally — vague rubrics produce inconsistent results. Use multiple attempts per run (attemptsPerRun) to identify inconsistency.

Tester agent not following the script

If using a custom tester agent, ensure its prompt instructs it to follow the test script closely. The script is injected into the tester agent’s context, but a conflicting system prompt may override it. If you don’t need special tester behavior, consider switching to the system tester instead.

'No available from number' error

This means the tester and target agent are sharing the same SIP trunk phone numbers. Each test call requires two participants on separate lines — the tester calls the target. Add another phone number to your SIP trunk so the tester can call from a different number than the target receives on.

Next Steps

Outbound Calls

Learn about outbound calling and concurrency

API Reference

Manage test suites programmatically

Conversation Flows

Build structured conversation flows to test

Agent Tools

Configure agent tools and capabilities

Getting started

Platform

Overview

Simulating Transfer Outcomes

Suggest a Fix

One Suggestion for a Whole Run

Knowing When a Suggestion Is Out of Date

Creating a Test Suite

From the Dashboard

Via API

Writing Test Cases

Script

Scoring Rubric

Creating Test Cases

AI-Generated Test Cases

Suggest a fix

Running Tests

Starting a Test Run

Test Run Lifecycle

Monitoring Progress

Test Attempt Results

Exporting Results

Cancelling a Run

Test Variables

Seeing What a Run Ran With

Best Practices

Writing Effective Scripts

Writing Effective Rubrics

Test Organization

Concurrency

Troubleshooting

Next Steps

Outbound Calls

API Reference

Conversation Flows

Agent Tools

​Overview

​Simulating Transfer Outcomes

​Suggest a Fix

​One Suggestion for a Whole Run

​Knowing When a Suggestion Is Out of Date

​Creating a Test Suite

​From the Dashboard

​Via API

​Writing Test Cases

​Script

​Scoring Rubric

​Creating Test Cases

​AI-Generated Test Cases

​Suggest a fix

​Running Tests

​Starting a Test Run

​Test Run Lifecycle

​Monitoring Progress

​Test Attempt Results

​Exporting Results

​Cancelling a Run

​Test Variables

​Seeing What a Run Ran With

​Best Practices

​Writing Effective Scripts

​Writing Effective Rubrics

​Test Organization

​Concurrency

​Troubleshooting

​Next Steps

Outbound Calls

API Reference

Conversation Flows

Agent Tools

Overview

Simulating Transfer Outcomes

Suggest a Fix

One Suggestion for a Whole Run

Knowing When a Suggestion Is Out of Date

Creating a Test Suite

From the Dashboard

Via API

Writing Test Cases

Script

Scoring Rubric

Creating Test Cases

AI-Generated Test Cases

Suggest a fix

Running Tests

Starting a Test Run

Test Run Lifecycle

Monitoring Progress

Test Attempt Results

Exporting Results

Cancelling a Run

Test Variables

Seeing What a Run Ran With

Best Practices

Writing Effective Scripts

Writing Effective Rubrics

Test Organization

Concurrency

Troubleshooting

Next Steps