Relay.app Docs
  • Getting Started
    • Introduction
    • Helpful Resources
    • FAQ
  • Triggers
    • Triggers 101
    • Webhook Trigger
    • Mailhook Trigger
    • Manual Trigger
    • Scheduled Trigger
    • Batch Triggers
    • RSS Trigger
  • Actions in Apps
    • Actions 101
  • DATA
    • Step Outputs
    • Find Steps
    • Lists (Arrays)
    • Inspecting Run Data
  • AI
    • AI Steps
    • Human-in-the-Loop AI reviews
    • AI Credits
    • Agentic Tool Use
    • Knowledge
    • Prompt Templates
    • Prompt Tips
    • Audio Transcription
  • Built-in actions
    • Transform data
    • Search Google
    • Scrape Text from Website
    • Custom HTTP Requests
    • Run Custom Code (JS)
  • Flow Control
    • Paths
    • Iterators
    • Wait steps
    • Sequences
  • Human-in-the-Loop
    • Human-in-the-Loop Steps
    • AI output reviews
    • Roles
  • Workflows
    • Folders and Organization
    • Sharing Workflows
    • Headings
    • Notes
  • TEMPLATES
    • About Workflow Templates
    • Using a Template (Importing)
    • Creating a Template (Exporting)
  • Workspace
    • Step & AI credit usage
    • Billing and Plans
    • Workspace administration
    • (Sharing) App Accounts
  • App-Specific FAQs
    • Airtable
    • Attio
    • Cal.com
    • Coda
    • DeepSeek
    • Discord
    • Fireflies
    • Gmail
    • Google Docs
    • Google Drive
    • Google Sheets
    • Microsoft Permissions
    • Microsoft Outlook Mail
    • Notion
    • OpenAI
    • OpenPhone
    • Slack
    • X (Twitter)
    • QuickBooks Online
Powered by GitBook
On this page
  • Setting up AI output review in the Editor
  • 1. Select the review conditions
  • 2. Choose the mechanism to notify the reviewer
  • 3. Select the reviewer (assignee)
  • 4. Personalize the notifications and reminders
  • Reviewing AI outputs

Was this helpful?

  1. Human-in-the-Loop

AI output reviews

Last updated 24 days ago

Was this helpful?

Relay.app's built-in Human-in-the-Loop tools make it very easy to ensure that a human first reviews the output of an AI step before the workflow proceeds.

Setting up AI output review in the Editor

In the Behavior section of every AI step, there's a question 'Should a person review the AI output?' that lets you (conditionally) turn on Human-in-the-Loop AI output reviews.

1. Select the review conditions

By default, the option to require a human review is turned off for newly created steps. To turn it on, toggle this setting to either

  • Yes, always if you want the AI output to be reviewed every time by a person before continuing the run. Note that if this workflow runs very frequently, you'll also get many review requests that need to be acted on

  • Only if at least one output field is empty

  • Only if all output fields are empty

  • Only if AI cannot generate a valid output

2. Choose the mechanism to notify the reviewer

Next, select the medium the reviewer should receive the notification on.

3. Select the reviewer (assignee)

4. Personalize the notifications and reminders

Finally, you may personalize the notification message that's sent every time a human review is needed. You may also set a due date and configure reminder notifications for when the assignee didn't review the AI output in time.

Reviewing AI outputs

Every time someone is notified to review the output of the AI, they have the following options:

  • Approve the AI output as-is. After the approval, the run will continue with the next step(s) in the workflow.

  • End the workflow run right away. If based on the AI output the assignee decides the workflow run shouldn't continue, they can end it right away, skipping all following steps.

  • Manually edit/override the AI output (i.e. to tweak values)

  • Re-prompt the AI. A new ouput will be genearted based on updated one-off instructions/clarifications in the prompt. The assignee will be notified again when the new output is ready for review (they'll have the same options again).

The assignee for the reviews should be a collaborator in your workflow. .

Learn more about Collaborators here
Human AI Output reviews are available for all AI steps
Select the medium to notify the reviewer
The assignee should be a collaborator of the Workflow
Optionally, personalize the review notifications and reminders