AI output reviews

Relay.app's built-in Human-in-the-Loop tools make it very easy to ensure that a human first reviews the output of an AI step before the workflow proceeds.

Setting up AI output review in the Editor

In the Behavior section of every AI step, there's a question 'Should a person review the AI output?' that lets you (conditionally) turn on Human-in-the-Loop AI output reviews.

1. Select the review conditions

Human AI Output reviews are available for all AI steps

By default, the option to require a human review is turned off for newly created steps. To turn it on, toggle this setting to either

• Yes, always if you want the AI output to be reviewed every time by a person before continuing the run. Note that if this workflow runs very frequently, you'll also get many review requests that need to be acted on

• Only if at least one output field is empty

• Only if all output fields are empty

• Only if AI cannot generate a valid output

2. Choose the mechanism to notify the reviewer

Select the medium to notify the reviewer

Next, select the medium the reviewer should receive the notification on.

3. Select the reviewer (assignee)

The assignee should be a collaborator of the Workflow

The assignee for the reviews should be a collaborator in your workflow. Learn more about Collaborators here.

4. Personalize the notifications and reminders

Optionally, personalize the review notifications and reminders

Finally, you may personalize the notification message that's sent every time a human review is needed. You may also set a due date and configure reminder notifications for when the assignee didn't review the AI output in time.

Reviewing AI outputs

Every time someone is notified to review the output of the AI, they have the following options:

• Approve the AI output as-is. After the approval, the run will continue with the next step(s) in the workflow.

• End the workflow run right away. If based on the AI output the assignee decides the workflow run shouldn't continue, they can end it right away, skipping all following steps.

• Manually edit/override the AI output (i.e. to tweak values)

• Re-prompt the AI. A new output will be generated based on updated one-off instructions/clarifications in the prompt. The assignee will be notified again when the new output is ready for review (they'll have the same options again).