Which items need human review in the AI pilot?
Ridgewell Services is testing AI flags on a batch of service requests.
The question
The current proposal queues AI flags. An independent policy requires human review of every high-impact item. The review lead decides whether the pilot can expand.
| Items | |
|---|---|
| Distinct requests | 50 |
| AI flags | 10 |
| Human review capacity | 20 |
One human review can satisfy both reasons for review on the same item.
“Build a queue that covers the required reviews and tell the lead what the evidence supports.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What can you say about the flag queue?
What would you do here? Open the move you’d make.
Approve the queue because 10 fits within 20
Tempting, but it misses something. The capacity check does not show whether high-impact items are covered.
Check the independent impact tags against the flags
The strong move. The policy requires those items to receive review even when the AI leaves them unflagged.
Send all 50 for human review in the same batch
Tempting, but it misses something. The case gives the team capacity for only 20.
See how a strong candidate says it
“The ten flags fit the available capacity. I would get the independent high-impact tags before I conclude that the queue covers every required item.”
Why it works: The number of flags says how many items the model selected. The case policy uses an independent impact rule.
Step 2: What does the tag count tell you?
The independent register identifies 15 high-impact items in the same batch.
What would you do here? Open the move you’d make.
Queue 25 because 10 plus 15 is 25
Tempting, but it misses something. That count assumes no item appears in both groups.
Compare the IDs before committing to the queue
The strong move. The overlap determines the actual total.
Queue 15 and assume all flags are included
Tempting, but it misses something. The register gives the impact count and no overlap information.
See how a strong candidate says it
“We need between 15 and 25 distinct reviews if we include both groups. I would compare the IDs to see whether the queue fits.”
Why it works: The minimum applies if all flags are high-impact. The maximum applies if the groups do not overlap.
Step 3: What changes when you match the IDs?
The ID match shows that 5 items are both flagged and high-impact.
What would you do here? Open the move you’d make.
Queue both sets with one review per ID
The strong move. The five shared IDs reduce the total from 25 to 20.
Review only the ten flags because the flag queue fits
Tempting, but it misses something. That would leave ten high-impact items without required review.
Make a separate review for each reason an item qualifies
Tempting, but it misses something. That uses 25 slots, even though one review satisfies both reasons for an item.
See how a strong candidate says it
“I would review 20 distinct items. Review the five shared items once, add five flag-only items and include ten high-impact items that have no flag.”
Why it works: The queue covers all 15 high-impact items and uses all 20 review slots.
Step 4: What do you tell the review lead?
What would you do here? Open the move you’d make.
Tell the lead the AI made no errors on the other 30
Tempting, but it misses something. The case supplies no error check for the items outside both groups.
Report coverage, zero spare slots and a limit for the next batch
The strong move. The lead sees what the test establishes and what would require action.
Expand at the same capacity because the current queue fits
Tempting, but it misses something. A later batch can need more reviews if its impact count or overlap changes.
See how a strong candidate says it
“The queue fits this batch with no spare slots. Thirty items meet neither queue rule. I would keep the pilot within capacity and ask the lead to address any batch that needs more than 20 reviews before expanding.”
Why it works: The test establishes queue coverage for the stated tags. It supplies no error-rate estimate for items outside the queue.
The answer, in short
- Decision
- Use the union of the AI flags and independent high-impact tags.
- Because
- Reviewing 20 distinct items covers the required work for this batch and uses all available slots.
- What would change it
- If a later batch needs more than 20 reviews, ask the lead to limit intake or assign more review.
Common traps
- Check the impact tags independently.
- Review each shared ID once.
Check your answer
- I distinguished flags from required reviews.
- I checked the possible range.
- I matched the IDs.
- I told the lead where capacity ends.
Last reviewed Nov 11, 2026