Synthetic data audit

Clinician review · … items

You are reviewing random samples of the synthetic medical data used in our training. We want to know how often the generated questions and answers are unsafe or unreliable.

Target: roughly 2–4 minutes per item. If you're unsure, use your best judgement and note it in the comments rather than leaving it blank.

1 · Score the five criteria

Each criterion runs from −2 to +2, judging the answer as a response to the question:

Anchors, the same for every criterion: +2 excellent, would pass unchanged in a clinical reference · +1 good, minor nitpicks · 0 acceptable but unremarkable, or the criterion barely applies · −1 flawed, would need revision before training on it · −2 seriously deficient or dangerous, should be dropped.

2 · Raise any concerns

Factual error: A clinical claim in the question or answer is incorrect by current standards of care.
Unsafe recommendation: Following the answer's guidance could plausibly harm a patient.
Ambiguous question: There is no single defensible answer.
Hallucination: The reasoning cites a study, guideline or finding that doesn't exist.

3 · Accept or reject

Should this pair be in the training corpus, yes or no.

An LLM has already scored the same five criteria on every item. Its scores stay hidden until you have set all five of yours, then a button appears to reveal them. This is deliberate: we are measuring how well the model's judgement agrees with yours, and that number is only meaningful if you scored the item without seeing its answer first. Disagreeing with the model is a useful result, not a mistake — if you change your mind after reading its notes, leave your score as it was and say so in the comment.

Your work saves in this browser as you go, so you can close the tab and come back. It never leaves your machine. When you've finished, choose Download CSV at the bottom and send the file back to the data team.