Evals, explained for Quality Assurance
How do you prove an AI change did not make the dossier worse? Like a QC test: a reference standard, a criterion set in advance, a recorded result.
Anyone in Quality Assurance has a ready question for any system that touches a regulated process: how do you know it still works after a change? For an analytical method, the answer is well known. For an artificial intelligence system, the answer tends to be vague: "we tested it", "it seems better", "users liked it".
At Bloom, the answer has a name: an eval.
What an eval is
An eval is the systematic, repeatable assessment of a system's output against an expected result. The logic is that of a quality control test:
- there is a reference standard;
- there is an acceptance criterion, defined before the test;
- the result is recorded.
If that structure sounds familiar, it is because it is the same one your department requires of a validated process.
What the output is compared against
A section drafted by the platform is compared with three kinds of reference:
- Previously approved dossiers. Real cases that went through Anvisa, with a known regulatory outcome. The most demanding reference available.
- Expert answer keys. Correct answers built by hand for test cases, including the hard ones and the known traps of each section.
- Criteria per section. Mandatory fields present, required format respected, the source cited for every claim.
The judge did not write it
Evaluation is done by a second agent, which receives the section produced, the source documents and that section's criteria, and answers objective questions:
- Is every claim supported by the cited passage?
- Are the mandatory fields filled in?
- Does the section contradict another part of the same dossier?
- Do format and terminology follow what Anvisa requires?
- Are units, dates and identifiers consistent?
- Does anything marked not applicable have a justification?
Separating who drafts from who judges is deliberate. The same agent tends to accept its own reasoning, just as nobody reviews their own report well.
Change control
This is the part Quality cares about most. The reference set is fixed and grows with every new case. When something changes in the platform, whether a model, an instruction or a rule, the whole set is reprocessed and the outputs are compared with the approved ones.
If a section that used to be correct starts to diverge, the change is held until the cause is understood. No change goes into production because it looked better on a single case.
That is the difference between "the new model looks good" and "the new model was reprocessed against the entire reference set and no approved section regressed".
And after the eval
The eval does not replace the person. It exists so the regulatory expert receives material that is already consistent and can spend their time on what takes judgement. Each evaluator's opinion is recorded next to the section, and no dossier is delivered without an expert's approval and signature.
If your department would like to go through the method in detail, or needs to send a supplier qualification questionnaire, write to our team. The full description is on the Dossier validation and accuracy page.