Adding more language-model checks does not automatically turn unreliable extraction into correct data, according to an InfoQ experiment that tested six reliability layers on the same records.
The authors used the CADEC corpus of 1,250 forum posts with annotated adverse-drug reactions. Their zero-shot extractor reached roughly 0.47 to 0.50 F1 under lenient span matching on development data, below a supervised system trained on hundreds of labeled files. They then tested deterministic validation, retrying failures, repeated sampling and voting, a second-model judge, withholding uncorroborated answers and human review.
The layers mainly separated answers by how much they could be trusted; they corrected few mistakes. Even identical runs at temperature zero varied by four F1 points. Asking a model to recall medical codes directly was especially poor: some returned codes that did not exist, while retrieving candidate codes deterministically and asking the model to choose performed better.
The study is an engineering case study, not a universal benchmark. Its judge was small, and the public dataset may have appeared in training data. Still, it supports a practical design rule: measure the share of usable outputs, replace generation with deterministic lookup where possible, and fix extraction quality before stacking costly verification calls.