Property Finder found that 96.8 percent of the labels in a computer-vision evaluation set had been copied from its incumbent model’s predictions. That circular setup made alternatives look wrong whenever they disagreed with the system they were supposed to replace, even when the incumbent answer was incorrect.
The real-estate platform evaluates 11 image attributes, including room type, furnishing, watermarks and photo quality. Its audit found hallucinated brand names in watermark labels, missing characters and single-room labels applied to images containing several spaces. Even the small human-labeled portion included categories outside the original specification.
The company rebuilt its ground truth with independent expert annotation. It used 256 UAE listing images to develop prompts and added 100 newly annotated images for the final comparison. A separate set of 230 Egypt images provided a regional check, although AWS says those results are not reported. The team iterated from prompt version zero to version 27, measuring micro F1 scores and studying failures.
AWS says the selected Amazon Bedrock model achieved higher accuracy at 86 percent lower cost. The dataset is still modest, especially for rare attributes, and the case study comes from AWS and its customer. Its broader lesson is stronger than the product claim: a test set generated by the current model measures agreement with that model, not correctness. Independent labels must be created before a replacement can receive a fair evaluation.