Exa says it improved the performance and cost of embedding one million documents by 47% after optimizing its models for AWS Inferentia2 and Trainium2 chips. The work began when a model upgrade required the search company to re-embed 30 billion high-priority documents while continuing to process new web data.
Engineers removed unnecessary weight transposes, enabled lower-precision matrix operations and replaced generic attention and feed-forward components with custom Neuron Kernel Interface code. The last changes tailored calculations to the model’s exact shapes, reducing padded work and memory transfers. Exa required outputs to maintain 0.999 cosine similarity with its reference model.
Coding agents developed the optimizations through a three-part harness. Each proposed version compiled on real hardware, ran a correctness check, produced a performance profile and completed a 30-minute saturation benchmark. Cheaper models searched compiler settings, while stronger reasoning and coding models investigated bottlenecks and wrote kernels.
The result is a company case study rather than an independent chip comparison, and it reflects Exa’s specific model. Its broader lesson is reproducible: agents were useful because automated checks constrained them. Performance code that merely compiled was not accepted without measured speed and numerical agreement.