Amazon SageMaker JumpStart has added a leaderboard that compares more than 70 open-weight models inside SageMaker Studio. At launch it includes models from Meta, Google, Qwen, IBM and Arcee, with each evaluated through the same AWS methodology rather than a mixture of vendor model cards.
Quality remains split into three public reasoning tests: BIG-Bench Hard, GPQA Diamond and MMLU-Pro. AWS does not combine them into a single score, allowing teams to choose which capability matters. Separate endpoint tests report time to first token, generated tokens per second and estimated hourly instance cost.
Users can filter by provider, license and input or output type, then switch among balanced, lowest-hourly-cost, lowest-latency and highest-throughput deployment configurations. The displayed cost covers the default SageMaker instance only; storage, data transfer, inference components and discounts are excluded.
A common harness improves like-for-like comparison, but benchmark performance is still only a shortlist. AWS recommends deploying candidates and evaluating them on the intended workload. Teams should also confirm license terms and use the same deployment profile they compared, because changing hardware or serving configuration can alter latency, throughput, cost and sometimes quality.