LinkedIn says a collection of infrastructure changes made training its AI job-search ranker about eight times faster. The production system distills judgments from larger “teacher” models into a 600-million-parameter student that can rank results cheaply enough for a high-volume search service.
During early experiments, an SGLang-based online mode serves several teachers directly inside the training loop. Local replicas across multiple nodes tripled distillation speed while letting researchers change teachers quickly. Once those choices stabilize, an offline mode precomputes teacher outputs and stores them on HDFS or NFS, avoiding repeated model calls.
Other gains come from larger batches through LiGer, multi-node training, FSDP2 and H200 clusters. LinkedIn found FP8 arithmetic did not help models below eight billion parameters because conversion overhead outweighed the savings. Together, rather than through one isolated trick, the changes produced the reported speedup.
The student combines guidance from an eight-billion-parameter relevance model and a 1.7-billion-parameter engagement model. LinkedIn reports a 24.48 percent gain in NDCG@10, a ranking-quality measure, and inference throughput rising from roughly 290 to more than 2,000 items per second per GPU after pruning and context compression. The system now powers natural-language job search for US users.