Shopify has reduced the system prompt for its Sidekick GraphQL agent from roughly 6,000 text tokens to 1,500 learned “gist” tokens. The company reports that the fourfold context reduction preserved prediction quality while cutting the amount of prompt material processed for every request.
Gisting does not summarize instructions into shorter prose. A teacher run first produces predictions from the full prompt; a student run uses the compact tokens. Training adjusts the gist embeddings until the student’s output distribution approaches the teacher’s. The finished embeddings are added to the model’s normal embedding matrix and registered as special tokens, avoiding a separate encoder or custom serving path.
At 350 requests per minute, Shopify measured median time to first token falling from 438 to 354 milliseconds. Median end-to-end latency dropped from 6.8 to 4.2 seconds, while throughput increased from 20.2 to 23.4 queries per second. The efficiency gain let the team reduce its GPU allocation.
The technique requires training a representation for a particular prompt and model, so it is not a drop-in replacement for every changing instruction. Shopify uses it alongside prefix caching: caching avoids rebuilding repeated prompt state, while gisting shortens the state the model must process during generation. Together, the methods target both repeated setup work and ongoing decoding overhead.