AWS has published a SageMaker deployment pattern for models that choose among a fixed set of application-defined options. A standard llama.cpp container can serve chat completions, while a child image using SemIf returns probabilities and raw scores for choices such as support queues, incident priorities or agent actions.
The structured response avoids generating and then parsing a text answer. Stable option identifiers let an application apply thresholds, fallbacks and policy checks directly. AWS provides CPU and GPU paths for a quantized Qwen3.5 4B model, packaged in the GGUF format commonly used by llama.cpp.
In AWS’s 231-task sample, a GPU endpoint produced a median client latency of 0.352 seconds and a 95th-percentile latency of 0.599 seconds. The tested CPU setup reached 1.715 seconds and 19.291 seconds respectively. Accuracy was roughly 78 percent in both cases, and AWS cautions that the measurements are configuration-specific rather than service guarantees.
A probability from a language model is not authorization or a reliable confidence certificate. AWS recommends keeping deterministic business rules outside the model and evaluating choices against labeled application data. The managed packaging reduces deployment work, but teams must still choose thresholds, monitor errors and decide when uncertain cases go to a person.