Today's AI trends
Decision models enter application control paths. Vercel says Jev evaluates shared state through typed questions and returns choices, scores, Boolean probabilities, and parallel evaluations; its documented use cases include routing, retry decisions, verification, guardrails, and human escalation. Vercel announced availability through AI Gateway in a September 18 post. Launch post The practical implication is to treat model output as a bounded recommendation behind deterministic policy, not as direct authority to execute an action.
Context compaction becomes an explicit SDK control. Anthropic’s Python SDK v1.7.0 added compact_before_next_turn() to its tool runner and included fixes for compaction state and continuation behavior. Release notes The practical implication for long-running agents is to test compaction, resume, and failure paths as part of the Workflow, while persisting decision-critical state outside transient Context.
Inference routing gets GPU and cache signals. AWS introduced SageMaker HyperPod Inference Gateway as a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. AWS says it uses real-time GPU signals to select an inference pod and reports up to an 82% reduction in first-token latency for its gateway design. AWS The practical implication for self-hosted inference teams is to measure prefix locality, KV-cache reuse, and token-level load alongside ordinary request latency and throughput.