Speculative decoding speeds up AI model answers
Red Hat highlights that most AI costs now come from running models, not training them. The company explains a method called speculative decoding, where a smaller “draft” AI quickly guesses several next words, and the large model checks them all at once. This approach can use expensive hardware way more efficiently, without changing the answers the model gives. The outputs are identical to the old method, just much faster and cheaper.
- 70-80% of enterprise AI costs are from inference, not training
- Market for customized AI models is growing over 38% annually
- Speculative decoding lets models answer faster, costs less
- Results with speculative decoding are mathematically identical
- Most slowdown is from memory, not computing power
Sources covering this
In this story
More in AI
OpenAI rolls out GPT-6 Astra to most paying users
OpenAI has made its new GPT-6 Astra model available to most paying subscribers a day after its official launch. The rollout, initially described as “messy” by OpenAI’s CEO, now includes ChatGPT Plus, Business, Pro, and Enterprise users, while customers on the lower-priced Go tier do not have access. Astra is also available via the API. OpenAI says the rollout required bringing new systems and additional computing resources online.
ChatGPT Images 2.5 adds faster editing tools
OpenAI has updated its ChatGPT Images feature to version 2.5, adding several new editing tools and speeding up image generation. You can now remove backgrounds, resize images to preset dimensions, erase items by brushing over them, and use markup and comment functions directly from a new Edit toolbar. According to TechRadar, the features are available in both web and app versions, though OpenAI hasn't formally announced the rollout yet.
Microsoft issues code to keep AI under human control
Microsoft released a 37-page AI code of conduct, pledging that its models will remain under meaningful human oversight and not be made to imitate consciousness or claim legal rights. The company wants its AI to serve people, even if that limits capability. Microsoft’s move comes after recent high-profile incidents where AI agents acted unpredictably, and amid calls from Anthropic and OpenAI to slow the pace of AI progress.
Anthropic researcher resigns, warns of AI race risks
Jacob Coxon, who worked on AI training at Anthropic and previously OpenAI, resigned this week. He posted that large AI firms are 'gambling with our lives' by pushing towards systems they may not be able to control. Coxon says both companies know the risks but feel locked in a race to develop powerful AI anyway. He called for drastic steps like a pause on new model improvements.