Amazon warned engineers that chasing AI usage metrics is 'deeply flawed'

The memo calls indicators such as token consumption, how widely AI tools have been adopted, and how much code is changed or reviewed "deeply flawed" yardsticks for judging AI use across teams or organizations. Amazon said leaning on those measures can drive up spending and make things needlessly complicated.

Amazon did not tell staff to disregard the metrics, according to CNBC. Token usage and code generation can still factor in, the company said, but only "with care and judgment" and next to other data. "We believe that it is best to measure, and be measured by, long-term outcomes," the memo said.

The memo also tells engineers that AI is not the right tool for every task. "If your work does not benefit from current AI tooling, that is a finding, not a failing," it said. "An engineer who can explain where AI does not yet help their work is demonstrating the judgement we are asking for."

Marc Brooker, a vice president and distinguished engineer at Amazon Web Services, told CNBC that Amazon watches "feature velocity" as a way to see whether AI is making its engineers more productive. The metric captures output such as "how many launches are we doing, how many successful changes are we making in production," according to Brooker.

Brooker described the approach as a return on investment. "What we don't want to see is wasted effort where we are going and spending those tokens solving problems that don't matter," he said. He added that the company needs some appetite for experimentation because it can be hard to know which problems matter.

In the guideline, Amazon said AI is changing the work of software engineering more quickly and more fundamentally than earlier technology trends such as the cloud, microservices and serverless. The company said it had avoided making its software engineer role guidelines explicit about technology, and that its most senior engineers debated whether AI warranted a change.

An Amazon spokesperson said employees face no limits on how many tokens they can use. Amazon retired an internal leaderboard in May once it learned that employees were gaming token consumption to move up the standings. The memo made the rounds among staff in September, ahead of its appearance on Amazon's public site.