Open-sourcing its agent evaluation platform, Microsoft is betting that how well agents use a product will soon matter as much as how well humans do. For CIOs, that shifts what “developer experience” means.
Software vendors have spent years polishing documentation and SDKs for human developers. Microsoft now argues the audience has changed: AI coding agents discover APIs, pick SDKs, follow docs, and call tools on a developer’s behalf, and an experience that works for a person can still lead an agent astray.
Its answer is Scope, an open-source platform, released under the MIT License, for evaluating AI coding agents. Built on Kubernetes, it drives existing agents such as GitHub Copilot, Claude Code, and Visual Studio Code with Copilot. Teams define a representative task, run it repeatedly, and score the results against criteria expressed as a dependency graph, with an LLM-based judge and full traces to show where an agent went wrong. Microsoft calls the discipline Developer Agent Experience, or DevAx, and wants it to become a continuous loop: test, fix the docs or tooling, test again.
The Azure Cosmos DB team offers the proof point. Comparing bare models against its Agent Kit across eight scenarios in five languages led to 20 Agent Kit and 27 documentation fixes. After one targeted update, hierarchical partition-key adoption rose from 47% to 100%, and OFFSET/LIMIT usage fell from 47% to 0%. Microsoft stresses that those gains apply to those scenarios, not to agents in general.
The broader impact is a shift in what counts as product quality. If agents increasingly choose which SDK to install, vendors with confusing docs or weak MCP servers may simply be passed over. Supporting Claude Code alongside Copilot suggests Microsoft wants Scope to be a neutral testbed, though it remains a Microsoft-originated project, and whether outside contributors shape it is an open question. Because runs can be scripted, agent behavior can also be tracked release over release, like performance or security testing.
For CIOs, the lesson reaches past vendors: internal APIs, SDKs, and developer portals now have agent users too. Scope is self-managed, so teams must run the infrastructure, secure the data and credentials, and pay for model usage. Microsoft itself cautions that agent behavior varies and automated judges can err, so scores should inform decisions, not make them. Its advice is to start with one task that matters, define success, and run it repeatedly.