Microsoft Announces Open Source Tool Evaluating Your Software's Optimization For AI Coding Agents (microsoft.com) 29
The problem? Understanding how AI coding agents interact with the software you build. "Whether you're maintaining an SDK, building a command-line interface, or publishing an MCP server or skills... you need to understand whether agents can discover and use what you ship, and what your team can change."
Results can vary, and configurations keep changing, with behaviors influenced by everything from custom instructions and MCP servers to skills and plugins. So after a year of development Microsoft has created Scope, "an open-source platform built on Kubernetes for turning those interactions into evidence product teams can act on."
Details are available on... the Microsoft open source blog: Available under the MIT License, Scope supports a continuous Developer Agent Experience (DevAx) learning loop grounded in repeatable experiments.
The Azure Cosmos DB team's experience shows what that loop can deliver... "The evaluations led to 20 Agent Kit issues and 27 documentation issues resolved," [says Jay Gordon, Senior Program Manager, Azure Cosmos DB]... These results connect evaluation to decisions a product team can make: which documentation needs clarification, where tooling needs a fix, and whether an update changes agent behavior.... [The gains] demonstrate how targeted changes, informed by observed behavior, can improve the way agents use a product.
For more on the shift from building for human developers to supporting agents, watch Your Product Was Built for Humans. Is It Ready for Agents? or join us at AI DevCon NYC conference in November.
Scope surfaces everything from incorrect assumptions in documentation (or recurring missed instructions) to whether agents select the intended SDK, initialize it correctly, and use its APIs, leading to clearer descriptions for command-line tool and more actionable errors.
Microsoft's open source blog also explains why AI agent evaluation needs an open-source foundation: The challenges behind agent experience aren't unique to one product or organization. SDK maintainers, tool authors, researchers, and agent developers are all asking how agents behave and what helps them succeed. No single team can represent every language, workflow, or integration. An open project can bring more scenarios, more failure cases, and more viewpoints into the evaluation process... Opening Scope's execution and evaluation code gives contributors a way to question the methods, investigate unexpected results, and improve the evaluation itself. A shared foundation also reduces duplicated infrastructure. Agent integrations, reusable criteria, and trace-analysis improvements can help other teams study their own products without rebuilding the same machinery. The goal isn't one universal benchmark or definition of success. It's a common starting point that teams can adapt to their needs.
The MIT License lets teams use and extend Scope, while self-managed deployments let them evaluate private workloads without publishing their code or run data.... Scope streams activity as the agent works and retains interactions and workspace snapshots for later review. Your team can follow tool calls, inspect generated files, and connect a failed requirement to the steps that preceded it. Those traces provide observable evidence of what the agent did, helping turn a failure into a concrete documentation, SDK, or tooling improvement.
Results can vary, and configurations keep changing, with behaviors influenced by everything from custom instructions and MCP servers to skills and plugins. So after a year of development Microsoft has created Scope, "an open-source platform built on Kubernetes for turning those interactions into evidence product teams can act on."
Details are available on... the Microsoft open source blog: Available under the MIT License, Scope supports a continuous Developer Agent Experience (DevAx) learning loop grounded in repeatable experiments.
The Azure Cosmos DB team's experience shows what that loop can deliver... "The evaluations led to 20 Agent Kit issues and 27 documentation issues resolved," [says Jay Gordon, Senior Program Manager, Azure Cosmos DB]... These results connect evaluation to decisions a product team can make: which documentation needs clarification, where tooling needs a fix, and whether an update changes agent behavior.... [The gains] demonstrate how targeted changes, informed by observed behavior, can improve the way agents use a product.
For more on the shift from building for human developers to supporting agents, watch Your Product Was Built for Humans. Is It Ready for Agents? or join us at AI DevCon NYC conference in November.
Scope surfaces everything from incorrect assumptions in documentation (or recurring missed instructions) to whether agents select the intended SDK, initialize it correctly, and use its APIs, leading to clearer descriptions for command-line tool and more actionable errors.
Microsoft's open source blog also explains why AI agent evaluation needs an open-source foundation: The challenges behind agent experience aren't unique to one product or organization. SDK maintainers, tool authors, researchers, and agent developers are all asking how agents behave and what helps them succeed. No single team can represent every language, workflow, or integration. An open project can bring more scenarios, more failure cases, and more viewpoints into the evaluation process... Opening Scope's execution and evaluation code gives contributors a way to question the methods, investigate unexpected results, and improve the evaluation itself. A shared foundation also reduces duplicated infrastructure. Agent integrations, reusable criteria, and trace-analysis improvements can help other teams study their own products without rebuilding the same machinery. The goal isn't one universal benchmark or definition of success. It's a common starting point that teams can adapt to their needs.
The MIT License lets teams use and extend Scope, while self-managed deployments let them evaluate private workloads without publishing their code or run data.... Scope streams activity as the agent works and retains interactions and workspace snapshots for later review. Your team can follow tool calls, inspect generated files, and connect a failed requirement to the steps that preceded it. Those traces provide observable evidence of what the agent did, helping turn a failure into a concrete documentation, SDK, or tooling improvement.
What's not to like? (Score:2)
Re: (Score:3)
It doesn't store the information is blockchains.
Re: (Score:2)
And it doesn't use MongoDB.
MongoDB is webscale. [youtube.com]
Not the problem in my mind (Score:3, Insightful)
translation (Score:1)
It doesn't answer a rather important question tho (Score:2)
Why should I care?
Maybe someone can give a cogent explanation of why I should care about this. But, on the face of it, I couldn't care less whether software I write can be easily ingested by an AI agent. I'm not writing code with a goal of making myself, or anyone who follows me in my role, redundant.
(Actually, regarding myself - I really wouldn't care if I got laid off at this point - it would just mean retiring now instead of sometime next year. But don't expect my help if the AI that's supposed to replac
Re: (Score:2)
Re: (Score:2)
There's probably at least a smattering of middle managers here convinced that they "get it" because they read Slashdot. This slashvertisement is aimed at them.
Re: (Score:2)
As said elsewhere, this is an ad. However, it's the start of "AI SEO" - that is, make sure *your* app/website/documentation is easy for agents to use, because if you don't someone else's will, and you'll lose out.
You might then think Microsoft's documentation would be available in HTML for humans, and Markdown for Agents - but no, they haven't got around to that yet. I don't suppose they have an MCP for it yet either.
oh dear... (Score:4, Insightful)
Mickeysoft lost the plot again.
The game-changing element of LLMs and agents is that they have a non-computer-language interface - you can talk to them in plain English (or Chinese or whatever). This makes complex computer interactions available to a much wider audience than any previous computing invention. Adapting humans to the AI is getting that backwards. It's a sign you don't understand the first thing about why AI is so widely successful.
I predict this will be an abject failure and a year from now will quietly be pushed to the very back of the bottom-most drawer, never to be spoken off again (though it is getting crowded in there for M$).
Meanwhile, in the real world, everyone I've talked to about these things already has a skills file to teach the AI about their personal coding style. This is what actual humans came up with, independently several times, and it works: Adapt the AI to you. Plus we've been writing coding style guides for humans for a long time now, we've got plenty of experience doing that at least somewhat right.
testing the docs (Score:2)
Adapting humans to the AI is getting that backwards.
i don't think that's what they're doing. some of the "gaps" they found in their documentation they classify as "broken samples, stale references, and missing language notes (whatever that is)". this is not a matter of style but correctness and consistence in documentation which should be a good thing for both agents and humans.
what's odd is their approach: instead of methodically checking the consistency itself they just let a bunch of agents loose trying to build a set of programs, then traced the problems
Re: (Score:2)
But again that's backwards. I let my AI coding assistant write tests exactly so that I have a reproducible, deterministic ground truth that I can inspect for logic errors. If you let the AI do the testing directly, it's a toss-up if today it feels like it or it decides that your desired result is "everything green" so it'll just give you that answer without actually testing anything.
Re: (Score:2)
i totally agree, but again we don't know if that's exactly what they are doing. their priority seems to be the improvement of their agents' output above all else, so those sets of apps they task the agents with would be the equivalent of test cases (defined by humans). if that is their goal and those "test cases" are sensible then that's just a form of testing of the documentation, if a bit convoluted. personally i don't agree with any of it, their motivation, their priorities or the process itself but ...
Re: (Score:2)
The game-changing element of LLMs and agents is that they have a non-computer-language interface - you can talk to them in plain English (or Chinese or whatever).
But no, you can't. This becomes immediately apparent when you try to talk to them in the way you would talk to a normal person. You ask them for something which should be simple because what you want to know is what most people would want to know. You ask it the simple question and it frequently takes the most obscure view of the question possible. So then you have to rephrase the question once or twice to be incredibly specific in a way that would be unnecessary if you were talking to a human, since they a
Re: (Score:2)
LLMs obviously aren't human. But that's not the point. The point is that you don't need to know a programming language to program a computer.
For example: Especially in the early days of smartphones, there were mountains of small tools that did something that's essentially a simple script. Egg timers, remind-me-tomorrow apps, tons of stuff that's basically a wrapper around a web search, and so on. Almost all of those are today something ChatGPT can do, and are easy enough to describe.
Re: (Score:2)
I would not trust ChatGPT to do literally any of those things correctly, as they can all be misinterpreted, and even when I ask AI to answer simple questions it often does that. If I want a timer, I run the timer app on my phone. If I want a reminder tomorrow, I run the calendar app on my phone. There's no question about whether it got the information into it I wanted, because I can see that I put it in there. I don't have to wonder whether the LLM will fail to put the information in, but then confidently t
Re: (Score:2)
You missed the point.
A friend of mine made a tea-timer app for the iPhone. Where you enter what tea you are making and it sets the correct number of minutes. Today, people would ask ChatGPT before they go on the App Store to search for something like that. That's the point. Not what YOU would do, but what the general public does. And for Jane Doe, ChatGPT has replaced a lot of these specialty apps.
Re: (Score:2)
Copilot is a stupid grey box with no correct/grounded in reality documentation on how to use it or configure it. You can ask Copilot, but it doesn't have to tell you the right answer. You could put copilot instructions in a .copilot file or something as the documentation suggested, but Copilot would tell you to put them somewhere else, and there wasn't a clear winner on what was correct/what worked. The instructions didn't work no matter where you put them and nobody could tell you where they needed to go.
Re: (Score:2)
Skills are great until they aren't. The problem is exactly what you say: You can literally create a skill by telling the LLM to create a skill. Which is... why the f do I need to make a skill if the AI already knows how to do it?
Properly created skills, that include reference material, specific instructions, and expert knowledge the LLM doesn't have or that's buried somewhere under 10 layers of reddit posts is genuinely useful. But most skills aren't. I'm constantly underwhelmed by hyped-up skills that, whe
Nope (Score:1)
This is an embarrassing turn of events. There's no such thing as optimizing code for AI. Either the AI knows what the fuck it's doing or it doesn't. If it struggles with some codebase that's because it's not smart enough, not because there's a problem with the codebase.
What is it (Score:2)
This is the example that helped me understand what they built [github.io]. It's a framework for benchmarking different configurations. If you want to know whether Claude or ChatGPT works better, this will help you. IF you want to know if adding a skill [agentskills.io] is worth it, then this will help you determine and keep track.
Then it will put it into a pretty graph that will make CEOs think you've done something useful.
The goalposts have become agentic! (Score:2)
Most popular use will be.... (Score:2)
....ensuring projects are hard to access by AI, including, but not limited to, new "are you human" checks.
Microsoft sacraficing Windows for AI (Score:2)
Wait, is this just a documentation quality rating? (Score:2)