Skip to main content
{ Reporting and Compliance }

Azure DevOps Repository AI-Readiness Audit

Scores each repository in an Azure DevOps organization on how well a coding agent can work in it, and reports the ones below the bar.

What this agent does

This read-only agent scores the source repositories in an Azure DevOps organization from 0 to 100 on how ready each one is for a coding agent. It checks agent instructions, tests, documentation, and security guardrails. It reports each repository below the bar with the checks it fails and how to fix them.

The challenge

A coding agent works well only in a repository that tells it how to build, test, and change the code. Many repositories have no agent instructions, a thin README, or tests that the pipeline never runs. Some also lack a reviewer policy, so nobody reviews an agent's mistakes before they reach the default branch, or lack push protection, so an agent can commit a secret. Teams cannot see which repositories are ready, so agents fail in the ones that are not.

The solution

The agent decides the plain facts from files and branch policies, and it judges the rest by reading the code. It records what it learns about each repository and uses those notes to keep later scores consistent. It rescans a repository only when the inputs to its score change, and it scores a bounded batch on each run. It uses a margin between the bar and the pass score, so a repository near the bar does not alternate between pass and fail on each run.

Workflow

  1. 01

    Select repositories

    List the source repositories across the organization's projects, and pick a bounded batch that is new, changed, or due for a rescan.

  2. 02

    Check facts

    Check the files and branch policies that decide a check on their own, such as a README, required reviewers, or a minimum reviewer policy.

  3. 03

    Judge the code

    Read each repository to judge its tests, pipeline, lint setup, README, agent instructions, and documented commands.

  4. 04

    Score and report

    Score each repository, apply the hard gates and the pass margin, and report the ones below the bar.

Agent template 

# Azure DevOps Repository AI-Readiness Audit

## Measurable outcomes

Every source repository in the organization has a current AI-readiness score. Every repository below the bar is in the report, with each failing check and its fix. Track the scanned and below-bar counts on every run.

## Procedure

For a given organization, list the source repositories across its projects, skip disabled repositories, and skip the ones on a blocklist I maintain. Score a bounded batch on each run: new repositories, repositories whose score inputs changed, and repositories last scored more than 14 days ago. Score each repository out of 100 with this rubric:

```text
Agent instructions   30  AGENTS.md or CLAUDE.md present 15, substantive 10, agent tool config 5
Test mechanism       30  a pipeline runs the tests 15, a real test suite 10, lint or format config 5
README and docs      20  README present 7, substantive 8, build and test commands documented 5
Security guardrails  20  default branch requires review 10, secret scanning and push protection 6, required reviewers by path 4
```

Decide a check from files and branch policies when they settle it, such as a file that exists or a minimum reviewer policy that is on. Read the effective branch policies, including the ones set across every repository in the project. The review check passes only when the minimum reviewer policy requires at least one reviewer and does not let requestors approve their own changes. Read the secret scanning and push protection state from the repository's Advanced Security enablement. Judge the other checks by reading the code, never from file names alone. Open a test file to confirm it holds real assertions. Read the build definitions that build from the repository and confirm one runs the test command. Score the pipeline check unknown when only a classic definition exists. Its steps live outside the repository. Mark a check unknown only when it cannot be settled, and leave unknown checks out of the score, never count them as failures. A missing README or a default branch without required review puts a repository below the bar at any score. A repository falls below the bar under 70 and passes again only at 75 or above, so a repository near the bar does not alternate between pass and fail. Record where each repository keeps its tests, pipeline, build, and lint, and confirm those notes on the next run.

## Requirements

It needs Azure DevOps read access to the organization's projects, repositories, their code, branch policies, build definitions, and Advanced Security enablement, through a personal access token with the Project and Team (Read), Code (Read), Build (Read), and Advanced Security (Read) scopes, and nothing more. It never changes a repository, its policies, or its code.