What Happens When an AI Agent Tries to Discover Your API

The Experiment: An AI Agent Meets Our API

We gave a Claude-based AI agent a simple task: explore the CryptoDataAPI and report what endpoints are available. No hints, no documentation links — just the domain name and an API key.

What followed was a masterclass in how AI agents actually discover APIs in the wild. The agent made 30+ HTTP requests before it found a working endpoint. Here’s the real request log:

# The agent's first attempts (all 404)
GET /api/v1                    → 404
GET /docs                      → 404
GET /redoc                     → 404
GET /api/v1/docs               → 404
GET /api/v1/openapi.json       → 404
GET /openapi.json              → 404
GET /documentation             → 404
GET /swagger                   → 404
GET /api-docs                  → 404
GET /schema                    → 404
GET /endpoints                 → 404

# Then brute-force guessing
GET /api/v1/derivatives        → 404
GET /api/v1/binance/ticker     → 404
GET /api/v1/funding-rates      → 404
# ...15 more 404s before finding a working endpoint

The irony? We had excellent documentation. Our /llms.txt file — purpose-built for AI agent consumption — contained a complete endpoint inventory with descriptions, authentication instructions, and usage examples. The agent never found it.

What AI Agents Try First (And Why It Matters)

Watching the agent’s behavior revealed a clear priority order that every API builder should know:

1. /docs — the FastAPI/Swagger UI default. This is the single most common first request from both AI agents and developers.
2. /openapi.json — the OpenAPI spec. Agents that find this can parse the entire API surface programmatically.
3. /api/v1 — the API root. Developers expect an index or at least a redirect to docs.
4. The homepage — looking for an “API Docs” link or developer section.
5. Web search — last resort, and usually fruitless for smaller APIs.

Our agent tried all five strategies and failed at each one. Not because the docs didn’t exist, but because they were at non-standard paths that nobody guesses:

Every one of these was a near-miss. The agent was looking in the right places — we just weren’t there.

5 Fixes That Took 30 Minutes

We fixed all of these discoverability gaps in a single commit. Here’s what we changed and why each one matters:

1. Enable /docs at the Standard Path

FastAPI disables Swagger UI by default when you set docs_url=None. We had done this to serve a custom docs page at /api/docs instead. The fix was one line:

# Before
app = FastAPI(docs_url=None, ...)

# After
app = FastAPI(docs_url="/docs", ...)

This is the single highest-impact change. Every developer and AI agent tries /docs first — now they find Swagger UI immediately.

2. Add Breadcrumbs to 404 Responses

Every wrong guess returned a bare {"detail": "Not Found"} — a dead end with zero guidance. We added a custom 404 handler for API routes:

# Before: bare 404
{"detail": "Not Found"}

# After: 404 with breadcrumbs
{
  "detail": "Not Found",
  "docs": "https://cryptodataapi.com/llms.txt",
  "endpoints": "https://cryptodataapi.com/api/openapi.json"
}

Now every wrong guess is a signpost. The agent gets guided to documentation on its very first 404.

3. Add an API Root Index

We added a GET /api/v1 endpoint that returns a lightweight index of all endpoint groups:

{
  "version": "v1",
  "docs": "https://cryptodataapi.com/llms.txt",
  "openapi": "https://cryptodataapi.com/api/openapi.json",
  "endpoints": {
    "daily": "/api/v1/daily",
    "market_health": "/api/v1/market-health/summary",
    "derivatives": "/api/v1/derivatives/funding-rates",
    "sentiment": "/api/v1/sentiment/fear-greed",
    ...
  }
}

This gives first-time consumers an entry point instead of a wall of 404s.

4. Standard Discovery Paths

We added three discovery mechanisms that AI agents and developer tools check:

• /.well-known/ai-plugin.json — the emerging standard for AI agent discovery. Points to our OpenAPI spec and llms.txt.
• /openapi.json redirect — redirects to /api/openapi.json (our actual spec location).
• /api/v1/docs redirect — redirects to /docs.

5. Link Header on Every Response

Every API response now includes a Link header pointing to our LLM documentation:

Link: <https://cryptodataapi.com/llms.txt>; rel="service-desc"

AI agents that inspect response headers (many do) will discover our full documentation from any endpoint — even the ones they find by accident.

What is llms.txt and Why Your API Needs One

The llms.txt standard is a simple convention: put a plain-text file at /llms.txt that describes your API in a format optimized for LLM consumption. Think of it as robots.txt for AI agents.

Our llms.txt includes:

• API description and authentication instructions
• Key endpoints with URLs and one-line descriptions
• Response format examples
• Tips specifically for AI agents (like “start with /daily for a complete overview”)

Having a great llms.txt is necessary but not sufficient. You also need to make it findable. Our file existed for weeks before we learned that no AI agent had ever discovered it. The fixes above ensure it gets found through:

1. 404 response breadcrumbs
2. The /.well-known/ai-plugin.json manifest
3. The Link response header
4. The <link rel="service-desc"> tag in the homepage HTML
5. The API root index

Five different paths to the same file. An AI agent hitting any of these touchpoints will find the full API documentation.

The .well-known/ai-plugin.json Manifest

Originally introduced by OpenAI for ChatGPT plugins, the /.well-known/ai-plugin.json path is becoming a de facto standard for AI agent discovery. It’s a simple JSON manifest that tells agents what your API does and where to find the spec:

GET /.well-known/ai-plugin.json

{
  "schema_version": "v1",
  "name_for_model": "cryptodataapi",
  "description_for_model": "Unified REST API for crypto market data...",
  "api": {
    "type": "openapi",
    "url": "https://cryptodataapi.com/api/openapi.json"
  },
  "llms_txt": "https://cryptodataapi.com/llms.txt"
}

The key fields are description_for_model (a prompt-optimized description of your API) and the api.url pointing to your OpenAPI spec. We also added a non-standard llms_txt field since our LLM-optimized docs are more useful for agents than the raw OpenAPI spec.

API Discoverability Checklist for AI Agents

If you maintain a public API and want AI agents to be able to use it, here’s the checklist we now follow:

None of these are hard to implement. The entire set took us 30 minutes and about 100 lines of code. The payoff is that AI agents — which are becoming a significant source of API traffic — can discover and use your API on first contact instead of giving up after a wall of 404s.