Your site has a
second reader now.
Technical SEO for the agent web. People get your full design. AI crawlers and agents get clean Markdown, structured data and tools they can call, from the same URL. We build it, or brief your team, and test it with real agents.
Trusted by B2B teams at
In short
What technical SEO with MaximusLabs is, in two sentences.
Technical SEO for the agent web makes one site work for four readers: search engines, AI crawlers, AI agents and people.
MaximusLabs makes your pages fast, indexable, readable without JavaScript and usable through Markdown, MCP, WebMCP and A2A, then measures every visit.
Your site now has four readers
One URL, served right to each- Search enginesIndex it
Googlebot and Bingbot render your pages and rank them. Still the base layer.
- AI crawlersQuote it
GPTBot, ClaudeBot and PerplexityBot read your HTML or Markdown, then cite you.
- AI agentsAct on it
Browser and API agents call your tools, fill your forms and book the demo.
- PeopleUse it
Your full design, unchanged. Same facts for every reader, never a different story.
Where agents
get stuck today.
We test your site the way each reader meets it: Googlebot, AI crawlers that fetch raw HTML, browser agents that click, and API agents that call tools. You get a score per layer and the fixes that move it.
Every reader tested
The same pages fetched as Googlebot, an AI crawler, a browser agent and an API agent.
Each layer checked
Access, rendering, Markdown, schema, protocols and actions, one by one.
Fixes in order
What to build first, ranked by what it unlocks for buyers and agents.
| Check | Googlebot | AI crawler | Browser agent | API agent |
|---|---|---|---|---|
| Can reach the page | Pass | 2 of 4 bots | Pass | — |
| Sees the prices and plans | Pass | JS only | Pass | — |
| Gets a Markdown version | — | None | — | None |
| Knows who Acme is | Pass | Pass | Pass | Pass |
| Can book a demo | — | — | Unlabelled form | No tool |
Rendered for
every reader.
The major AI crawlers fetch your raw HTML without running JavaScript. If pricing, features or docs only appear after scripts load, they see an empty shell. We move key templates to server or static rendering and keep the interactive UI for people.
Raw HTML vs rendered
What each crawler sees on every key template, side by side.
The right mode per template
Server, static or hybrid rendering, built in the stack you already run.
Text first, interactivity after
Interactive parts load on top of the text, never instead of it.
<html>
<head>
<title>Pricing | Acme</title>
<script src="/app.js"></script>
</head>
<body>
<div id="root"></div>
</body>
</html>No plans, no prices, no features. Nothing to quote.One URL,
two formats.
When an AI agent asks for a page, it gets clean Markdown: the content without layout or scripts, so it reads and quotes you accurately. A person on the same URL still gets your full design.
Content negotiation
Agents that ask for Markdown get Markdown. Browsers get HTML. Same URL.
llms.txt and Markdown copies
A map of your key pages for AI tools, plus a .md copy of each.
Parity check
Both formats carry the same facts on every release. Never a separate version for bots.
| Page | HTML → Markdown | Parity |
|---|---|---|
| /pricing | 214 KB → 4 KB | Match |
| /product/forecasting | 256 KB → 7 KB | Match |
| /vs/kestrel | 188 KB → 5 KB | Price differs |
| /docs/integrations | 142 KB → 9 KB | Match |
| /security | 96 KB → 3 KB | Match |
# Acme > Revenue forecasting for B2B teams. ## Product - [Pricing](/pricing.md) - [Integrations](/integrations.md) - [Security](/security.md)
Speak the
agent protocols.
Agents don't only read pages. They call tools. We build these endpoints for you, or spec them for your team, each scoped to exactly what you choose to expose.
MCP server
Lets assistants like ChatGPT and Claude search your docs, pricing and integrations.
WebMCP tools
Page actions like book_demo that browser agents can call. A proposed standard, in a Chrome trial.
A2A and NLWeb
An agent card other agents can read, and a plain-language endpoint over your content.
| Protocol | What agents get | Where it lives | Status |
|---|---|---|---|
| MCP | Search docs, pricing and integrations from any assistant | mcp.acme.com | Live |
| WebMCP | Page tools a browser agent can call: book_demo, compare_plans | on /pricing, /demo | Building |
| A2A | An agent card other agents can read | /.well-known/agent-card.json | Live |
| NLWeb | Plain-language questions over your site | /ask | Live |
document.modelContext.registerTool({
name: "book_demo",
description: "Book an Acme demo",
inputSchema: { email, teamSize },
execute: bookDemo
})Pages agents
can operate.
Browser agents read your page much like a screen reader does: through labels, roles and structure. Unlabelled buttons and div-only forms stop them. We make every key action clear to agents and to people.
Accessibility tree audit
Every button, link and field checked for a name, role and state.
Forms agents can finish
Labelled fields, clear errors, and no hover-only menus or hidden steps.
Tested with real agents
We run browser agents through your demo and trial flows, before and after.
| Role | Accessible name | Agent can use it? |
|---|---|---|
| heading | "Book a demo" | Yes |
| textbox | "Work email" | Yes |
| textbox | (no name) | No · team size has no label |
| combobox | "Region" | Yes |
| button | (no name) | No · icon-only submit |
| link | "See pricing" | Yes |
Tell machines
what you are.
Structured data tells agents and AI engines exactly what your company and product are. We write one connected JSON-LD graph into your server HTML, tie it to your profiles, and tell engines when it changes.
One graph, server-rendered
Company, product, articles and FAQs in the first HTML response, linked by @id.
One connected entity
sameAs links tie your site to the profiles you already own.
Freshness signals
Accurate lastmod dates and IndexNow pings on every change.
{
"@context": "https://schema.org",
"@type": "SoftwareApplication",
"name": "Acme",
"applicationCategory":
"BusinessApplication",
"publisher": {
"@type": "Organization",
"@id": "#acme",
"sameAs": [
"linkedin.com/company/…",
"wikidata.org/wiki/…"
]
}
}| Template | Status |
|---|---|
| HomepageOrganization · WebSite | Valid |
| Product pagesSoftwareApplication | Valid |
| Blog articlesArticle · FAQPage | Valid |
| Comparison pagesWebPage · ItemList | Warning |
| PricingProduct · Offer | Missing |
Decide who gets
in, and why.
Not every bot wants the same thing. Some train models, some answer searches, some fetch a page because a person asked. We set a policy per purpose and keep the crawlers behind AI answers open.
A policy per purpose
Training, search and user-triggered fetches, each allowed or blocked on purpose.
Verified agents
Signed agent requests are recognised, not challenged.
CDN and firewall rules
Stops scrapers without blocking AI answers.
| Crawler | Used for | robots.txt | CDN / WAF |
|---|---|---|---|
| ChatGPT search | Allowed | Allowed | |
| Model training | By choice: no | Allowed | |
| User-asked fetch | Allowed | Challenged | |
| Perplexity search | Allowed | Challenged | |
| Gemini training | Allowed | n/a | |
| Google Search | Allowed | Allowed |
User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Claude-User Allow: /
The basics,
still first.
None of this works on a site Google can't index. We still run the full classic audit, ranked by the revenue pages each issue touches, and verify every fix.
Indexation and canonicals
Every page you want found is indexed, under one clear URL.
Core Web Vitals
LCP, INP and CLS per template, fixed where buyers land.
Site structure
Topic clusters, orphan pages and click depth to pricing.
| Classic check | At audit | Now |
|---|---|---|
| Canonical conflicts | 22 | 0 |
| Redirect chains | 64 | 12 |
| Orphan pages | 23 | 4 |
| Clicks from home to /pricing | 5 | 2 |
See every
agent visit.
Server logs show what page analytics miss: which crawlers and agents fetched which pages, in which format, and what happened next. We set up the reporting and read it with you every month.
Visits by purpose
Training crawls, search crawls and user-asked fetches, counted apart.
Formats and tool calls
Markdown served, llms.txt reads and MCP tool calls, per day.
AI referrals
People arriving from AI answers, tied to demos in GA4.
| Most fetched by agents | Format | Fetches |
|---|---|---|
| /pricing | markdown | 3.1K |
| /docs/integrations | markdown | 2.4K |
| /vs/kestrel | html | 1.2K |
From audit to
agent-ready.
Every engagement runs the same five steps. Each one leaves something you can open and check.
- 01
We test every reader
Googlebot, AI crawlers, browser agents and API agents on your key pages.
Audit31 findings · 5 blockers - 02
You get one roadmap
Fixes and builds ranked by what they unlock, each with an owner.
Roadmap31 items · 4 critical - 03
We build, or we brief
We ship it ourselves, or spec it for your developers and review the work.
Agent tools6 live via MCP + WebMCP - 04
We prove it works
Re-crawls, agent test runs and parity checks after every release.
Fix tracker18 of 31 verified - 05
We watch every visit
Agent visits, tool calls and AI referrals, from your logs and GA4.
Agent visits18.4K in 30 days
Most stop at Google.
We build for agents.
SEO agencies stop at Google. Dev shops build without a search plan. We do both, for every engine and agent.
| What you compare | SEO agency | Dev agency | |
|---|---|---|---|
| Who it tests for | Googlebot | People in a browser | Googlebot, AI crawlers, browser agents and API agents |
| Markdown for agents | Not offered | If asked, with no search plan | Same URL, same facts, parity checked every release |
| MCP, WebMCP and A2A | Not offered | Can build | Built for you, or specced for your team |
| Rendering | Flags the problem | Fixes it without search in mind | Planned around what crawlers and agents read |
| Tied to AI visibility | Partly | No | Linked to citations in every AI model |
Questions
What technical teams ask us first. Something else on your mind? Ask us on the strategy call.
What is technical SEO for AI agents?
It is the work that lets search engines, AI crawlers and AI agents reach, read and use your site: server rendering, Markdown for agents, structured data, bot policy and protocols like MCP and WebMCP. MaximusLabs audits all of it, builds the fixes or briefs your team, and measures every agent visit.
Do AI crawlers run JavaScript?
The major AI crawlers fetch your raw HTML and do not run JavaScript; a 2024 Vercel and MERJ study found none of them rendered it, while Google’s Gemini uses Googlebot, which does. MaximusLabs server-renders the pages that matter so every crawler sees the same content.
Is serving Markdown to AI agents cloaking?
Not when both formats carry the same content. Cloaking means showing bots different content to manipulate results. MaximusLabs serves the same facts as Markdown on the same URL, with Vary: Accept and a canonical, and checks parity on every release.
What is llms.txt, and does it help?
llms.txt is a proposed file, from September 2024, that maps your key pages for AI tools. No major AI engine has confirmed using it, and Google says Search ignores it. MaximusLabs ships it because it is cheap and helps agents, then measures whether anyone fetches it.
What are MCP, WebMCP and A2A, and do we need them?
MCP lets AI assistants call your tools and data, WebMCP (a proposed standard now in a Chrome origin trial) lets your pages expose actions to browser agents, and A2A lets agents discover and work with each other. MaximusLabs recommends them only where an agent has a real job to do on your site.
Should we block AI training crawlers?
That is a business choice. Blocking training crawlers such as GPTBot and ClaudeBot does not block the search and answer crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. MaximusLabs sets a policy per purpose, so you can opt out of training and still be cited.
How is this different from MaximusLabs’ B2B SEO service?
Technical SEO makes the site readable and usable by search engines, AI crawlers and agents. B2B SEO decides what to rank for and writes the pages that win buyers. Most MaximusLabs clients fix the foundation first.
Who builds the fixes: MaximusLabs or our developers?
Both, depending on the format you choose. MaximusLabs either builds and ships the fixes itself, or specs each one for your developers and reviews the work, then re-checks every fix either way.
What usually comes next
Ready for the agent web.
Book a call. We'll test your site the way Google, AI crawlers and agents see it, and show you what to fix first.