Optimizing for Agentic AI

Hi new friends! Old friends already know that I love new technology. Almost as much as I hate a grift. These past 75 years trapped in an AI death spiral have been a touch… rough.

Big new things lead to FOMO. FOMO always leads to influencers and services. Good on those industrious folks working hard to make sense of the madness! Others though perhaps so altruistic in their efforts. Influence can be lucrative. Two years of unfounded claims, neverending SaaS, and really nothing sustainable to show for it … Well, kids. It does have me a touch jaded.

Until recently. It's almost like Google missed me, too! To celebrate our return, Capital G launched off delightful, grounded documentation about the next next big thing: Agentic AI. It does things! For you! Crowds go wild! We'll all be rich and famous!

Instead of hype and call for price, we get a breakdown of the three ways agents interpret a website:

1. Screenshots let agents use vision models to identify elements visually.
2. Raw HTML gives agents the DOM structure and hierarchy.
3. Accessibility tree provides what Google calls a “high-fidelity map” of interactive elements, stripped of visual noise.

Who knew that Skynet was really powered by Cmd + Shift + 5? Turns out a least a couple of folks. Arun Baby penned an in-depth guide on the mechanics. AI screenshots operate roughly like this:

1. Capture and Processing: Agents take a screen snapshot (often PNG) of a webpage or app.
2. Vision Understanding (OCR/GUI Detection): Multimodal AI (like Claude or GPT-4o) processes the image, using Optical Character Recognition (OCR) to read micro-text and detecting GUI elements to understand the layout and interactivity.
3. Contextualization: The AI identifies the intent behind the screenshot—such as identifying a bug in a UI, converting a design screenshot into code, or interpreting a visual task.
4. Actionable Feedback: The agent maps visual information to commands (e.g., "click [X, Y] coordinate"), allowing it to interact with web pages or suggest UI improvements.
5. Web/Application Automation: Agents often integrate with frameworks to automate actions, such as capturing a screenshot, finding a button, and clicking it.

Myriam's Jessier's work on Image SEO for Multimodal AI is sure as hell exceptionally relevant now. You want those fonts readable. You need that alt text for grounding. Stylized fonts? Just another new way that ill thought out redesign may just bite you in the ass.

Raw HTML is nothing new. JavaScript-frameworks have been ruing the day LLMs became the tech revolution with a 10 year old stack (viva la app store). If you're not familiar with the accessibility tree, it provides “high-fidelity map” of interactive elements, stripped of visual noise. You can preview the accessibility tree in Chrome DevTools.

It's refreshing to have a source now when I respond to my 50th solicited DM asking if I'm ready to "turbocharge my future by converting my page to markdown." This article has a list of ways to help agents navigate your site, and that isn't one.