Logo

Blog

Insights & Updates

Discover the latest thoughts, insights, and updates from our team on AI, technology, and building the future.

Featured

Illustration of a honey jar attracting AI icons like chatbot, search, eye, and speech bubble, with the text ‘Honeypot for AI SEO’ above.
Wednesday, September 3, 2025
Setting Up Honeypots to Monitor AI Agent Crawling Behavior for SEO
We set up a honeypot to observe how AI agents like ChatGPT and Perplexity access web content. By monitoring their real-time queries, we uncovered key insights—such as how agents break down user questions and respond to tailored API endpoints—providing valuable benefits for content optimization and understanding agentic search behavior.
Sherin Thomas (boatbuilder)
Illustration of an AI robot examining a website through a magnifying glass, observing only a reflected version to symbolize cached content access
Thursday, August 21, 2025
How LLMs Like ChatGPT Really Crawl the Web And Cache Content: An AI Agent Behavior Experiment
AI search engines like ChatGPT and Perplexity often cite websites without live visits, thanks to clever caching and selective crawling. By observing these bots and AI Agents using a fake site, it’s clear they sometimes masquerade as regular browsers and rely heavily on previously stored content. This experiment uncovers how AI agents access and use online information, challenging common assumptions about analytics and web citations.
Sherin Thomas (boatbuilder)
RSS Feed
Wednesday, September 3, 2025
Sherin Thomas (boatbuilder)

Setting Up Honeypots to Monitor AI Agent Crawling Behavior for SEO

We set up a honeypot to observe how AI agents like ChatGPT and Perplexity access web content. By monitoring their real-time queries, we uncovered key insights—such as how agents break down user questions and respond to tailored API endpoints—providing valuable benefits for content optimization and understanding agentic search behavior.

Read more
Thursday, August 21, 2025
Sherin Thomas (boatbuilder)

How LLMs Like ChatGPT Really Crawl the Web And Cache Content: An AI Agent Behavior Experiment

AI search engines like ChatGPT and Perplexity often cite websites without live visits, thanks to clever caching and selective crawling. By observing these bots and AI Agents using a fake site, it’s clear they sometimes masquerade as regular browsers and rely heavily on previously stored content. This experiment uncovers how AI agents access and use online information, challenging common assumptions about analytics and web citations.

Read more