Blog
Insights & Updates
Discover the latest thoughts, insights, and updates from our team on AI, technology, and building the future.
Setting Up Honeypots to Monitor AI Agent Crawling Behavior for SEO
We set up a honeypot to observe how AI agents like ChatGPT and Perplexity access web content. By monitoring their real-time queries, we uncovered key insights—such as how agents break down user questions and respond to tailored API endpoints—providing valuable benefits for content optimization and understanding agentic search behavior.
How LLMs Like ChatGPT Really Crawl the Web And Cache Content: An AI Agent Behavior Experiment
AI search engines like ChatGPT and Perplexity often cite websites without live visits, thanks to clever caching and selective crawling. By observing these bots and AI Agents using a fake site, it’s clear they sometimes masquerade as regular browsers and rely heavily on previously stored content. This experiment uncovers how AI agents access and use online information, challenging common assumptions about analytics and web citations.