Blog
Insights & Updates
Discover the latest thoughts, insights, and updates from our team on AI, technology, and building the future.
Thursday, August 21, 2025
Sherin Thomas (boatbuilder)
How LLMs Like ChatGPT Really Crawl the Web And Cache Content: An AI Agent Behavior Experiment
AI search engines like ChatGPT and Perplexity often cite websites without live visits, thanks to clever caching and selective crawling. By observing these bots and AI Agents using a fake site, it’s clear they sometimes masquerade as regular browsers and rely heavily on previously stored content. This experiment uncovers how AI agents access and use online information, challenging common assumptions about analytics and web citations.