SEO Weekly

AI crawler access

How search and training crawlers reach websites, and services affecting their access. Every SEO Weekly story on AI crawler access, newest first. 3 stories since September 2026.

  • One robots.txt setting will soon block all AI training

    A shared token called _usage_ai-training would replace blocking each AI company's bot one by one.

    · Getting into AI answers

  • Cloudflare now blocks Google by default

    A September 15 change means some sites behind Cloudflare may be shutting Google out without the owner knowing.

    · Do not do this

  • Be the person who already knew

    One email a day. Under three minutes. You will hear about a Google change before it shows up in your traffic.

  • Cloudflare change shifted which AI bots reach your site

    A study of 1,046 sites found search crawlers got in far more often after the change, and training crawlers got blocked slightly more.

    · Why your traffic moved

People on AI crawler access

Get the next one by email

A daily read of what the sharpest SEOs on X actually said. What changed in search, whether it touches your site, what to do. Every quote links out.