Should I be blocking AI crawlers?

That is a genuine business decision and it depends on whether your content is the product. A publisher selling subscriptions has a real reason to block. A dental practice that wants to be recommended when somebody asks an assistant for a dentist nearby almost certainly does not. What matters is that you chose, rather than inherited it from a plugin.

Does blocking AI crawlers hurt my Google ranking?

Blocking Google-Extended does not affect ordinary Google Search ranking. It controls whether your content is used for Gemini and AI Overview grounding. They are separate controls and blocking one does not block the other.

My robots.txt looks fine but I am still not cited anywhere. Why?

Being readable is necessary, not sufficient. Reaching and reading you is the mechanical floor. Whether an assistant quotes you depends on whether your pages actually answer the question somebody asked, and on systems nobody outside them can inspect.

Free tool

Can AI assistants read your website?

We read your robots.txt the way a crawler does and tell you which of ten AI assistant crawlers are blocked from your site, then count how much of your content exists before any JavaScript runs. Both decide whether an assistant can quote you at all.

We read the page, its headers, its certificate and its public DNS records. We do not log in, we do not probe admin paths, and we identify ourselves in your server logs as OxwynXRay.

What this check cannot tell you

We read robots.txt. We cannot see rules applied at your CDN, firewall or WAF, and that is where accidental blocking most often lives, because a security product added it without telling anyone. So a clean result here means not blocked in robots.txt, which is narrower than reachable. It also cannot tell you whether any assistant will cite you: that is a decision made inside systems nobody outside them can inspect, and a tool claiming otherwise is guessing.

Blocking is often accidental

Plenty of publishers block AI crawlers deliberately to protect their content, and that is a perfectly reasonable choice. The problem is that most small business sites that block them never decided to. The rule arrived with a plugin default, a host template or a security setting, and nobody has read the file since.

Training and search are two different crawlers

GPTBot collects content for training. OAI-SearchBot powers what ChatGPT shows when it searches the live web. Blocking one has no effect on the other, and they need separate rules. The same split exists for Anthropic and for Google, where Google-Extended controls Gemini grounding without touching your ordinary search ranking.

The invisible half: content that only exists after JavaScript

Search engines run JavaScript. Several AI crawlers do not. A site whose content only appears after hydration is, to those crawlers, an empty page. This is the most common reason a modern, well-built site is invisible to assistants while ranking perfectly well on Google, and it never shows up in any ordinary SEO tool.

Why this is not the same as SEO

Nobody has published a ranking algorithm for AI answers, and anybody selling you optimisation for one is guessing. What can be checked is mechanical and not in dispute: can the crawler reach the page, and can it read the content. Everything past that is opinion, and we are not going to sell you ours as fact.

Allowed in is only the first of three

What is actually documented about AI search, and what is confident invention

Questions

Should I be blocking AI crawlers?: That is a genuine business decision and it depends on whether your content is the product. A publisher selling subscriptions has a real reason to block. A dental practice that wants to be recommended when somebody asks an assistant for a dentist nearby almost certainly does not. What matters is that you chose, rather than inherited it from a plugin.
Does blocking AI crawlers hurt my Google ranking?: Blocking Google-Extended does not affect ordinary Google Search ranking. It controls whether your content is used for Gemini and AI Overview grounding. They are separate controls and blocking one does not block the other.
My robots.txt looks fine but I am still not cited anywhere. Why?: Being readable is necessary, not sufficient. Reaching and reading you is the mechanical floor. Whether an assistant quotes you depends on whether your pages actually answer the question somebody asked, and on systems nobody outside them can inspect.

This tool is one part of the full X-Ray, which measures security headers, your certificate, speed, indexability and what your site publishes about itself. Same scan, same free report.