New Robots.txt AI Training Opt-Out: What LI Businesses Should Do

If you own a business in Nassau or Suffolk, there’s a good chance you have never opened your website’s robots.txt file, and until now that was perfectly fine. A new announcement on the Google Search Central blog makes that small file a little more important. In a post titled Supporting a shared AI training control for websites, Google said it plans to support a shared AI training robots.txt signal that lets a website tell multiple AI providers, in one step, whether its content can be used to train AI models.

This post walks through what was announced, what this new control does and does not affect, and why our advice for most local businesses is simple: leave it unset or allow training. If you care about being recommended when customers ask ChatGPT, Gemini or Google’s AI features for a local business (the work we focus on in our GEO, AEO and AI SEO services), these details matter more than they first appear.

What Google Search Central Actually Announced

The announcement is shorter than the headlines around it, so here is what Google’s post says, without embellishment.

• Google plans to support a shared AI training control in robots.txt.
• The goal is to let websites express their AI training preferences across multiple AI providers in a single step.
• The control is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF). The IETF is the long-standing standards body that develops many of the technical rules the internet runs on.
• It builds on Google-Extended, the AI training control Google has offered since 2023.
• Existing company-specific controls, like Google-Extended, will keep working alongside the new shared signal.
• Google describes this as a draft proposal shaped by feedback from publishers, civil society groups, regulators and other stakeholders who asked for industry-wide controls on AI training.
• Google says the proposal is consistent with the EU Code of Practice for General Purpose AI Model Providers under the EU AI Act, which recognizes robots.txt as a primary way for publishers to express machine-readable preferences.
• Google aims to support the new control along with other AI providers in 2027 and says it will share updates on its progress.

Two things stand out for a business owner. First, this is a plan, not something switched on today. Second, Google’s post refers to "participating AI providers" and "other AI providers" without listing which ones have committed. You may see this story described as several big AI companies agreeing on one signal. That may turn out to be the direction things go, but if a specific provider matters to you, look for that company’s own confirmation rather than assuming.

How the Shared AI Training Robots.txt Signal Would Work

A quick refresher on the file itself. Robots.txt is a plain text file that sits at the root of your website (yourbusiness.com/robots.txt). Automated programs that visit websites, often called crawlers or bots, check it to see what the site owner allows. It has been around for decades and is mostly used to keep search engines out of areas like admin pages or duplicate content. If you want the full plain-English version, our guide to XML sitemaps and robots.txt covers the basics, and Google maintains its own introduction to robots.txt.

Today, if you want to keep your content out of AI training, you generally have to write separate instructions for each AI company’s crawler. Google-Extended is Google’s version of that: a name you can address in robots.txt to tell Google not to use your content for training its AI models.

The new approach adds a shared "policy token." Think of it as a single, industry-wide label you can put in robots.txt that means "here is my preference about AI training," which any participating provider can read. Google hasn’t published final syntax for it in this announcement, and the IETF discussions are ongoing, so don’t go hunting for a line to paste in yet.

Here is how Google says it will interpret the signal:

1. If your robots.txt uses the shared policy token to opt out of training, and you have not written a specific Google-Extended rule, Google plans to treat that as if you had disallowed Google-Extended.
2. If you have written a specific Google-Extended rule, your company-specific instruction still matters, because Google only applies that interpretation when Google-Extended is not explicitly specified.
3. Other AI providers can handle the shared signal in a similar way, which is what makes it possible to opt out of everyone at once or pick and choose provider by provider.

In short: one broad switch for everyone, with the option to override it for individual companies.

Why Most Long Island Businesses Should Not Opt Out

For a large publisher, a news outlet or a company whose content is the product, opting out of AI training can be a reasonable business decision. For a plumber in Hicksville, a dental practice in Huntington, a law firm in Garden City or a boutique in Port Jefferson, the math usually runs the other way.

Your website exists to get you found and to convince people to call. Your service pages, your about page and your FAQs are marketing. You want as many systems as possible to understand who you are, what you do and where you do it. When AI tools learn about your business accurately, that understanding can show up later when someone asks for a recommendation.

There is also a more concrete reason in Google’s own announcement. The post states that websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. "Grounding" is when an AI assistant looks up live information from the web to support its answer, often with a link back to the source. Combine that with Google’s statement that a shared opt-out (with no specific Google-Extended rule) will be treated as disallowing Google-Extended, and the takeaway for a local business is clear: a broad opt-out could affect whether your site can be surfaced and clicked in Gemini Apps responses.

That is a real visibility cost, and for most small businesses it outweighs the benefit of keeping a few service pages out of training data. Your hours, your services and your service area are not trade secrets. They are exactly what you want AI assistants to repeat correctly.

The practical recommendation:

• Leave it unset if you haven’t thought about AI training and don’t have a specific reason to block it. Doing nothing is a valid choice here.
• Allow training explicitly if you want to be clear about your preference, especially with Google-Extended, given what Google says about Gemini Apps eligibility.
• Block selectively only if you have content that genuinely shouldn’t be reused, and even then, consider whether that content belongs on a public website at all.

Training, Search and AI Answers Are Three Different Things

The most common confusion we expect from this news is owners assuming that an AI training opt-out will hide them from Google, or that allowing training is the same as showing up in AI answers. These are separate questions, controlled in different ways.

Google Search rankings

Google has stated in its crawler documentation that Google-Extended does not affect whether your site is included in Google Search and is not used as a ranking signal. Your regular Google Search visibility is governed by how Googlebot crawls and indexes your site. An AI training opt-out is not a tool for managing your search rankings, in either direction.

AI answers that cite or recommend you

Whether an AI assistant mentions your business when someone asks "who’s the best HVAC company near Massapequa?" depends on many factors: whether the tool can access your site, how clearly your site describes your business, your reviews, your Google Business Profile, and what other sites say about you. Training preferences are one piece of the puzzle at most. As Google’s announcement notes, for Gemini Apps specifically, allowing Google-Extended is tied to eligibility for grounding responses and clicks.

Model training

Training is when an AI company uses large amounts of content to build or improve the underlying model. This is what the new shared signal is about. It is a long-term, background process, and it is separate from the live lookups many assistants do when answering a question.

Keeping these three apart will save you from making a change that sounds protective but quietly works against your marketing.

When Opting Out Might Make Sense for Your Business

We don’t think every business should allow everything. There are cases where an opt-out deserves a real conversation:

• Your content is what you sell. If you publish paid courses, original research, detailed guides or premium material that customers pay for, you may not want that absorbed into AI models for free.
• You have creative work on public pages. Photographers, designers and artists sometimes prefer to limit how their work is used, even knowing robots.txt is a preference signal rather than a lock.
• You have a strong principled position. Some owners simply don’t want their work used for AI training. That is a legitimate choice, as long as you make it knowing the visibility trade-off.

If you fall into one of these groups, the new control gives you more flexibility than before. You could opt out broadly using the shared signal, then explicitly allow specific providers whose AI answers matter to your customers. That pick-and-choose approach is exactly what Google says the combination of the shared token and company-specific controls is meant to allow.

One honest caveat: robots.txt is a set of instructions that well-behaved crawlers choose to follow. It is not a security measure. Anything truly sensitive should never be on a public page in the first place.

What to Check on Your Website Before 2027

Because Google is aiming for 2027, nothing about your site needs to change today. That said, now is a good time to make sure you aren’t already blocking AI tools by accident. Owners are sometimes surprised by what is in their robots.txt, usually because a plugin, a previous developer or a hosting setting made a change nobody remembers.

1. Look at your robots.txt file. Type your domain followed by /robots.txt into a browser. It’s public, so you can see exactly what crawlers see.
2. Look for AI-related rules. Lines naming Google-Extended or other AI crawlers with a "Disallow" next to them mean someone has already chosen to block those bots. Make sure that was a deliberate decision.
3. Make sure you aren’t blocking everything. A line telling all bots to stay away from your entire site is sometimes left over from when a site was being built. That would hurt you far more than any AI setting.
4. Check your platform and security settings. Some website builders, hosting dashboards, security plugins and network services now include options to block AI bots. Find out whether any of those toggles are switched on and who switched them.
5. Write down your decision. Note what you chose and why, so the next person who works on your site doesn’t undo it without realizing.
6. Revisit when the standard is final. Google says it will provide updates. When the shared token’s final format is published, decide whether you want to add it at all.

If you’ve been reading about other files aimed at AI tools, our post on llms.txt and AI crawlers explains how that idea differs from robots.txt and whether it’s worth your time.

How This Fits Your Larger AI Visibility Plan

A robots.txt setting can only stop you from being seen. It can’t make you recommended. That part comes from the same fundamentals that have always mattered, now read by more kinds of machines:

• Clear service pages that say what you do, where you do it and who you help, in plain language.
• A complete, accurate Google Business Profile with current hours, services and photos.
• Consistent business information (name, address, phone) across directories.
• Genuine reviews that describe the work you did.
• Helpful answers to the questions customers actually ask, written so both people and AI tools can understand them.

The businesses that do well in AI answers tend to be the ones that make it easy for these systems to understand them. Blocking training doesn’t help with that, and allowing training doesn’t replace it.

This is also where AI should work for your team, not around it. Nobody on your staff needs to become a robots.txt expert. They need to know your customers’ questions well enough to answer them on your website, and someone needs to check now and then that AI tools describe your business correctly. If you want to see where you stand right now, our guide on how to audit your brand’s visibility across AI chatbots walks through a simple process you can run yourself.

Your Next Step: A Five-Minute Robots.txt Check

Here is the whole thing, condensed. Google plans to support a shared AI training signal in robots.txt, aiming for 2027 alongside other AI providers. It won’t change your Google Search rankings. A broad opt-out could, per Google’s own post, affect Gemini Apps eligibility unless you explicitly allow Google-Extended. For most Long Island businesses, the right move is to leave the setting unset or allow training, and to make sure nothing on your site is already blocking AI tools by accident.

So take five minutes this week: open yourdomain.com/robots.txt, look for any lines that block Google-Extended, AI crawlers or your whole site, and confirm with whoever manages your website that every rule there is intentional. Then put a reminder on your calendar to revisit it when Google publishes the final details.

If you’d like a second set of eyes on your robots.txt or your overall AI visibility, reach out to MJW Media and we’ll give you a straight answer about what, if anything, needs to change.

Frequently Asked Questions

Related Resources

• GEO, AEO & AI SEO Services
• AI Visibility Monitoring
• XML Sitemaps & Robots.txt: A Plain-English Guide
• llms.txt and AI Crawlers: Should Your Website Have One in 2026?
• How to Audit Your Brand’s Visibility Across AI Chatbots