Quick answer: On November 11, 2026, United updated its website terms to keep search indexing open while requiring written permission before any AI agent searches, books or buys on united.com or its app. Agencies should tell clients that search crawlers, AI training crawlers and AI agents are different, and that most small businesses want AI tools to find them.
Key takeaways
- United’s updated terms allow search engines to index its public pages but require prior written permission before any automated or AI agent searches, books or buys on its website or app.
- The terms also bar anyone from using United’s data to train, fine-tune, ground or evaluate AI models, and United updated its contract of carriage the same day.
- Search Engine Journal notes that none of this relies on robots.txt. As of November 14, United’s robots.txt named no AI crawlers or agents.
- Search crawlers, AI training crawlers and AI agents acting for users are three different things, with different controls and different business effects.
- United has reasons that most small business clients don’t share. A local business usually wants AI search tools and assistants to find it, recommend it and send it bookings, so blanket blocking can hurt.
- Agencies should review each client’s robots.txt, firewall rules and terms together, then agree a written AI access policy with the client.
What did United change about AI agent access?
United updated its website terms on November 11, 2026. Search engines can still index its public pages, but any AI agent that searches, books or buys on united.com or the United app now needs prior written permission. The terms also ban using United’s fares, schedules and other data to train or ground AI models.
Matt G. Southern reported the details for Search Engine Journal on November 14, 2026. According to his reading of the terms and United’s own announcement:
- Search indexing stays open. Search engines indexing public pages are exempt, along with assistive technology and standard browsers used by a person.
- Agents need permission. The terms define an automated agent broadly, covering AI assistants, bots, scrapers, browser extensions and built-in browser or app features, even when a person is supervising. Built-in agent features in browsers are not exempt.
- No AI training or grounding. Fares, schedules, availability, pricing and other site or app data can’t be used to train, fine-tune, ground, evaluate or otherwise develop an AI model.
- The contract of carriage changed too. SEJ reports that it now lists using AI agents to make, change or service a ticket as a prohibited practice, and that United can cancel bookings made, changed or serviced by AI agents without notice. Agent bookings made before November 11 will not be canceled.
United’s announcement, titled “United Restricts Access for Third-Party AI Agents,” gives three reasons: account security when customers share logins with AI assistants, the risk of unintended purchases, account changes or use of MileagePlus miles, and possible conflicts with FAA and DOT rules. Its example is check-in, which needs a person to confirm that bags hold no banned items. SEJ also cites a Wall Street Journal report in which CEO Scott Kirby said, “I don’t want to create another middleman.”
United is not alone. SEJ points out that Amazon blocked Meta’s Muse agent in September under its Conditions of Use, and that eBay added “buy-for-me” agents to the automated tools in its User Agreement that need permission. The common thread is large platforms protecting their booking and checkout flows.
What is the difference between search crawlers, AI training crawlers and AI agents?
A search crawler fetches pages to build a search index, so a business can appear in results. An AI training crawler collects content that may be used to train AI models. An AI agent acts for a specific user in real time, such as visiting a page, comparing options or completing a booking. Each has different controls and business effects.

Clients tend to lump all of these together as “AI bots,” which is how blanket blocking happens. The table sets out what each type does and how it is usually controlled, using the providers’ own documentation.
| Type | Examples | What it does | How it is controlled | What blocking costs |
|---|---|---|---|---|
| Search crawler | Googlebot | Builds Google’s search index, including Search features | robots.txt rules for Googlebot | Visibility in Google Search |
| AI search crawler | OAI-SearchBot | Surfaces sites in ChatGPT’s search features | robots.txt rules for OAI-SearchBot | Appearing in ChatGPT search answers |
| AI training control or crawler | Google-Extended, GPTBot | Signals whether content may be used to train AI models | robots.txt product tokens | Little or nothing in Google Search, per Google; check each provider’s other effects |
| User-initiated agent | ChatGPT-User, browser agents | Visits pages or acts for a specific person in real time | robots.txt may not apply; terms, logins and bot protection | Customers who use assistants to research or book |
| Search AI features | AI Overviews, AI Mode | Uses Google’s index to generate answers | Search Console’s Search generative AI setting | Traffic and impressions from those features |
A few details from the official documentation are worth knowing before you talk to a client:
- Google-Extended is a control, not a crawler. Google says it has no separate user agent string and “does not impact a site’s inclusion in Google Search,” nor is it a ranking signal. It governs whether crawled content can be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
- OpenAI splits search and training. Its crawler overview says sites that opt out of OAI-SearchBot won’t be shown in ChatGPT search answers, while disallowing GPTBot signals that content should not be used to train foundation models.
- User-triggered visits play by different rules. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person starts those actions.
- AI Overviews have their own switch. Search Console’s Search generative AI control can keep a site out of AI Overviews and AI Mode, and Google says it isn’t used as a ranking or inclusion signal for the rest of Search.
Why doesn’t United’s approach rely on robots.txt?
Robots.txt only works for bots that identify themselves and choose to follow it. Many personal AI agents run inside a browser or app and publish no user agent to target. So United put its rules in its website terms and contract of carriage, which bind the user and the agent’s operator, rather than in a crawl file.
SEJ checked United’s robots.txt on November 14 and found no AI crawlers or agents named in it. The article also cites earlier reporting that personal agents like Meta’s Muse publish no user agent for robots.txt rules to target. That is the practical lesson for agencies: robots.txt, terms of service and bot protection are three separate layers, and they can easily say different things.
For a client, that means a robots.txt review alone doesn’t answer the question “what can AI do on our site?” A firewall or CDN rule may already be challenging agents the client wants, or the terms page may say nothing at all. Our post on how bot protection can block Googlebot after a CDN move shows how quickly broad bot rules catch crawlers nobody meant to block.
Should small business clients block AI agents like United did?
Usually not. United blocks agents to protect its own booking flow, customer accounts and regulated processes. Most small businesses have the opposite problem: they want to be found, recommended and booked. Blocking AI search crawlers or helpful agents can remove them from answers their customers already rely on, with nothing gained in return.
In our view, the right default for a typical local or service business is to keep search crawlers and AI search crawlers allowed, make the decision on AI training separately, and only restrict agents where there is a clear reason. Some clients do have one:
- Accounts and payments. If customers log in, hold balances or store payment details, agents acting in those accounts carry the same risks United described.
- Regulated steps. Where a process needs a person to confirm something, as with United’s check-in example, an agent completing it may be a problem.
- Pricing and inventory as the product. Businesses whose data is the asset, such as listings or rate databases, may reasonably restrict scraping and training.
Everyone else should think hard before copying a large company’s rules. Visibility in AI answers is increasingly part of what clients pay agencies for, as we covered in why AI search visibility is becoming a core white-label SEO deliverable.
How should agencies review a client’s AI access settings?
Review AI access as one policy across four layers: robots.txt, firewall and CDN bot rules, the website terms, and the Search Console generative AI setting. Check what each layer currently allows, confirm it matches what the client wants, and write the agreed policy down so developers, hosts and your team all work from the same rules.

- Read the robots.txt file. List every rule that names Googlebot, Google-Extended, OAI-SearchBot, GPTBot or other AI tokens, plus any blanket disallow rules. Flag anything that blocks search or AI search crawlers by accident.
- Check firewall and CDN bot settings. Ask the developer or host whether “block AI bots” or challenge-all options are switched on. These often sit outside the SEO team’s view.
- Check Search Console. Confirm the Search generative AI setting matches the client’s wishes, so the site isn’t kept out of AI Overviews and AI Mode by mistake.
- Read the terms page. Note whether it says anything about scraping, automated access or AI. If the client wants to restrict agents or training, that is a question for their lawyer, not for the agency.
- Test the booking path. Make sure the key pages, prices, services and contact or booking details are visible in plain HTML, so assistants that do visit can read them.
- Write the policy down. One short page: what is allowed, what is restricted, who owns each layer and when it will be reviewed.
Most of these checks belong in a regular technical review. Our post on why technical SEO still matters for AI search explains why crawl access sits underneath visibility in both classic and AI results.
What should agencies tell clients about AI agents and crawlers?
Tell clients that United’s change reflects United’s business, not a new rule for everyone. Explain the three bot types in plain language, recommend keeping search and AI search crawlers allowed, treat AI training as a separate choice, and agree a written policy. Make clear that any changes to terms or contracts need their own legal advice.

A simple conversation script for a client call:
- Open with the news. “You may have seen that United now requires permission for AI agents. That’s about protecting bookings and customer accounts on a major airline’s site. It isn’t something Google or OpenAI requires of you.”
- Explain the difference. “There are three kinds of AI access: crawlers that put you in search results, crawlers that collect content for training, and assistants acting for a customer. Each one is a separate decision.”
- Give our default. “For a business like yours, we recommend keeping Google and AI search crawlers allowed so you can appear in Google, AI Overviews and ChatGPT search.”
- Ask about training. “Do you mind your content being used to train AI models? Opting out through Google-Extended doesn’t affect your Google Search rankings, according to Google. It’s your call.”
- Ask about agents. “Do customers log in, pay or manage accounts on your site? If not, there’s usually little reason to block assistants that help people find and book you.”
- Agree next steps. “We’ll check your robots.txt, firewall and Search Console settings, then send you a one-page policy to approve. Anything about your terms or contracts should go to your lawyer.”
Questions to have answered before the call ends:
- Do you want to appear in AI answers such as AI Overviews and ChatGPT search?
- Are you comfortable with your content being used for AI training?
- Do customers log in, pay or hold accounts on your site?
- Who controls your firewall, CDN and terms page?
Keep the framing calm. Clients hear “AI bots” and assume threat, so it helps to lead with what they gain from being visible. Our guide on what ChatGPT’s business analytics plans mean for clients is a useful follow-up for clients who want to know how AI visibility will be measured.
What about the shared AI training opt-out?
Separately from United’s news, Google said on October 30, 2026 that it will support a shared AI training control in robots.txt, based on an IETF AIPREF draft. It lets a site state one training preference for participating AI providers. Google’s crawler documentation says it intends to support the token __usage_ai-training in early 2027.
For agencies, this changes the mechanics of the training decision, not the advice. Google says that if a site uses the shared token to opt out and doesn’t mention Google-Extended, it will treat Google-Extended as disallowed, and company-specific controls keep working alongside it. That makes training the easiest of the three decisions to separate from search visibility, which is exactly the separation clients need to understand.
Frequently asked questions
Does United’s new policy block Google from indexing its site?
No. United’s updated terms keep search indexing open, and search engines indexing public pages are exempt. What needs written permission is any AI agent that searches, books or buys on united.com or the United app, plus any use of United’s data to train or ground AI models.
Will blocking GPTBot or Google-Extended hurt a client’s rankings?
Google says Google-Extended does not affect a site’s inclusion in Google Search and isn’t a ranking signal. OpenAI says GPTBot relates to training foundation models, while OAI-SearchBot controls appearance in ChatGPT search answers. Blocking OAI-SearchBot is the setting that removes a site from those answers.
Can robots.txt stop AI agents that act for users?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person starts those actions, and many personal agents publish no user agent to target. That is why United relied on its terms and contract of carriage instead. Bot protection and logins are the practical controls.
How can a client stay out of AI Overviews without leaving Google Search?
Google offers a Search generative AI setting in Search Console that keeps a site’s links and content out of AI Overviews, AI Mode and generative AI features in Discover. Google says it isn’t used as a ranking or inclusion signal for the rest of Search. Most businesses lose visibility by using it.
Should an agency write AI terms of service for a client?
No. Agencies can explain the technical options and document the client’s chosen policy, but website terms and contracts are legal documents. If a client wants terms like United’s, refer them to their own lawyer and then align robots.txt and bot settings with whatever they decide.
What should you do next?
Use United’s news as a reason to check every client’s AI access, not as a template to copy. Review robots.txt, firewall rules, the Search Console generative AI setting and the terms page, then agree a short written policy that keeps search and AI search crawlers allowed unless the client has a clear reason not to. If you want a partner to run these checks and explain them to clients under your agency’s brand, see our white-label SEO services at White Label SEO.
Want AI access reviews done for your clients?
At White Label SEO, we audit robots.txt, bot protection and AI search settings for your clients and deliver plain-English reports under your agency’s brand. Tell us how many sites you manage, and we’ll show you how we’d review them.
Sources
- Search Engine Journal (Matt G. Southern), “United Requires Permission For AI Agents, Keeps Search Indexing Open”, November 14, 2026. https://www.searchenginejournal.com/united-ai-agents-permission-search-indexing/597418/
- United Airlines, “United Restricts Access for Third-Party AI Agents”, November 11, 2026. https://united.mediaroom.com/news-releases?item=125496
- Google Search Central Blog (John Mueller), “Supporting a shared AI training control for websites”, October 30, 2026. https://developers.google.com/search/blog/2026/10/ai-training-control
- Google Crawling Infrastructure, “List of Google’s common crawlers”, accessed November 15, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Search Console Help, “Search generative AI control”, accessed November 15, 2026. https://support.google.com/webmasters/answer/16908024?hl=en
- OpenAI, “Overview of OpenAI Crawlers”, accessed November 15, 2026. https://developers.openai.com/api/docs/bots

