Skip to content

[Bots] Managed robots.txt #22026

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: production
Choose a base branch
from
Open

[Bots] Managed robots.txt #22026

wants to merge 3 commits into from

Conversation

patriciasantaana
Copy link
Contributor

Summary

Add documentation on Managed robots.txt

Documentation checklist

@github-actions github-actions bot added product:bots Related to Bots product size/s labels Apr 28, 2025
@hyperlint-ai Hyperlint AI
Copy link
Contributor

hyperlint-ai bot commented Apr 28, 2025

Howdy and thanks for contributing to our repo. The Cloudflare team reviews new, external PRs within two (2) weeks. If it's been two weeks or longer without any movement, please tag the PR Assignees in a comment.

We review internal PRs within 1 week. If it's something urgent or has been sitting without a comment, start a thread in the Developer Docs space internally.

PR Change Summary

Added documentation for managing robots.txt to guide users in directing AI crawlers on their websites.

  • Introduced a new documentation page for Managed robots.txt
  • Provided instructions for implementing robots.txt based on different plans
  • Updated the sidebar order for static resource protection documentation

Modified Files

  • src/content/docs/bots/additional-configurations/static-resources.mdx

Added Files

  • src/content/docs/bots/additional-configurations/managed-robots-txt.mdx
  • src/content/partials/bots/enable-managed-robots-txt.mdx
How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 28, 2025 •
edited
Loading

Preview URL: https://7265e20f.preview.developers.cloudflare.com
Preview Branch URL: https://patricia-pcx14359-robots-txt.preview.developers.cloudflare.com

Files with changes (up to 15)

Original Link Updated Link
https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ https://patricia-pcx14359-robots-txt.preview.developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
https://developers.cloudflare.com/security/settings/ https://patricia-pcx14359-robots-txt.preview.developers.cloudflare.com/security/settings/
https://developers.cloudflare.com/bots/additional-configurations/static-resources/ https://patricia-pcx14359-robots-txt.preview.developers.cloudflare.com/bots/additional-configurations/static-resources/

@patriciasantaana patriciasantaana marked this pull request as ready for review April 28, 2025 18:22
@patriciasantaana patriciasantaana requested a review from a team as a code owner April 28, 2025 18:22
marciocloudflare
marciocloudflare approved these changes Apr 29, 2025
View reviewed changes
src/content/docs/bots/additional-configurations/managed-robots-txt.mdx
</Tabs>

:::note
When you select the toggle to implement a `robots.txt` file on your domain, existing `robots.txt` files will be replaced.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When you select the toggle to implement a `robots.txt` file on your domain, existing `robots.txt` files will be replaced.
When you enable the `robots.txt` file on your domain, existing `robots.txt` files will be replaced.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this is what you meant, but if so I think it's easier to understand - and also, what we have on our style guide

src/content/docs/security/settings.mdx
Comment on lines +103 to +104
| [Block definitely automated traffic](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
| [Block likely bots](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no #bot-settings anchor link on these pages

src/content/docs/security/settings.mdx
Comment on lines +105 to +106
| [Managed `robots.txt`](/bots/additional-configurations/managed-robots-txt/) | **Security** > **Bots** > **Configure Bot Fight Mode<br/>Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
| [Allow verified bots](/bots/get-started/super-bot-fight-mode/#bot-settings) | **Security** > **Bots** > **Configure Super Bot Fight Mode<br/>Security** > **Bots** > **Configure Bot Management** |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. No #bot-settings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marciocloudflare marciocloudflare marciocloudflare approved these changes

Assignees

@pedrosousa pedrosousa

@dcpena dcpena

@haleycode haleycode

Labels
product:bots Related to Bots product product:security size/s
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@patriciasantaana @marciocloudflare @pedrosousa @dcpena @haleycode