feat: add respect_robots_txt_file option
#1162
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a new boolean flag, respect_robots_txt_file, to automatically skip crawling disallowed URLs based on a site's robots.txt rules. Key changes include the addition of tests for robots.txt handling across multiple crawler implementations, integration of robots.txt checking in the crawling pipeline, and the implementation of a RobotsTxtFile utility.
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/server_endpoints.py | Added a static ROBOTS_TXT response to simulate a robots.txt file. |
| tests/unit/server.py | Introduced a new endpoint to serve robots.txt and updated routing logic. |
| tests/unit/crawlers/_playwright/test_playwright_crawler.py | Added tests verifying that the PlaywrightCrawler correctly respects robots.txt. |
| tests/unit/crawlers/_parsel/test_parsel_crawler.py | Introduced tests for the ParselCrawler to validate robots.txt respect. |
| tests/unit/crawlers/_beautifulsoup/test_beautifulsoup_crawler.py | Added tests to ensure BeautifulSoupCrawler adheres to robots.txt rules. |
| tests/unit/_utils/test_robots.py | New tests for generating, parsing, and validating robots.txt file behavior. |
| src/crawlee/crawlers/_playwright/_playwright_crawler.py | Integrated robots.txt enforcement in the link extraction logic. |
| src/crawlee/crawlers/_basic/_basic_crawler.py | Updated request adding and session handling to respect robots.txt directives. |
| src/crawlee/crawlers/_abstract_http/_abstract_http_crawler.py | Added robots.txt checking in link extraction for HTTP-based crawling. |
| src/crawlee/_utils/robots.py | Implemented the RobotsTxtFile class for parsing and handling robots.txt data. |
| pyproject.toml | Added dependency for protego to support robots.txt parsing. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
| @@ -40,6 +40,7 @@ dependencies = [ | |||
| "eval-type-backport>=0.2.0", | |||
| "httpx[brotli,http2,zstd]>=0.27.0", | |||
| "more-itertools>=10.2.0", | |||
| "protego>=0.4.0", | |||
There was a problem hiding this comment.
It's fun to see another scrapy project here, but I guess that it guarantees some stability, so... all good.
There was a problem hiding this comment.
Yes, I was planning to use RobotFileParser, but it doesn't support Google's specification. 😞
src/crawlee/_utils/robots.py
Outdated
| self._robots = robots | ||
| self._original_url = URL(url).origin() | ||
|
|
||
| @staticmethod |
There was a problem hiding this comment.
I'd prefer using @classmethod and the Self return type annotation
| robots_txt_file = await self._get_robots_txt_file_for_url(url) | ||
| return not robots_txt_file or robots_txt_file.is_allowed(url) | ||
|
|
||
| async def _get_robots_txt_file_for_url(self, url: str) -> RobotsTxtFile | None: |
There was a problem hiding this comment.
I believe we should use some synchronization mechanism so that we don't fetch the same robots.txt file multiple times in parallel.
Co-authored-by: Jan Buchar <Teyras@gmail.com>
src/crawlee/_utils/robots.py
Outdated
|
|
||
| from typing import TYPE_CHECKING | ||
|
|
||
| from protego import Protego # type: ignore[import-untyped] |
There was a problem hiding this comment.
Could we please rather update the project toml rather than using a type ignore?
src/crawlee/_utils/robots.py
Outdated
| """Create a RobotsTxtFile instance from the given content. | ||
| Args: | ||
| url: the URL of the robots.txt file |
There was a problem hiding this comment.
Could you please use sentences in arg description? (applies to all occurencies)
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Description
respect_robots_txt_file.Issues
respectRobotsTxtFilecrawler option #1144Testing
respect_robots_txt_filefunctioning in ‘EnqueueLinksFunction’ for crawlersRobotsTxtFile