Skip to content
@scrapy

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Pinned Loading

  1. scrapy scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python 65.1k 12.1k

  2. scrapyd scrapyd Public

    A service daemon to run Scrapy spiders

    Python 3.1k 581

  3. parsel parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    Python 1.4k 180

  4. w3lib w3lib Public

    Python library of web-related functions

    HTML 419 139

  5. protego protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    DIGITAL Command Language 95 39

  6. itemadapter itemadapter Public

    Common interface for data container classes

    Python 70 22

Repositories

Showing 10 of 35 repositories
  • cssselect Public

    CSS Selectors for Python

    scrapy/cssselect's past year of commit activity
    Python 311 73 5 7 Updated Nov 1, 2026
  • w3lib Public

    Python library of web-related functions

    scrapy/w3lib's past year of commit activity
    HTML 419 BSD-3-Clause 139 5 13 Updated Nov 1, 2026
  • scrapy-mcp-official Public

    Scrapy MCP server

    scrapy/scrapy-mcp-official's past year of commit activity
    Python 14 Apache-2.0 1 0 0 Updated Nov 1, 2026
  • formerly Public

    Deprecate a class that users inherit from

    scrapy/formerly's past year of commit activity
    Python 2 BSD-3-Clause 1 0 0 Updated Nov 1, 2026
  • scrapy-lint Public

    A linter for Scrapy projects.

    scrapy/scrapy-lint's past year of commit activity
    Python 22 MIT 9 20 23 Updated Nov 1, 2026
  • parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    scrapy/parsel's past year of commit activity
    Python 1,367 BSD-3-Clause 179 7 6 Updated Oct 30, 2026
  • scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    scrapy/scrapy's past year of commit activity
    Python 65,148 BSD-3-Clause 12,083 107 (4 issues need help) 80 Updated Oct 30, 2026
  • itemadapter Public

    Common interface for data container classes

    scrapy/itemadapter's past year of commit activity
    Python 70 BSD-3-Clause 22 2 3 Updated Oct 29, 2026
  • itemloaders Public

    Library to populate items using XPath and CSS with a convenient API

    scrapy/itemloaders's past year of commit activity
    Python 49 BSD-3-Clause 25 8 2 Updated Oct 28, 2026
  • protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    scrapy/protego's past year of commit activity
    DIGITAL Command Language 95 BSD-3-Clause 39 0 1 Updated Oct 27, 2026
View all repositories