40 open source projects by scrapinghub

NER toolkit for HTML data

Extract price amount and currency symbol from a raw text string

Useful test spiders for Scrapy

A client interface for Scrapinghub's API

Python parser for Adblock Plus filters

Scrapy Training companion code

python parser for human readable dates

Convert Javascript code to an XML document

Skinfer is a tool for inferring and merging JSON schemas

Automatic Item List Extraction

A python implementation of DEPTA

UNMAINTAINED Python wrapper for Wappalyzer (utility that uncovers the technologies used on websites)

A scalable frontier for web crawlers

A chat bot for Slack (https://slack.com).

Visual scraping for Scrapy

A simple algorithm for clustering web pages, suitable for crawlers

Crawlera tools

A python binding for crfsuite

HTTP API for Scrapy spiders

Extract embedded metadata from HTML markup

Lightweight, scriptable browser as a service with an HTTP API

Scrapy Extension for monitoring spiders execution.

Frontera backend to guide a crawl using PageRank, HITS or other ranking algorithms based on the link structure of the web graph, even when making big crawls (one billion pages).

Collection of Scrapy utilities (extensions, middlewares, pipelines, etc)

Automatic unit test generation for Scrapy.

Scrapy entrypoint for Scrapinghub job runner

Tool to flatten stream of JSON-like objects, configured via schema

MongoDB extensions for Scrapy

High Level Kafka Scanner

Paginating the web

Software stack with latest Scrapy and updated deps

Scrapinghub Command Line Client

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations

Restrict crawl and scraping scope using matchers.

Web scraping Page Objects core library

Parse numbers written in natural language

A python library detect and extract listing data from HTML page.

Analyze scraped data

Python bindings for CLD2.

pypi caching service using devpi and docker