627 Open source projects that are alternatives of or similar to proxycrawl-python

Declarative web scraping

Retrieve real (with Javascript executed) HTML code from an URL, ultra fast and supports multiple parallel loading of webs

No description or website provided.

Elegant Scraper and Crawler Framework for Golang

[Unmaintained] A simple and clean video/music/image downloader 👾

**[ARCHIVED]** website changes tracker 🔍

Distributed crawler powered by Headless Chrome

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Crawly, a high-level web crawling & scraping framework for Elixir.

Extract structured data from web sites. Web sites scraping.

Command line program to download documents from web portals.

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

Monitor instagram user account and automatically post new images to discord channel via a webhook. Working 2022!

An open source web crawling platform

NodeJS package that fetches a given URL's title, description, images, links etc.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Nodejs web scraper. Contains a command line, docker container, terraform module and ansible roles for distributed cloud scraping. Supported databases: SQLite, MySQL, PostgreSQL. Supported headless clients: Puppeteer, Playwright, Cheerio, JSdom.

A Scraper made 100% in Python using BeautifulSoup and Tor. It can be used to scrape both normal and onion links. Happy Scraping :)

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

🦀 Scraping & crawling websites using Rust, and ReasonML

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

A Python Tool For google Hacking

Search for and retrieve US Patent and Trademark Office Patent Data

A collection of tools to (semi-)automatically collect and analyze data from online discussions on Facebook groups and pages.

✂️ High performance, multi-threaded image scraper

Creating Scrapy scrapers via the Django admin interface

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

A tutorial-based introduction to web scraping with Python.

Scraping the status of WhatsApp contacts

🕸 List of mini projects that involve web scraping 🕸

A Scraper, Downloader, & Recorder for static open directories.

Easily crawl news portals or blog sites using Storm Crawler.

🎬 An attempt at the most complete IMDb API

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.

Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.

simple multi-level scraper json input/output for Cheerio

HTTP API for Scrapy spiders

The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

All-in-one Python (And now Go!) module to help solve captchas with Capmonster, 2captcha and Anticaptcha API's!

PHP Scraper - an highly opinionated web-interface for PHP

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Your will to enroll in Udemy course is here, but the money isn't? Search no more! This python program searches for your desired course in more than [insert big number here] websites, compares the last updated date, and gives you the download link of the latest one back, but you also have the choice to see the other ones as well!

This class can retrieve search results from Google.

Implementation of "Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading." In Findings of ACL2021

At present contains scraped data from around 1500 problems present on the site. More to follow....

reborn of https://bitbucket.org/rflechner/scrapysharp

Universal scrapping tool, which allows you to extract data using multiple environments

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Tutorial for web scraping / crawling with Node.js.

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

A CLI for Mozilla Readability. Get clean, uncluttered, ready-to-read HTML from any webpage!

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

scrapers for building your own image databases

A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

Pahe.ph [Pahe.in] Movies Website Scraper