355 Open source projects that are alternatives of or similar to scrapy-wayback-machine

Random User-Agent middleware based on fake-useragent

Scrape, standardize and share public meetings from local government websites

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

Scrapy Training companion code

A Scrapy extension to log items coverage when the spider shuts down

Scraping Ebay's products using Scrapy Web Crawling Framework

A framework for creating semi-automatic web content extractors

Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.

Scrapy crawler to collect data on the back catalog of songs listed for sale.

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Netflix like full-stack application with SPA client and backend implemented in service oriented architecture

Faster requests on Python 3

Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).

scrapy 常用爬网必备工具包

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

⏪ Tools to Work with the Various Internet Archive Wayback Machine APIs

An unofficial API for Quora.

Dynamic configurable crawl (动态可配置化爬虫)

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Công cụ quét và phân tích từ khoá các trang báo mạng Việt Nam

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cungmua with flexible configs

Crawling and scraping the Web for fun and profit

Python scripts for building 'Short Jokes' dataset, featured on Kaggle

Tutorial: Web scraping in Python with Beautiful Soup

Basic setup with random user agents and IP addresses for Python Scrapy Framework.

Twitter Intelligence OSINT project performs tracking and analysis of the Twitter

A web application that generates timetables for university students at the University of Toronto

Crawlzone is a fast asynchronous internet crawling framework for PHP.

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Araignée LeBonCoin avec Scrapy et ElasticSearch

Exercises, data, and more for our 2017 summer workshop (funded by the Estes Fund and in partnership with Project Jupyter and Berkeley's D-Lab)

Web scrapping and related analytics using Python tools

Selenium-python but lighter: Helium is the best Python library for web automation.

🎥 🍿 Streaming Public domain movies

Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.

A scrapy middleware to use rotated proxy ip list.

Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

python爬虫项目集合

Web Scraping Framework

A Programming language for Web Scraping

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Django to build a Search Engine website --- README_zh.md (including: implementation roadmap, distributed-crawler and coping with anti-crawling strategies).

Web Scraping, Document Deduplication & GPT-2 Fine-tuning with a newly created scam dataset.

Use Puppeteer to download pens from Codepen.io as single html pages

Navigator for Web Archive

This repository provides everything you need to get started with Python for (social science) research.

Using scrapy to scrape tripadvisor in order to get users' reviews.

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, SHFE and news data crawlers on BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

使用asyncio和aiohttp开发的轻量级异步协程web爬虫框架

Analyze scraped data

A Chrome extension for writing custom web scraping programs and web automation programs. Just demonstrate how to collect the first row of data, then let the extension write the program for collecting all rows.

PHP Scraper - an highly opinionated web-interface for PHP

python+scrapy+splash 爬取拉勾全站职位信息

Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)

Zillow Scraper for Python using Selenium

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Web scraping Page Objects core library

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

MetaData html scraper and parser for Node.js (supports Promises and callback style)

Apify actor that opens a web page in headless Chrome and analyzes the HTML and JavaScript objects, looks for schema.org microdata and JSON-LD metadata, analyzes AJAX requests, etc.

Building a Concurrent Web Scraper with Python and Selenium