282 Open source projects that are alternatives of or similar to pythonSpider

A framework for creating semi-automatic web content extractors

Simple but useful Python web scraping tutorial code.

lots of spider (很多爬虫）

🎊 Design and implement of lightweight crawler framework.

Scrapy, a fast high-level web crawling & scraping framework for dart and Flutter

This repository provides usage examples for the Python module Newspaper3k.

V2EX爬虫

Python web-scraping library that wraps urllib2 and BeautifulSoup

python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取（生成cookie）、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答

Scrapy project boilerplate done right

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码

基于Scrapy的爬虫demo

A simple flask api for Atlas Obscura.

A tutorial-based introduction to web scraping with Python.

163music spider by scrapy.

scrapy 常用爬网必备工具包

The source code for animecenter

秀人网爬虫 55156爬虫

Simple and friendly Bot for Instagram, using Selenium and Scrapy with Python.

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

🕸 List of mini projects that involve web scraping 🕸

web scraping tool for grailed.com

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

Simplify your ETL processes with these hands-on data sanitation tips, tricks, and best practices

extract data from html table

Document Search Engine Tool

Use Scrapy crawl foreign exchange rate from BOC (Bank of China)

Docker microservice & Crawler by scrapy

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

基于Scrapy+Elasticsearch+Django搭建的分布式电影搜索

免费 IP 代理池。Scrapy 爬虫框架插件

小密圈备份，圈子/话题/图片/文件。

A middleware of cookies persistence for Scrapy

Script using requests module to register accounts to Slam Jam Socialism raffles.

一些简单有趣的Python小Demo

Web scraper project to collect the useful Hong Kong weather data from HKO website

Spin up Tor containers and then proxy HTTP requests via these Tor instances

收集各种免费的 Python 爬虫项目

A dart port of the famous python library beautifulsoup

An Inshorts API which provides news for different categories from the Inshorts Website. Built using Flask and Beautiful Soup.

A Web Crawler that scrapes using YAML and python code.

Personal Data Engineering Projects

Pawn package manager for SA-MP

A Scrapy demo : Download all images from a site

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

Scraping Ebay's products using Scrapy Web Crawling Framework

爬虫学习仓库，适合零基础的人学习，对新手比较友好

No description or website provided.

Using Scrapy to crawl Autohome, storage into MonogDB, simple analysis and NLP coming soon

Bangumi小工具

Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).

Scrapy项目（mysql+mongodb豆瓣top250电影）

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

No description or website provided.

Automate the process of sending referral request and cold mailing on LinkedIn

Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.

Scrape your favorite Anime from Okanime.com without effort

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy