496 Open source projects that are alternatives of or similar to Spider_python

百度贴吧爬虫(基于scrapy和mysql)

Asynchronous Library made using Python and aiohttp to get proxies from multiple services!

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

swiss army knife for hackers

Scrapy Training companion code

Document Search Engine Tool

Free proxy server, continuously crawling and providing proxies, based on Tornado and Scrapy. 免费代理服务器，基于Tornado和Scrapy，在本地搭建属于自己的代理池

a simple & tiny scrapy clustering solution, considered a drop-in replacement for scrapyd

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

A web application that contains several unit tests for the purpose of .NET security

基于微博的数据挖掘与社交舆情分析

Redis-based components for Scrapy.

😩Tool For Taobao/Tmall| 儿时玩具已经过时

Scrapy, a fast high-level web crawling & scraping framework for dart and Flutter

scrapy框架爬取51job(scrapy.Spider)，智联招聘(扒接口)，拉勾网(CrawlSpider)

Building a model to recognize incentives for landscape restoration in environmental policies from Latin America, the US and India. Bringing NLP to the world of policy analysis through an extensible framework that includes scraping, preprocessing, active learning and text analysis pipelines.

all kinds of scrapy demo

☕🗄CAching Proxy in Python – Simple file based python http proxy

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

免费 IP 代理池。Scrapy 爬虫框架插件

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

基于Scrapy框架的豆瓣电影爬虫

HomeAssistant HeWeather Plugin

用爬虫爬取小说网站上所有小说，存储到数据库中，并用爬到的数据构建自己的小说网站

Minimalist Requests wrapper to work within rate limits of any amount of services simultaneously. Parallel processing friendly.

Backend para coleta e disponibilização dos dados 📜

163music spider by scrapy.

微博爬虫：每天定时爬取微博热搜榜的内容，留下互联网人的记忆。

PTT 網路版爬蟲

最新动态在这里【我的程序员日志】

A python wrapper for Indian Railways Enquiry API!

lots of spider (很多爬虫）

htmlquery is golang XPath package for HTML query.

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

Code examples and slides from my 2021 talk Server-Side is Dead! Long Live Server-Side (+ HTMX), presented at DjangoCon and Code Code Code

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

知名網站avgle下載器

an awesome public proxy server crawler based on scrapy framework

Spin up Tor containers and then proxy HTTP requests via these Tor instances

Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner

Borrow cookies from your browser's authenticated session for use in Python scripts.

The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

An easy command line interface for the 2018 World Cup

capture pictures from website like sina, lofter, huaban and so on

HTTP-based DSL for for validating RESTful APIs

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

The source code for animecenter

使用Scrapy采集淘宝数据，Flask展示

💂 Wrap NodeJS request module to retry http requests in case of errors

Crawling zhihu, jobbole, lagou by Scrapy, and using Elasticsearch+Django to build a Search Engine website --- README_zh.md (including: implementation roadmap, distributed-crawler and coping with anti-crawling strategies).

An image scraper that scraps images from unsplash.com

A Python obfuscator using HTTP Requests and Hastebin.

给uiautomatorview添加xpath等待

R interface to Python requests module

Scrapy middleware to handle javascript pages using selenium

A Facebook crawler

Random User-Agent middleware based on fake-useragent

Guzzle, an extensible PHP HTTP client