496 Open source projects that are alternatives of or similar to Spider_python

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

爬虫学习仓库，适合零基础的人学习，对新手比较友好

🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish

Analysis of the characteristics of different countries

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

show full stack technology applications : Scrapy + webservice[restful] + websocket + VueJS + MongoDB

《Python爬虫开发 从入门到实战》配套源代码。

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

Simple but useful Python web scraping tutorial code.

An open source webapp for scraping: towards a public service for webscraping

收集各种免费的 Python 爬虫项目

Async requests-like httplib for python.

Build and deploy a machine learning app from scratch 🚀

A curated list of awesome packages, articles, and other cool resources from the Scrapy community.

C++ Requests: Curl for People, a spiritual port of Python Requests.

use multiple proxies with Scrapy

django-request is a statistics module for django. It stores requests in a database for admins to see, it can also be used to get statistics on who is online etc.

Robot Framework keyword library wrapper for requests

在线地址: http://119.23.223.90:8000

A module that integrates selenium and requests session, encapsulates common page operations, can achieve seamless switching between the two modes.

纯Java实现的支持W3C Xpath 1.0标准语法的HTML解析器。A html parser with xpath base on Jsoup and Antlr4. Maybe it is the best in java,ha ha.Just try it.

A fluent api for working with XML in PHP

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

接口自动化测试框架——python版，支持HTTP，dubbo协议接口

豆瓣电影/豆瓣读书 Scarpy 爬虫

🎊 Design and implement of lightweight crawler framework.

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

camaro is an utility to transform XML to JSON, using Node.js binding to native XML parser pugixml, one of the fastest XML parser around.

🎭 Twisted Deferred Thread backend for Requests.

BaseX Main Repository.

a class that uses scraped proxies to make http GET/POST requests (Python requests)

A mocking library for requests

swiss army knife for hackers

Redis-based components for Scrapy.

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

Minimalist Requests wrapper to work within rate limits of any amount of services simultaneously. Parallel processing friendly.

htmlquery is golang XPath package for HTML query.

Borrow cookies from your browser's authenticated session for use in Python scripts.

💂 Wrap NodeJS request module to retry http requests in case of errors

Kotlin HTTP requests library. Similar to Python requests.

python powered Intelligent System

Custom PHP curl library for the Laravel 5 framework - developed by Ixudra

基于 scrapy-redis 的通用分布式爬虫框架

BeGoneAds is a script that puts some popular hosts file lists into the systems hosts file as a adblocker measure.

A backup tool for renren.com

Android 本地网络小说爬虫，基于jsoup及xpath

A framework for creating semi-automatic web content extractors

Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

Türkçe olarak hazırlanmış Python programlama dili ile ilgili içeriklerin derlendiği sayfa.

The Zipru scraper developed in the Advanced Web Scraping Tutorial.

eXist Native XML Database and Application Platform

大众点评评论文本挖掘，包括点评数据爬取、数据清洗入库、数据分析、评论情感分析等的完整挖掘项目

Web Content Retrieval for Humans™

Extract content from webpages using CSS Selectors, XPath, and JS expressions

Dead easy interface for executing many HTTP requests asynchronously. Also provides helper functions for executing embarrassingly parallel async coroutines.

一个灵活、友好的爬虫框架

Crawlera middleware for Scrapy

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

A set of spiders and scrapers to extract location information from places that post their location on the internet.