1173 Open source projects that are alternatives of or similar to Ecommercecrawlers

爬取拉勾、BOSS直聘、智联招聘、51job、赶集招聘、58招聘等职位信息

基于搜狗微信搜索的微信公众号爬虫接口

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。

Life is short, *****.

Scrapy as a Web Crawler for Azure Search Samples

bing、google、baidu搜索引擎爬虫。python3.6 and scrapy

PTT 網路版爬蟲

Simple but useful Python web scraping tutorial code.

一个超级轻量的百度图片爬虫

Scrapy-based Crawlers for news of Taiwan

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具

APIs for loginning some websites by using requests.

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

🌈Python3网络爬虫实战：QQ音乐歌曲、京东商品信息、房天下、破解有道翻译、构建代理池、豆瓣读书、百度图片、破解网易登录、B站模拟扫码登录、小鹅通、荔枝微课

Crawlera middleware for Scrapy

Redis-based components for Scrapy.

A Facebook crawler

HTTP API for Scrapy spiders

swiss army knife for hackers

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、百度指数、维普万方、Zlibraty、Oalib、小说、招标网、采购网、小红书》

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

crawler framework, distributed crawler extractor

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

Socialite is an OAuth2 Authentication tool. It is inspired by laravel/socialite, you can easily use it without Laravel.

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

scrapy专利爬虫（停止维护）

百度贴吧、哔哩哔哩、毕方、网易云音乐 云签到

🎊 Design and implement of lightweight crawler framework.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

A framework for creating semi-automatic web content extractors

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Mini program development framework

A multi-thread crawler framework with many builtin image crawlers provided.

Python入门网络爬虫之精华版

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

爬虫, http代理, 模拟登陆!

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Koa middleware and api sdk for wechat oauth, qq oauth, baidu oauth and weibo oauth

Some scrapy and web.py exmaples

依赖Scrapy和搜狗搜索微信公众号文章

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Sample of using proxies to crawl baidu search results.

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

Github 仓库及用户分析爬虫

微信机器人，基于Python itchat接口功能实例展示：01-itchat获取微信好友或者微信群分享文章、02-itchat获取微信公众号文章、03-itchat监听微信公众号发送的文章、04 itchat监听微信群或好友撤回的消息、05 itchat获得微信好友信息以及表图对比、06 python打印出微信被删除好友、07 itchat自动回复好友、08 itchat微信好友个性签名词云图、09 itchat微信好友性别比例、10 微信群或微信好友撤回消息拦截、11 itchat微信群或好友之间转发消息

odoo 微信小程序商城模块 (WeChat Mini Program for Odoo)

Creates an XML-Sitemap by crawling a given site.

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

👾 Fast and simple video download library and CLI tool written in Go

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

Web crawler.

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Reconnaissance , A tool that every pentester/bughunter might wanna consider into their arsenal

百度AI开放平台 Node.js SDK

PhotoBrowser is a light weight photo browser, like the wechat, weibo image viewer.