Examples Of Web Crawlers一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
Stars: ✭ 10,724 (+7615.11%)
Black WidowGUI based offensive penetration testing tool (Open Source)
Stars: ✭ 124 (-10.79%)
Sentinel CrawlerXenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器,分布式爬虫
Stars: ✭ 118 (-15.11%)
Instagram Profilecrawl💻 Quickly crawl the information (e.g. followers, tags, etc...) of an instagram profile. No login required!
Stars: ✭ 110 (-20.86%)
NewspaperNews, full-text, and article metadata extraction in Python 3. Advanced docs:
Stars: ✭ 11,545 (+8205.76%)
GraphqueryGraphQuery is a query language and execution engine tied to any backend service.
Stars: ✭ 112 (-19.42%)
Php CrawlerA php crawler that finds emails on the internets
Stars: ✭ 119 (-14.39%)
FawkesFawkes is a tool to search for targets vulnerable to SQL Injection. Performs the search using Google search engine.
Stars: ✭ 108 (-22.3%)
Dota2🐸 Python package for interacting with Dota 2 Game Coordinator
Stars: ✭ 129 (-7.19%)
Moodle Downloader 2A Moodle downloader that downloads course content fast from Moodle (eg. lecture pdfs)
Stars: ✭ 118 (-15.11%)
4chan DownloaderPython3 script to continuously download all images/webms of multiple 4chan thread simultaneously - without installation
Stars: ✭ 136 (-2.16%)
SquidwarcSquidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
Stars: ✭ 125 (-10.07%)
Jianso movie🎬 电影资源爬虫,电影图片抓取脚本,Flask|Nginx|wsgi
Stars: ✭ 114 (-17.99%)
Saliens HackHack for Sailens, the game of Steam Summer Sale 2018 - AutoSelect Planet, Invincibility, and InstaKill
Stars: ✭ 113 (-18.71%)
Bitlbee SteamSteam protocol plugin for BitlBee
Stars: ✭ 122 (-12.23%)
Steamtools🛠「Steam++」是一个开源跨平台的多功能Steam工具箱。
Stars: ✭ 4,458 (+3107.19%)
LinkcrawlerCross-platform persistent and distributed web crawler 🔗
Stars: ✭ 109 (-21.58%)
Pspider简单易用的Python爬虫框架,QQ交流群:597510560
Stars: ✭ 1,611 (+1058.99%)
Tiebamanager(已跑路)百度贴吧吧务管理工具,自动扫描帖子并处理违规帖
Stars: ✭ 119 (-14.39%)
ScrapyScrapy, a fast high-level web crawling & scraping framework for Python.
Stars: ✭ 42,343 (+30362.59%)
Proton CallerRun any Windows program through Proton
Stars: ✭ 130 (-6.47%)
OnegramThis repository is no longer maintained.
Stars: ✭ 137 (-1.44%)
Docs《数据采集从入门到放弃》源码。内容简介:爬虫介绍、就业情况、爬虫工程师面试题 ;HTTP协议介绍; Requests使用 ;解析器Xpath介绍; MongoDB与MySQL; 多线程爬虫; Scrapy介绍 ;Scrapy-redis介绍; 使用docker部署; 使用nomad管理docker集群; 使用EFK查询docker日志
Stars: ✭ 118 (-15.11%)
DecryptloginAPIs for loginning some websites by using requests.
Stars: ✭ 1,861 (+1238.85%)
Go spider[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
Stars: ✭ 1,745 (+1155.4%)
BaiducrawlerSample of using proxies to crawl baidu search results.
Stars: ✭ 116 (-16.55%)
Memex ExplorerViewers for statistics and dashboarding of Domain Search Engine data
Stars: ✭ 115 (-17.27%)
Goclone Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
Stars: ✭ 134 (-3.6%)
Instagram BotAn Instagram bot developed using the Selenium Framework
Stars: ✭ 138 (-0.72%)
Pkulaw spider爬取北大法宝网http://www.pkulaw.cn/Case/
Stars: ✭ 113 (-18.71%)
Crawlab LiteLite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (-12.23%)
Lcrawl一只优雅的正方教务系统爬虫。
Stars: ✭ 112 (-19.42%)
Red hawkAll in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
Stars: ✭ 1,898 (+1265.47%)
BaiduspiderBaiduSpider,一个爬取百度搜索结果的爬虫,目前支持百度网页搜索,百度图片搜索,百度知道搜索,百度视频搜索,百度资讯搜索,百度文库搜索,百度经验搜索和百度百科搜索。
Stars: ✭ 105 (-24.46%)
RemoteplaywhateverTiny application that lets you force remote play together any game you have in your steam library including non-steam ones.
Stars: ✭ 138 (-0.72%)
Pylinkvalidatorpylinkvalidator is a standalone and pure python link validator and crawler that traverses a web site and reports errors (e.g., 500 and 404 errors) encountered.
Stars: ✭ 109 (-21.58%)
Qqmusicspider基于Scrapy的QQ音乐爬虫(QQ Music Spider),爬取歌曲信息、歌词、精彩评论等,并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料
Stars: ✭ 120 (-13.67%)
LumberjackAn automated website accessibility scanner and cli
Stars: ✭ 109 (-21.58%)
Mm131MM131网站图片爬取 🚨
Stars: ✭ 129 (-7.19%)
Amazonbigspider😱Full Automatic Amazon Distributed Spider | 亚马逊分布式四国际站采集选款产品|账号admin,密码adminadmin
Stars: ✭ 140 (+0.72%)
SearchAn Open Source Search Engine
Stars: ✭ 139 (+0%)
Zhihu Spider一个获取知乎用户主页信息的多线程Python爬虫程序。
Stars: ✭ 137 (-1.44%)
DiggerDigger is a powerful and flexible web crawler implemented by pure golang
Stars: ✭ 130 (-6.47%)
TmodloaderA mod to make and play Terraria mods. Supports Terraria 1.4 installations - TML itself is 1.3 Terraria currently
Stars: ✭ 2,130 (+1432.37%)