487 Open source projects that are alternatives of or similar to CrawlBox

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

A collection of awesome web crawler,spider in different languages

Continuous scalable web crawler built on top of Flink and crawler-commons

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

A simple but powerful web crawler library for .NET

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

The simple, easy to use command line web crawler.

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

简单易用的Python爬虫框架，QQ交流群：597510560

Rust Web Crawler saving pages on Redis

A simple distributed crawler for zhihu && data analysis

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Auto Crawler Ptt Beauty Image Use Python Schedule

Just a simple web crawler which return crawled links as IObservable using reactive extension and async await.

Scrape bot, to get you an offline copy of tutorials

PTT 網路版爬蟲

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

Biblioteca feita em Python com o objetivo de facilitar o acesso a dados de seus investimentos na bolsa de valores(B3/CEI) através do Portal CEI.

Distributed task redisqueue(最简单python分布式函数调度框架)

📷 Automate full website screenshots and PDF generation with multiple viewport support.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

It is a web crawler which crawls the stackoverfolw website (http://stackoverflow.com/) and finds the most popular technologies at current point of time by getting the tags info of the newest questions asked on the website.

Brutal Wordlist Generator is a java based Application software used to generate the wordlist with best of UX interface

🤖 Prerendering for JavaScript powered websites. Great solution for PWAs (Progressive Web Apps), SPAs (Single Page Applications), and other websites based on top of front-end JavaScript frameworks

Create awesome wordlist with python, demo: https://asciinema.org/a/101677

A simple and flexible web crawler framework for java.

a fast password wordlist generator, Smartlist creation and password hybrid-mask analysis tool written in pure safe Rust

🌀 crawl bilibili user info and video info for data analysis | BiliBili爬虫

Company Passwords Profiler (aka ComPP) helps making a bruteforce wordlist for a targeted company.

练习NLP，分析淘宝评论的项目

Use JSON to list all elements (with css 3 and jquery selector) that you want to crawl.

A Python DNS crawler to find identical domain names under different TLDs.

扫描“微信读书”已购图书并下载本地PDF的爬虫

🌈Python3网络爬虫实战：QQ音乐歌曲、京东商品信息、房天下、破解有道翻译、构建代理池、豆瓣读书、百度图片、破解网易登录、B站模拟扫码登录、小鹅通、荔枝微课

A support library for Ronin. Like activesupport, but for hacking!

Raspagem de dados para iniciante usando Scrapy e outras libs básicas

Crawling CV conference papers with Python.

📚 An ultimate collection wordlists of the best-known CMS

Practical pinned gist which show your latest medium status 📌

🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |

A web crawler for Go

Customized Password/Passphrase List inputting Target Info

Interactive CLI Web Crawler

Python Script to help/automate the WiFi hacking exercises.

Leak other players' temporary workspaces for ctf and wargames.

Google search results crawler, get google search results that you need - php

一个用来快速提取网页内容的信息采集（爬虫）框架， 实现了对网页的动态加载与控制。

Explore a website recursively and download all the wanted documents (PDF, ODT…)

WEB PENETRATION TESTING TOOL 💥

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

Extract endpoints marked as disallow in robots files to generate wordlists.

Amazingly fast response crawler to find juicy stuff in the source code! 😎🔥

🐤️ Lost Ark wait notifier