All Projects → abaykan → CrawlBox

abaykan / CrawlBox

Licence: Unlicense License
Easy way to brute-force web directory.

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to CrawlBox

Spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (+455.93%)
Mutual labels:  crawler, web-crawler
Infinitycrawler
A simple but powerful web crawler library for .NET
Stars: ✭ 97 (-17.8%)
Mutual labels:  crawler, web-crawler
Maman
Rust Web Crawler saving pages on Redis
Stars: ✭ 39 (-66.95%)
Mutual labels:  crawler, web-crawler
Supercrawler
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
Stars: ✭ 306 (+159.32%)
Mutual labels:  crawler, web-crawler
Abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+1561.86%)
Mutual labels:  crawler, web-crawler
Spider Flow
新一代爬虫平台,以图形化方式定义爬虫流程,不写代码即可完成爬虫。
Stars: ✭ 365 (+209.32%)
Mutual labels:  crawler, web-crawler
Terpene Profile Parser For Cannabis Strains
Parser and database to index the terpene profile of different strains of Cannabis from online databases
Stars: ✭ 63 (-46.61%)
Mutual labels:  crawler, web-crawler
Crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Stars: ✭ 8,392 (+7011.86%)
Mutual labels:  crawler, web-crawler
Crawlab Lite
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (+3.39%)
Mutual labels:  crawler, web-crawler
Pspider
简单易用的Python爬虫框架,QQ交流群:597510560
Stars: ✭ 1,611 (+1265.25%)
Mutual labels:  crawler, web-crawler
flink-crawler
Continuous scalable web crawler built on top of Flink and crawler-commons
Stars: ✭ 48 (-59.32%)
Mutual labels:  crawler, web-crawler
Antch
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Stars: ✭ 198 (+67.8%)
Mutual labels:  crawler, web-crawler
Gopa
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (+134.75%)
Mutual labels:  crawler, web-crawler
Awesome Crawler
A collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+3961.86%)
Mutual labels:  crawler, web-crawler
Spidy
The simple, easy to use command line web crawler.
Stars: ✭ 257 (+117.8%)
Mutual labels:  crawler, web-crawler
D4n155
OWASP D4N155 - Intelligent and dynamic wordlist using OSINT
Stars: ✭ 105 (-11.02%)
Mutual labels:  crawler, wordlist
Zhihu Crawler People
A simple distributed crawler for zhihu && data analysis
Stars: ✭ 182 (+54.24%)
Mutual labels:  crawler, web-crawler
Strong Web Crawler
基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。
Stars: ✭ 238 (+101.69%)
Mutual labels:  crawler, web-crawler
spiderable-middleware
🤖 Prerendering for JavaScript powered websites. Great solution for PWAs (Progressive Web Apps), SPAs (Single Page Applications), and other websites based on top of front-end JavaScript frameworks
Stars: ✭ 29 (-75.42%)
Mutual labels:  crawler
auto crawler ptt beauty image
Auto Crawler Ptt Beauty Image Use Python Schedule
Stars: ✭ 35 (-70.34%)
Mutual labels:  crawler

CrawlBox |

Awesome GitHub forks GitHub stars HitCount GitHub license TypeScript
Open Source Love bitHound PyPI - Python Version Build Status

Easy way to brute-force web directory.


Operating Systems Tested

  • MacOSX
  • Kali Linux
  • Ubuntu 16.04

Usage

python crawlbox.py [-h] [-v] [-w WORDLIST] url

positional arguments:

url            specific target url, like domain.com

optional arguments:

  -h, --help     show this help message and exit
  -v, --version  show program's version number and exit
  -w WORDLIST    specific path to wordlist file
  -d DELAY       add delay between requests

Example

web site scan with internal wordlist

python crawlbox.py www.domain.com

web site scan with external wordlist

python crawlbox.py www.domain.com -w wordlist.txt

Install

(as root)

  git clone https://github.com/abaykan/crawlbox.git
  cd crawlbox/
  pip install -r requirements.txt
  python crawlbox.py -h

note: tested with python 2.7.6

Media Review

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].