1913 Open source projects that are alternatives of or similar to Not Your Average Web Crawler

A Python based web application scanner to gather OSINT and fuzz for OWASP vulnerabilities on a target website.

A high performance web crawler in Elixir.

A Facebook crawler

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

High performance crawler for Nodejs

Scraping scripts for various websites.

A collection of awesome web crawler,spider in different languages

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

一款信息泄漏利用工具，适用于.git/.svn源代码泄漏和.DS_Store泄漏

OSINT Swiss Army Knife

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Elegant Scraper and Crawler Framework for Golang

快速、简洁且强大的PHP爬虫框架

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Crawly, a high-level web crawling & scraping framework for Elixir.

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Free Web Scraping Tool with Java

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

一个超级轻量的百度图片爬虫

HTTP API for Scrapy spiders

Web path scanner

[Unmaintained] A simple and clean video/music/image downloader 👾

Distributed crawler powered by Headless Chrome

Dark Web OSINT Tool

Tool for information gathering, IPReverse, AdminFInder, DNS, WHOIS, SQLi Scanner with google.

API of DouYin for Humans used to Crawl Popular Videos and Musics

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

抖音 SDK，数据采集，爬虫抓取不是梦

A multi-thread crawler framework with many builtin image crawlers provided.

带你了解一下Golang的市场行情

Gospider - Fast web spider written in Go

Popular Pentesting scanner in Python3.6 for SQLi/XSS/LFI/RFI and other Vulns

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

🐾 Creeper - The Next Generation Crawler Framework (Go)

massive SQL injection vulnerability scanner

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A python web scraper for public email lists.

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Auto Scanning to SSL Vulnerability

vulnx 🕷️ is an intelligent bot auto shell injector that detect vulnerabilities in multiple types of cms { `wordpress , joomla , drupal , prestashop .. `}

💐 Full Amazon Automatic Download

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Rust Web Crawler saving pages on Redis

Incredibly fast crawler designed for OSINT.

Creating Scrapy scrapers via the Django admin interface

Open Vulnerability Assessment Scanner - Scanner for Greenbone Vulnerability Management (GVM)

A Fast Broken Link Hijacker Tool written in Python

Async Python 3.6+ web scraping micro-framework based on asyncio

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

[DEPRECATED] Simple, flexible, delightful web crawler/spider package

Tổng hợp script crawl dữ liệu từ các mạng xã hội & website tiếng Việt

It's a simple tool for test vulnerability shellshock

Golang爬虫 爬取汽车之家 二手车产品库

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

python crawler spider