499 Open source projects that are alternatives of or similar to Strong Web Crawler

Fawkes is a tool to search for targets vulnerable to SQL Injection. Performs the search using Google search engine.

Headless Chrome For Java （Java 爬虫）

Discover hidden deepweb pages

Simple but useful Python web scraping tutorial code.

A scalable web crawler framework for Java.

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot

Website screenshot service powered by Node.js, SlimerJS and PhantomJS

A web crawler (for bug hunting) that gathers more than you can imagine.

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

Capture screenshot of a website and return it as a stream

Distributed crawler powered by Headless Chrome

爬虫, http代理, 模拟登陆!

A Facebook crawler

根据关键词抓取微博数据，再生成词云

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

A bridge between node and PhantomJS

Get (almost) original messages from google group archives. Your data is yours.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

PhantomJS resource pool based on generic-pool

a new crawler based on python with more function including Network fingerprint search

Dynamic meta tags in your AngularJS single page application

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

字体混淆服务

Precompiled packages for AWS Lambda

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Declarative web scraping

Crawler for Nintendo Switch eShop

Java 網路資料爬蟲包

LinkedIn Scraper (currently working 2020)

📢 Ptt 文章通知機器人！Notify Ptt Article in Realtime

Crawl BookCorpus

抖音 SDK，数据采集，爬虫抓取不是梦

Html网页正文提取

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

爬取菜鸟教程网站并转PDF__python_crawer_by_chrome

下载搜狗、百度、QQ输入法的词库文件的 python 爬虫，可用于构建不同行业的词汇库

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

CoCrawler is a versatile web crawler built using modern tools and concurrency.

Open-source Enterprise Grade Search Engine Software

OSINT Swiss Army Knife

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

Crawl a website and run it through Google lighthouse

A high-level browser automation library.

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

A simple tool for fetching usable proxies from several websites.

🔍 简单的搜索引擎, django 框架

A python3 crawler for crawling Pixiv ranking top and any illustrator all artworks

Crawler for LinkedIn full profiles 2019

GUI based offensive penetration testing tool (Open Source)

Find web directories without bruteforce

A GUI client of schannel powered by therecipe/qt and golang

The fast website crawler

那些年，我爬过的北科。一个由浅入深的定向爬虫教程。

💦 Tools to Work with the 'Splash' JavaScript Rendering Service in R