487 Open source projects that are alternatives of or similar to Spidy

疫情数据爬虫，2019新型冠状病毒数据仓库，轨迹数据，同乘数据，报道

High performance crawler for Nodejs

pylinkvalidator is a standalone and pure python link validator and crawler that traverses a web site and reports errors (e.g., 500 and 404 errors) encountered.

vulnx 🕷️ is an intelligent bot auto shell injector that detect vulnerabilities in multiple types of cms { `wordpress , joomla , drupal , prestashop .. `}

🥄 A package for building specific Proxy Pool for different Sites.

Web crawler.

Cross-platform persistent and distributed web crawler 🔗

🔍 简单的搜索引擎, django 框架

Crawler for LinkedIn full profiles 2019

Script to fetch webcomics and use them to create ebooks.

An automated website accessibility scanner and cli

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Reconnaissance , A tool that every pentester/bughunter might wanna consider into their arsenal

Fawkes is a tool to search for targets vulnerable to SQL Injection. Performs the search using Google search engine.

The fast website crawler

Python爬虫代理IP池(proxy pool)

File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch

A Scrapy extension to log items coverage when the spider shuts down

news-please - an integrated web crawler and information extractor for news that just works.

Crawl some picture for fun

抖音爬虫. 通过手机代理爬取用户的作品和用户的喜欢

🐞简单轻便的Java爬虫框架，只要会一点简单的正则表达式和简单的css选择器就能轻松的采集数据。

抖音爬虫，tiktok crawler，抖音数据采集接口，抖音视频去水印，百分百成功，不需要服务器，不需要代理 IP。

用严肃的数据来回答“什么样的企业会到什么样的大学招聘”？

Raspagem de dados para iniciante usando Scrapy e outras libs básicas

Crawler used to crawl papers

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

Tor website crawler (specific for Alphabay at the time)

TumblTwo, an Improved Fork of TumblOne, a Tumblr Downloader.

Crawl websites for accessibility issues from the command line.

Alternative for VK Audio API

Simple CORPORA list crawler

Dataset search engine, discovering data from a variety of sources, profiling it, and allowing advanced queries on the index

爬取amazon/bestbuy/costco/6pm 的商品详情

一款分布式爬虫平台，帮助你更好的管理和开发爬虫。 内置一套爬虫定义规则（模版），可使用模版快速定义爬虫，也可当作框架手动开发爬虫。(兴趣使然的项目，用的不爽了就更新)

Implements the crawler package into Symfony

A simple and flexible web crawler framework for java.

massive SQL injection vulnerability scanner

Scraping scripts for various websites.

Explore a website recursively and download all the wanted documents (PDF, ODT…)

A Tumblr Blog Backup Application

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

python requests + Django+ nodejs koa+ mysql to crawl eastmoney fund and stock data,for data analysis and visualiaztion .

A Tumblr Backup Application

A scalable web crawler framework for Java.

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

Life is short, *****.

A lite distributed Java spider framework :-)

练习NLP，分析淘宝评论的项目

A web crawler (for bug hunting) that gathers more than you can imagine.

🐤️ Lost Ark wait notifier

A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.

Universal scrapping tool, which allows you to extract data using multiple environments

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

爬虫, http代理, 模拟登陆!

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.