756 Open source projects that are alternatives of or similar to Supercrawler

Continuous scalable web crawler built on top of Flink and crawler-commons

A simple but powerful web crawler library for .NET

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

简单易用的Python爬虫框架，QQ交流群：597510560

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

A collection of awesome web crawler,spider in different languages

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

The simple, easy to use command line web crawler.

Creates an XML-Sitemap by crawling a given site.

Script that generates a sitemap by crawling a given URL

A simple distributed crawler for zhihu && data analysis

Easily create XML sitemaps for your website.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Rust Web Crawler saving pages on Redis

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

📷 Automate full website screenshots and PDF generation with multiple viewport support.

Easy way to brute-force web directory.

Script to fetch webcomics and use them to create ebooks.

A fast tool to fetch URLs from HTML attributes by crawl-in.

Recursive and multi-threaded broken link checker

This package provides a CLF-based reactive planning system, described in paper: Efficient Anytime CLF Reactive Planning System for a Bipedal Robot on Undulating Terrain. The reactive planning system consists of a 5-Hz planning thread to guide a robot to a distant goal and a 300-Hz Control-Lyapunov-Function-based (CLF-based) reactive thread to co…

Library for generating Google sitemap XML files

An R web crawler and scraper

This repository contains all the code and files needed to simulate the notspot quadrupedal robot using Gazebo and ROS.

A tool to find redirection chains in multiple URLs

Pa11y CI is a CI-centric accessibility test runner, built using Pa11y

模拟登录各类网站，操作 API 完成各种不可描述的事情

build you own internet of robots!

🔌 Generate a highly customizable sitemap XML in Strapi CMS

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

A symfony bundle that provides tools to build a rich application sitemap. The main goals are : simple, no databases, various namespace (eg. google image), respect constraints etc.

我的爬虫合集

Award-Winning FRC Library by StuyPulse Team 694

Simple Weibo Scraper

一个灵活、友好的爬虫框架

Dingtalk for jenkins

python requests + Django+ nodejs koa+ mysql to crawl eastmoney fund and stock data,for data analysis and visualiaztion .

A sitemap (sitemap.xml) querying and parsing library for .NET

Web Application Security Scanner Framework

Just a crawler based on tg-cli for Telegram. Deprecated by now, please use telegram-export.

🇰🇷 Realtime integrated information analysis service

The fastest dork scanner written in Go.

Crawl GitHub APIs and store the discovered orgs, repos, commits, ...

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

ROBOTIS Dynamixel SDK (Protocol1.0/2.0)

JL Sitemap - Component sitemap for Joomla

Awesome Webots

line-bot-tutorial use python flask

Simple kinematics calculation toolkit for robotics

Dataflow system for building self-driving car and robotics applications.

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

An Architecture for the Versatile Control of Legged Robots

读书笔记《自己动手写网络爬虫》，自己敲的代码。主要记录了网络爬虫的基本实现，网页去重的算法，网页指纹算法，文本信息挖掘

基于MiraiGo的客户端，使用反向 websocket 收发私聊、群聊消息，消息格式类似onebot。支持多账号，很稳定

磁力網站U3C3介紹以及域名更新

Source code for the IndieWeb search engine.