1051 Open source projects that are alternatives of or similar to Sentinel Crawler

Golang爬虫 爬取豆瓣电影Top250

Secret and/ credential patterns used for gf.

Fawkes is a tool to search for targets vulnerable to SQL Injection. Performs the search using Google search engine.

Use Pentaho's open source data integration tool (Kettle) to create Extract-Transform-Load (ETL) processes to update a Socrata open data portal. Documentation is available at http://open-data-etl-utility-kit.readthedocs.io/en/stable

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

利用文本挖掘技术进行新闻热点关注问题分析

Scrapy, a fast high-level web crawling & scraping framework for Python.

爬虫代理IP池服务，可供其他爬虫程序通过restapi获取

The MarkLogic Data Hub: documentation ==>

Profile what you care, monitor how it goes (support C/C++)

A scalable web crawler framework for Java.

VIS components for a security system, to monitor the state and confirm the system's health running.

A Moodle downloader that downloads course content fast from Moodle (eg. lecture pdfs)

A system monitor written in Rust and C

A web crawler (for bug hunting) that gathers more than you can imagine.

新浪微博相册大图多线程爬虫。

一只优雅的正方教务系统爬虫。

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

tumblr解析网站

DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server

An online tool (crawler) to analyze users performance in online judges (coding competition websites). Supported OJ: POJ, HDU, ZOJ, HYSBZ, CodeForces, UVA, ICPC Live Archive, FZU, SPOJ, Timus (URAL), LeetCode_CN, CSU, LibreOJ, 洛谷, 牛客OJ, Lutece (UESTC), AtCoder, AIZU, CodeChef, El Judge, BNUOJ, Codewars, UOJ, NBUT, 51Nod, DMOJ, VJudge

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Verify that a request is from Google crawlers using Google's DNS verification steps

Don's blog的后台管理系统，基于Vue.js + Node.js + Koa.js + MySQL + Redis + WebSocket。

Easily disable or enable a monitor on your Mac.

此项目当前不再更新，只适用于3.0和3.2版本

Production ready Koa2 boilerplate.

最简單直接的 Koa2 的脚手架

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

vue2聊天室，图灵机器人，node爬虫

a puppeteer walker 🕷 🕸

Monitors off---white.com for new product releases and restocks

Python: An all-in-one Web Crawler, Web Parser and Web Scrapping library!

equivalent to kafka-connect 🔧 for nodejs ✨🐢🚀✨

It covers the blockade principle of most anti-climbing strategies and corresponding solutions.👽👽👽👽（涵盖了大部分的反爬策略的封锁原理以及对应的解决方案。）

Viewers for statistics and dashboarding of Domain Search Engine data

A visual process builder for Laravel

聚合koa2+pm2+vue-cli+element+axios的前后端一体开发脚手架

Some classic web crawler projects.一些经典的爬虫

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

locopy: Loading/Unloading to Redshift and Snowflake using Python.

Enforce HTTPS in node.js koa apps

A luigi powered analytics / warehouse stack

APIs for loginning some websites by using requests.

Python爬虫，京东自动登录，在线抢购商品

Free system monitor for OSX and macOS. See all system information at a glance in the menu bar.

Some scrapy and web.py exmaples

💻 Quickly crawl the information (e.g. followers, tags, etc...) of an instagram profile. No login required!

python crawler spider

A Singer.io Target for Postgres

🎬 电影资源爬虫,电影图片抓取脚本,Flask|Nginx|wsgi

抖音 SDK，数据采集，爬虫抓取不是梦

Simple Koa Boilerplate for APIs

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

GraphQL is a new way of communicating with your server. It eliminates the problems of over- and under-fetching, incorporates strong data types, has built-in introspection, documentation and deprecation capabilities, and is implemented in many programming languages. This all leads to gloriously low-latency user experiences, better developer experiences, and much increased productivity. Because of all this, GraphQL is typically used as a replacement for (or companion to) RESTful API services.

scrapy专利爬虫（停止维护）

Cross-platform persistent and distributed web crawler 🔗

A kernel network manager with monitoring and limiting capabilities for macOS. #nsacyber