876 Open source projects that are alternatives of or similar to flink-crawler

JAW: A Graph-based Security Analysis Framework for JavaScript and Client-side CSRF

163music spider by scrapy.

Build a small, 3 domain internet using Github pages and Wikipedia and construct a crawler to crawl, render, and index.

github repo for MyAnimeList analysis. Also links to the MAL dataset.

Real-time Data Warehouse with Apache Flink & Apache Kafka & Apache Hudi

⭐ 图虫网爬虫

🕷️ A simple crawler (spider) writen in php just for fun, with zero dependencies

An AliExpress spider for Node

一些爬虫开发的技巧和案例

😚 Q & A website based on Spring Boot.

大数据学习笔记，学习路线，技术案例整理。

以Node.js基于express以及爬虫实现的视频资源后端

A web spider for shodan.io without using the Developer API.

TiDB connectors for Flink/Hive/Presto

FofaMap是一款基于Python3开发的跨平台FOFA数据采集器，支持网站图标查询、批量查询和自定义查询FOFA数据，能够根据查询结果自动去重并生成对应的Excel表格。另外春节特别版还可以调用Nuclei对目标进行漏洞扫描，让你在挖洞路上快人一步。

Get (almost) original messages from google group archives. Your data is yours.

动漫之家漫画站电脑版原图爬虫

A crawler based on Phantom. Allows discovery of dynamic content and supports custom scrapers.

Fetch, store and access user agent strings for different browsers

gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口测试&压力测试库，DB操作库等。

A bash script to spider a site, follow links, and fetch urls (with built-in filtering) into a generated text file.

基于 Selenium 的知乎关键词爬虫

大数据相关内容汇总，包括分布式存储引擎、分布式计算引擎、数仓建设等。关键词：Hadoop、HBase、ES、Kudu、Hive、Presto、Spark、Flink、Kylin、ClickHouse

抖音接口、抖音API、抖音数据爬虫、抖音直播数据、抖音直播Api、抖音视频Api、抖音爬虫、抖音去水印、抖音视频下载、抖音视频解析、抖音直播监控、抖音数据采集

妹子图全站采集10G套图资源

文章采集工具 Article collection tool

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户

利用墨墨背单词的分享功能拿每日20个的单词上限奖励（多线程

Compares price of the product entered by the user from e-commerce sites Amazon and Flipkart 💰 📊

Flink Forward China 2018 第一届记录，视频记录 | 文档记录 | 不仅仅是流计算 | More than streaming

A web spider framework

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

some small project and some articles

基于Electron爬虫程序

Real-time ETL developed by Flink, data from MySQL to Greenplum. Use canal to parse the MySQL binlog, put it into kafka, use Flink to consume kafka and assemble the data into Greenplum, and more data sources and target sources will be added in the future.

Go 抓取股票数据爬虫

Screen scraping and web crawling framework

一个基于Scrapy的数据采集爬虫代码库

银行、支行名称。中国各地区各银行支行名称数据爬虫，数据来源微信商户平台，已经整理可直接导入的sql文件

This taobao spider has been archived

Get Aliexpress product details in JSON

flink 流处理源码分析

Script that generates a sitemap by crawling a given URL

日常爬虫

千万级图片爬虫、视频爬虫 [开源版本] Image Spider

Polite, slim and concurrent web crawler.

Command Line Tool to download torrents

图片爬取下载工具，极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。

How to use Parquet in Flink

scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot

🌌 High productivity semi-automatic crawler generator 🛠️🧰

Apache Flink connectors for Pravega.

Grapy, a fast high-level web crawling framework for Python 3.3 or later base on asyncio.

A spiders' program for weixin which made by Express & cheerio

拟物校园，一个开源的高校教务移动化解决方案。

golang spider Crawler 爬虫 电影

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

The complete web scraping toolkit for PHP.

A lightweight crawling/spider framework for everyone(support JavaScript!).✨

Tutorial for web scraping / crawling with Node.js.