580 Open source projects that are alternatives of or similar to Nutch

Mapreduce project by Hadoop, Nutch, AWS EMR, Pig, Tez, Hive

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

Apache Hive

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

SQL backend to dplyr for Impala

Hive Storage Handler for interoperability between BigQuery and Apache Hive

Continuous scalable web crawler built on top of Flink and crawler-commons

Fundamentals of Spark with Python (using PySpark), code examples

와이빅타 엔지니어링팀의 자료를 정리해두는 곳입니다.

Export Hadoop YARN (resource-manager) metrics in prometheus format

The simple, easy to use command line web crawler.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

OLAP and ETL of Big Data

An alternative to the "hive standalone" jar for connecting Java applications to Apache Hive via JDBC

Apache Tez

Hive JDBC "uber" or "standalone" jar based on the latest Apache Hive version

OWASP Mth3l3m3nt Framework is a penetration testing aiding tool and exploitation framework. It fosters a principle of attack the web using the web as well as pentest on the go through its responsive interface.

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Big Data Toolkit for the JVM

AI on Hadoop

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

✂A collection of useful .htaccess snippets.

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Extended datasource support for Spark/Hadoop on Aliyun E-MapReduce.

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Norconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.

Download a large list of files concurrently

Apache Geode

Enabling Continuous Data Processing with Apache Spark and Azure Event Hubs

Prometheus exporter for Apache.

An Instagram bot developed using the Selenium Framework

The official home of the Presto distributed SQL query engine for big data

HBase客户端数据管理软件

네이버 뉴스 수집을 위한 도구

🐍💻📊 All material from the PyCon.DE 2018 Talk "Beyond Jupyter Notebooks - Building your own data science platform with Python & Docker" (incl. Slides, Video, Udemy MOOC & other References)

📅🇨🇳 中国法定节假日数据 自动每日抓取国务院公告

An Apache httpd module for integrating with Apereo CAS Server project.

Convert .htaccess redirects to nginx.conf redirects

Mirror of Apache Hadoop common

Mirror of Apache Calcite - Avatica

A large-scale entity and relation database supporting aggregation of properties

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learni…

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

Automate the installation of Let's Encrypt SSL on the free plan of ServerPilot

Ephemeral Hadoop clusters using Google Compute Platform

A client library for Brazilian Correios APIs and services (SIGEP & SRO).

Griffon Data Science Virtual Machine

6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

Mirror of Apache Qpid Proton

Mirror of Apache Hadoop HDFS

Laravel based usenet indexer

Crawler for linguistic corpora

Hadoop Crypto Ledger - Analyzing CryptoLedgers, such as Bitcoin Blockchain, on Big Data platforms, such as Hadoop/Spark/Flink/Hive

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

Apache Guacamole installation bash script for RHEL 7 and CentOS 7 including options for Nginx, HTTPS, SSL, LDAP, Let's Encrypt certificates and more

Spark、Flink等离线任务的调度以及实时任务的监控

A collection of awesome web scaper, crawler.