damiencarol / Jsr203 Hadoop
Licence: apache-2.0
A Java NIO file system provider for HDFS
Stars: ✭ 35
Programming Languages
java
68154 projects - #9 most used programming language
Projects that are alternatives of or similar to Jsr203 Hadoop
py-hdfs-mount
Mount HDFS with fuse, works with kerberos!
Stars: ✭ 13 (-62.86%)
Mutual labels: hadoop, hdfs
leaflet heatmap
简单的可视化湖州通话数据 假设数据量很大，没法用浏览器直接绘制热力图，把绘制热力图这一步骤放到线下计算分析。使用Apache Spark并行计算数据之后，再使用Apache Spark绘制热力图，然后用leafletjs加载OpenStreetMap图层和热力图图层，以达到良好的交互效果。现在使用Apache Spark实现绘制，可能是Apache Spark不擅长这方面的计算或者是我没有设计好算法，并行计算的速度比不上单机计算。Apache Spark绘制热力图和计算代码在这 https://github.com/yuanzhaokang/ParallelizeHeatmap.git .
Stars: ✭ 13 (-62.86%)
Mutual labels: hadoop, hdfs
fastdata-cluster
Fast Data Cluster (Apache Cassandra, Kafka, Spark, Flink, YARN and HDFS with Vagrant and VirtualBox)
Stars: ✭ 20 (-42.86%)
Mutual labels: hadoop, hdfs
God Of Bigdata
专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
Stars: ✭ 6,008 (+17065.71%)
Mutual labels: hadoop, hdfs
fsbrowser
Fast desktop client for Hadoop Distributed File System
Stars: ✭ 27 (-22.86%)
Mutual labels: hadoop, hdfs
fluent-plugin-webhdfs
Hadoop WebHDFS output plugin for Fluentd
Stars: ✭ 57 (+62.86%)
Mutual labels: hadoop, hdfs
Devops Python Tools
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Function, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Stars: ✭ 406 (+1060%)
Mutual labels: hadoop, hdfs
Hadoop For Geoevent
ArcGIS GeoEvent Server sample Hadoop connector for storing GeoEvents in HDFS.
Stars: ✭ 5 (-85.71%)
Mutual labels: hadoop, hdfs
wasp
WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.
Stars: ✭ 19 (-45.71%)
Mutual labels: hadoop, hdfs
ros hadoop
Hadoop splittable InputFormat for ROS. Process rosbag with Hadoop Spark and other HDFS compatible systems.
Stars: ✭ 92 (+162.86%)
Mutual labels: hadoop, hdfs
skein
A tool and library for easily deploying applications on Apache YARN
Stars: ✭ 128 (+265.71%)
Mutual labels: hadoop, hdfs
bigdata-fun
A complete (distributed) BigData stack, running in containers
Stars: ✭ 14 (-60%)
Mutual labels: hadoop, hdfs
jsr203-hadoop
JSR 203 Implementation for Hadoop Distributed File System
Maven artifact
<dependency>
<groupId>com.github.jsr203hadoop</groupId>
<artifactId>jsr203hadoop</artifactId>
<version>1.0.1</version>
</dependency>
Prerequisites
jsr203-hadoop requires git, maven (3.0.4 or later), and JDK 1.7 or later.
Download and build
$ git clone git://github.com/damiencarol/jsr203-hadoop.git
$ cd jsr203-hadoop
$ mvn install
Use mvn -DskipTests if you do not want to execute the tests.
Example
jsr203-hadoop provide NIO 2 access to your HDFS cluster. For
example, you can build URI starts with hdfs://<namebode>:<port> and use standard NIO API :
URI uri = new URI("hdfs://" + host + ":" + port + "/somefile");
Path file = Paths.get(uri);
System.out.println("File [" + file.toString() + "] exists = '" + Files.exists(file) + "'");
Status
This project is still a beta.
The following features are complete.
| Feature | Status | Description |
|---|---|---|
| Path support - build from string/URI | Done | |
| Path support - resolve, relativize, equals, ... | Done | |
| Directory stream (provide ability to list files in directory, get status, ...) | Done | |
| Create/delete file/directory | Done | |
| Input/Output | Partial | Full support of SeekableByteChannel, FileChannel pending |
| Attribute view "basic" | Done | Full support (fileKey is implemented) |
| Attribute view "owner" | Done | Full support (Use Hadoop UGI) |
| Attribute view "posix" | Done | |
| Hadoop dedicated view "hadoop" for hadoop specific attributes (len, block size, replication factor) | Done | More attributes will come |
More features will come :
- Security and permissions
- complete list of hadoop specific attributes
For more details, see the Reference guide.
More information
- License: Apache License, Version 2.0.
- Author: Damien Carol
- Source code: http://github.com/damiencarol/jsr203-hadoop
Ressources
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].