All Projects → harrisiirak → webhdfs

harrisiirak / webhdfs

Licence: MIT license
Node.js WebHDFS REST API client

Programming Languages

javascript
184084 projects - #8 most used programming language

Projects that are alternatives of or similar to webhdfs

BigInsights-on-Apache-Hadoop
Example projects for 'BigInsights for Apache Hadoop' on IBM Bluemix
Stars: ✭ 21 (-76.14%)
Mutual labels:  hadoop, webhdfs
teraslice
Scalable data processing pipelines in JavaScript
Stars: ✭ 48 (-45.45%)
Mutual labels:  hadoop
Hadoop Connectors
Libraries and tools for interoperability between Hadoop-related open-source software and Google Cloud Platform.
Stars: ✭ 218 (+147.73%)
Mutual labels:  hadoop
ambari-hdp-docker
Dockerfiles and Docker Compose for HDP 2.6 with Blueprints
Stars: ✭ 23 (-73.86%)
Mutual labels:  hadoop
Hadoop Attack Library
A collection of pentest tools and resources targeting Hadoop environments
Stars: ✭ 228 (+159.09%)
Mutual labels:  hadoop
hadoop-ansible
Install hadoop cluster with ansible
Stars: ✭ 35 (-60.23%)
Mutual labels:  hadoop
Calcite
Apache Calcite
Stars: ✭ 2,816 (+3100%)
Mutual labels:  hadoop
TonY
TonY is a framework to natively run deep learning frameworks on Apache Hadoop.
Stars: ✭ 687 (+680.68%)
Mutual labels:  hadoop
JavaFramework
Simple Java Framework,designed for easily develop Spring based java program.Support Bigdata And metadata management.A common elasticsearch comm query tool and so on.
Stars: ✭ 16 (-81.82%)
Mutual labels:  hadoop
kafka-connect-fs
Kafka Connect FileSystem Connector
Stars: ✭ 107 (+21.59%)
Mutual labels:  hadoop
phoenix
Apache Phoenix / Hbase Spring Boot Microservices
Stars: ✭ 23 (-73.86%)
Mutual labels:  hadoop
Devops Bash Tools
550+ DevOps Bash Scripts - AWS, GCP, Kubernetes, Kafka, Docker, APIs, Hadoop, SQL, PostgreSQL, MySQL, Hive, Impala, Travis CI, Jenkins, Concourse, GitHub, GitLab, BitBucket, Azure DevOps, TeamCity, Spotify, MP3, LDAP, Code/Build Linting, pkg mgmt for Linux, Mac, Python, Perl, Ruby, NodeJS, Golang, Advanced dotfiles: .bashrc, .vimrc, .gitconfig, .screenrc, .tmux.conf, .psqlrc ...
Stars: ✭ 226 (+156.82%)
Mutual labels:  hadoop
orion
Management and automation platform for Stateful Distributed Systems
Stars: ✭ 77 (-12.5%)
Mutual labels:  hadoop
Luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Stars: ✭ 15,226 (+17202.27%)
Mutual labels:  hadoop
gomrjob
gomrjob - a Go Framework for Hadoop Map Reduce Jobs
Stars: ✭ 39 (-55.68%)
Mutual labels:  hadoop
Sparkrdma
RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark
Stars: ✭ 215 (+144.32%)
Mutual labels:  hadoop
bigdatatutorial
bigdatatutorial
Stars: ✭ 34 (-61.36%)
Mutual labels:  hadoop
RecommendationEngine
Source code and dataset for paper "CBMR: An optimized MapReduce for item‐based collaborative filtering recommendation algorithm with empirical analysis"
Stars: ✭ 43 (-51.14%)
Mutual labels:  hadoop
dpkb
大数据相关内容汇总,包括分布式存储引擎、分布式计算引擎、数仓建设等。关键词:Hadoop、HBase、ES、Kudu、Hive、Presto、Spark、Flink、Kylin、ClickHouse
Stars: ✭ 123 (+39.77%)
Mutual labels:  hadoop
yarn-prometheus-exporter
Export Hadoop YARN (resource-manager) metrics in prometheus format
Stars: ✭ 44 (-50%)
Mutual labels:  hadoop

node-webhdfs

Build Status NPM version

Hadoop WebHDFS REST API (2.2.0) client library for node.js with fs module like (asynchronous) interface.

Examples

Writing to the remote file:

var WebHDFS = require('webhdfs');
var hdfs = WebHDFS.createClient();

var localFileStream = fs.createReadStream('/path/to/local/file');
var remoteFileStream = hdfs.createWriteStream('/path/to/remote/file');

localFileStream.pipe(remoteFileStream);

remoteFileStream.on('error', function onError (err) {
  // Do something with the error
});

remoteFileStream.on('finish', function onFinish () {
  // Upload is done
});

Reading from the remote file:

var WebHDFS = require('webhdfs');
var hdfs = WebHDFS.createClient({
  user: 'webuser',
  host: 'localhost',
  port: 80,
  path: '/webhdfs/v1'
});

var remoteFileStream = hdfs.createReadStream('/path/to/remote/file');

remoteFileStream.on('error', function onError (err) {
  // Do something with the error
});

remoteFileStream.on('data', function onChunk (chunk) {
  // Do something with the data chunk
});

remoteFileStream.on('finish', function onFinish () {
  // Upload is done
});

TODO

  • Implement all calls (GETCONTENTSUMMARY, GETFILECHECKSUM, GETHOMEDIRECTORY, GETDELEGATIONTOKEN, RENEWDELEGATIONTOKEN, CANCELDELEGATIONTOKEN, SETREPLICATION, SETTIMES)
  • Improve documentation
  • More examples

Tests

Running tests:

npm test

License

MIT

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].