GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → minio → nifi-minio

minio / nifi-minio

Licence: Apache-2.0 license
A custom ContentRepository implementation for NiFi to persist data to MinIO Object Storage

Programming Languages

java
68154 projects - #9 most used programming language

Labels

nifinifi-processorsnifi-dockernifi-custom-processornifi-controller

Projects that are alternatives of or similar to nifi-minio

nifi-sqllookup-services-bundle
NIFI controllers for SQL record and attributes lookups with built-in caching
Stars: ✭ 18 (-33.33%)
Mutual labels:  nifi, nifi-processors
nifi-extracttext-processor
Apache NiFi Custom Processor Extracting Text From Files with Apache Tika
Stars: ✭ 31 (+14.81%)
Mutual labels:  nifi, nifi-processors
NiFi-Rule-engine-processor
Drools processor for Apache NiFi
Stars: ✭ 34 (+25.93%)
Mutual labels:  nifi, nifi-processors
nifi-minifi
Apache MiNiFi (a subproject of Apache NiFi)
Stars: ✭ 115 (+325.93%)
Mutual labels:  nifi
MonitoFi
MonitoFi: Health & Performance Monitor for your Apache NiFi
Stars: ✭ 40 (+48.15%)
Mutual labels:  nifi
nifi-prometheus-reporter
A reporting task in Nifi which is capable of sending monitoring statistics as prometheus metrics to a prometheus pushgateway.
Stars: ✭ 48 (+77.78%)
Mutual labels:  nifi
nifikop
The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
Stars: ✭ 122 (+351.85%)
Mutual labels:  nifi
kubernetes-nifi-cluster
Apache Nifi cluster running in kubernetes
Stars: ✭ 81 (+200%)
Mutual labels:  nifi
CogStack-NiFi
Building data processing pipelines for documents processing with NLP using Apache NiFi and related services
Stars: ✭ 22 (-18.52%)
Mutual labels:  nifi
nifi
Deploy a secured, clustered, auto-scaling NiFi service in AWS.
Stars: ✭ 37 (+37.04%)
Mutual labels:  nifi
nifi
Helm nifi chart
Stars: ✭ 29 (+7.41%)
Mutual labels:  nifi
nifi-influxdb-bundle
InfluxDB Processors For Apache NiFi
Stars: ✭ 30 (+11.11%)
Mutual labels:  nifi
nifi-stateless-operator
An Operator for scheduling and executing NiFi Flows as Jobs on Kubernetes
Stars: ✭ 52 (+92.59%)
Mutual labels:  nifi
Nifi
Apache NiFi
Stars: ✭ 2,931 (+10755.56%)
Mutual labels:  nifi
helm-nifi
Helm Chart for Apache Nifi
Stars: ✭ 124 (+359.26%)
Mutual labels:  nifi
terraform-provider-nifi
Terraform provider for interacting with NiFi cluster
Stars: ✭ 49 (+81.48%)
Mutual labels:  nifi
nifi-sandbox
Sandbox for Apache nifi
Stars: ✭ 21 (-22.22%)
Mutual labels:  nifi
daf-kylo
Kylo integration with PDND (previously DAF).
Stars: ✭ 20 (-25.93%)
Mutual labels:  nifi
NiFi4Trading
NiFi Bundle for FIX Protocol
Stars: ✭ 14 (-48.15%)
Mutual labels:  nifi
nifi-fds
Mirror of Apache NiFi Flow Design System
Stars: ✭ 25 (-7.41%)
Mutual labels:  nifi

nifi-minio

A custom ContentRepository implementation for NiFi to persist data to MinIO Object Storage

Build

mvn clean; mvn package;
cp nifi-minio-nar/target/nifi-minio-nar-1.0.0.nar $NIFI_HOME/lib

Configure Apache NiFi

Add following entries in $NIFI_HOME/conf/nifi.properties

...
# Content Repository
nifi.content.repository.implementation=io.minio.nifi.MinIORepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=1000
# S3 specific settings
nifi.content.repository.s3_endpoint=s3://play.min.io/
nifi.content.repository.s3_access_key=Q3AM3UQ867SPQQA43P2F
nifi.content.repository.s3_secret_key=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
nifi.content.repository.s3_ssl_enabled=true
nifi.content.repository.s3_path_style_access=true
nifi.content.repository.s3_cache=10000
nifi.content.repository.directory.default=/nifiminio
nifi.content.repository.archive.enabled=true
nifi.content.viewer.url=../nifi-content-viewer/
...

Start Apache NiFi

$NIFI_HOME/nifi.sh run
...
2019-07-12 07:00:13,353 INFO [main] org.apache.nifi.web.server.JettyServer http://72.28.97.54:8080/nifi
2019-07-12 07:00:13,354 INFO [main] org.apache.nifi.web.server.JettyServer http://127.0.0.1:8080/nifi
2019-07-12 07:00:13,355 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2019-07-12 07:00:13,356 INFO [main] org.apache.nifi.NiFi Controller initialization took 15132490080 nanoseconds (15 seconds).

Now visit http://localhost:8080/nifi to start configuring data flows.

TODO

  • Implement NiFiProperties.CONTENT_ARCHIVE_MAX_RETENTION_PERIOD with bucket lifecycle policies.
  • Implement NiFiProperties.CONTENT_ARCHIVE_MAX_USAGE_PERCENTAGE with bucket lifecycle policies.
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].