Data Ingestion Platform(DiP)

Check out the real time data ingestion using Data Ingestion Platform (DiP) which harness the powers of Apache Apex, Apache Flink, Apache Spark and Apache Storm to give real time data ingestion and visualization.

DiP comes along with a UI which allows to switch between multiple data streaming engines and combines them under one single platform.

DiP Features

• Multiple Sources
• Multiple File Formats
• Easy to use UI
• Data Visualization
• High Level API’s
• Java, Scala , Client bindings

DiP Technology Stack

• Source System – Web Client
• Messaging System – Apache Kafka
• Target System – HDFS, Apache HBase, Apache Hive
• Reporting System – Apache Phoenix, Apache Zeppelin
• Streaming API’s – Apache Apex, Apache Flink, Apache Spark and Apache Storm
• Programming Language – Java
• IDE – Eclipse
• Build tool – Apache Maven
• Operating System – CentOS 7

DiP Architecture

The DiP architecture has four blocks in the middle layer one for each streaming engine namely Apex Streaming, Flink Streaming, Spark Streaming and Storm Streaming respectively.

DiP comes with an easy to use UI that offers the following features –

• Switch easily between the supported streaming engines just by clicking on a radio button.
• Supports xml, json and tsv data formats
• Use text area to enter data manually for getting processed
• Process files for batch processing by simply uploading them

DiP on Apex

Apache Apex is an enterprise grade native YARN big data-in-motion platform that unifies stream processing as well as batch processing. It processes big data in-motion in a highly scalable, highly performant, fault tolerant, stateful, secure, distributed, and an easily operable way.

Blog link - https://techblog.xavient.com/real-time-data-ingestion-dip-apache-apex-co-dev-opportunity/ GitHub link - https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-apex

DiP on Flink

Apache Flink is an open source platform for distributed stream and batch data processing. Flink's core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.

Blog link- https://techblog.xavient.com/data-ingestion-platformdip-real-time-data-analysis-flink-streaming/ GitHub link - https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-flink

DiP on SparkStreaming

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams.

Blog link - https://techblog.xavient.com/real-time-data-ingestion-dip-spark-streaming-co-dev-opportunity/ GitHub link - https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-spark

DiP on Storm

Apache Storm is a free and open source distributed real time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!

Blog link - https://techblog.xavient.com/real-time-data-ingestion-easy-and-simple-co-dev-opportunity/ GitHub link - https://github.com/XavientInformationSystems/Data-Ingestion-Platform/tree/master/dataingest-storm

Credits Xavient

Technical team Neeraj Sabharwal Mohiuddin Khan Inamdar Gautam Marya Puneet Singh Sumit Chauhan