big-data-europe / Docker Hadoop Spark Workbench
[EXPERIMENTAL] This repo includes deployment instructions for running HDFS/Spark inside docker containers. Also includes spark-notebook and HDFS FileBrowser.
Stars: ✭ 536
Labels
Projects that are alternatives of or similar to Docker Hadoop Spark Workbench
Mask
🎭 A CLI task runner defined by a simple markdown file
Stars: ✭ 495 (-7.65%)
Mutual labels: makefile
Photoprocessing
A demo of how you can process photos leveraging the ndk.
Stars: ✭ 513 (-4.29%)
Mutual labels: makefile
K8s Prom Hpa
Kubernetes Horizontal Pod Autoscaler with Prometheus custom metrics
Stars: ✭ 474 (-11.57%)
Mutual labels: makefile
Awesome Asyncio Cn
😎 Python Asyncio 精选资源列表,囊括了网络框架,库,软件等资源
Stars: ✭ 501 (-6.53%)
Mutual labels: makefile
Sbt Pack
A sbt plugin for creating distributable Scala packages.
Stars: ✭ 438 (-18.28%)
Mutual labels: makefile
Envoy
Envoy proxy中文文档 - https://www.servicemesher.com/envoy/
Stars: ✭ 509 (-5.04%)
Mutual labels: makefile
Lets Split Guide
This guide covers building a Let's Split v2
Stars: ✭ 501 (-6.53%)
Mutual labels: makefile
Androideagleeye
An Xposed and adbi based module which is capable of hooking both Java and Native methods targeting Android OS.
Stars: ✭ 454 (-15.3%)
Mutual labels: makefile
Kubernetes Handbook
Kubernetes Handbook (Kubernetes指南) https://kubernetes.feisky.xyz
Stars: ✭ 4,511 (+741.6%)
Mutual labels: makefile
Python Regex Cheatsheet
Python 2.7 Regular Expression cheatsheet, as a restructured text document and Makefile to convert it to PDF
Stars: ✭ 496 (-7.46%)
Mutual labels: makefile
Pinn
An enhanced Operating System installer for the Raspberry Pi
Stars: ✭ 530 (-1.12%)
Mutual labels: makefile
Why Linux Is Better
Objective reasons to prefer Linux to Windows.
Stars: ✭ 518 (-3.36%)
Mutual labels: makefile
Bare Arduino Project
Start your Arduino projects right out of the box
Stars: ✭ 505 (-5.78%)
Mutual labels: makefile
How to use HDFS/Spark Workbench
To start an HDFS/Spark Workbench:
docker-compose up -d
docker-compose does not work to scale up spark-workers, for distributed setup see swarm folder
Starting workbench with Hive support
Before starting the next command, check that the previous service is running correctly (with docker logs servicename).
docker-compose -f docker-compose-hive.yml up -d namenode hive-metastore-postgresql
docker-compose -f docker-compose-hive.yml up -d datanode hive-metastore
docker-compose -f docker-compose-hive.yml up -d hive-server
docker-compose -f docker-compose-hive.yml up -d spark-master spark-worker spark-notebook hue
Interfaces
- Namenode: http://localhost:50070
- Datanode: http://localhost:50075
- Spark-master: http://localhost:8080
- Spark-notebook: http://localhost:9001
- Hue (HDFS Filebrowser): http://localhost:8088/home
Important
When opening Hue, you might encounter NoReverseMatch: u'about' is not a registered namespace
error after login. I disabled 'about' page (which is default one), because it caused docker container to hang. To access Hue when you have such an error, you need to append /home to your URI: http://docker-host-ip:8088/home
Docs
Count Example for Spark Notebooks
val spark = SparkSession
.builder()
.appName("Simple Count Example")
.getOrCreate()
val tf = spark.read.textFile("/data.csv")
tf.count()
Maintainer
- Ivan Ermilov @earthquakesan
Note: this repository was a part of BDE H2020 EU project and no longer actively maintained by the project participants.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].