All Projects → davidgatti → How To Understand Streams In Nodejs

davidgatti / How To Understand Streams In Nodejs

Licence: mit
🚰 A step by step explanation how to take advantage of Streams in NodeJS

Programming Languages

javascript
184084 projects - #8 most used programming language

Projects that are alternatives of or similar to How To Understand Streams In Nodejs

How To Stream Movies Using Nodejs
🎥 How to Stream Movies to a HTML 5 video tag using NodeJS
Stars: ✭ 336 (+216.98%)
Mutual labels:  stream, article
Transport Eta
Twitch streamed 🎥playground repo, README speaks to you.
Stars: ✭ 223 (+110.38%)
Mutual labels:  stream, article
Advanced Php
最近打算写一些php一些偏微妙的教程,比如关于多进程、socket等相关,都是自己的一些感悟心得
Stars: ✭ 1,271 (+1099.06%)
Mutual labels:  stream
Ios Learning Materials
📚Curated list of articles, web-resources, tutorials and code repositories that may help you dig a little bit deeper into iOS [and Apple Platforms].
Stars: ✭ 1,380 (+1201.89%)
Mutual labels:  article
Node Lei Stream
Read/Write stream line by line 按行读写流
Stars: ✭ 94 (-11.32%)
Mutual labels:  stream
Pinbox
PinBox is a homebrew for 3DS system to stream content from a windows PC to 3DS.
Stars: ✭ 88 (-16.98%)
Mutual labels:  stream
Rtmp Rtsp Stream Client Java
Library to stream in rtmp and rtsp for Android. All code in Java
Stars: ✭ 1,338 (+1162.26%)
Mutual labels:  stream
Cloud Media Scripts
Upload and stream media from the cloud with or without encryption. Cache all new and recently streamed media locally to access quickly and reduce API calls
Stars: ✭ 84 (-20.75%)
Mutual labels:  stream
Rxviz
Rx Visualizer - Animated playground for Rx Observables
Stars: ✭ 1,471 (+1287.74%)
Mutual labels:  stream
Getnews.tech
A web server that fetches data from the News API and formats it for display in the terminal.
Stars: ✭ 94 (-11.32%)
Mutual labels:  article
Nginx Vod Module
NGINX-based MP4 Repackager
Stars: ✭ 1,378 (+1200%)
Mutual labels:  stream
Hyperapp Render
Render Hyperapp to an HTML string with SSR and Node.js streaming support.
Stars: ✭ 93 (-12.26%)
Mutual labels:  stream
Backoffice Administration
Stars: ✭ 89 (-16.04%)
Mutual labels:  stream
Csv Stream
📃 Streaming CSV Parser for Node. Small and made entirely out of streams.
Stars: ✭ 98 (-7.55%)
Mutual labels:  stream
Phpstreams
A streams library for PHP inspired by the Java 8 Streams API.
Stars: ✭ 87 (-17.92%)
Mutual labels:  stream
Emberconf 2017
A collection of links that summarize EmberConf 2017
Stars: ✭ 103 (-2.83%)
Mutual labels:  stream
Iostreams
IOStreams is an incredibly powerful streaming library that makes changes to file formats, compression, encryption, or storage mechanism transparent to the application.
Stars: ✭ 84 (-20.75%)
Mutual labels:  stream
Good Articles By Sort
本仓库用来存放我看过的认为比较好的文章---根据分类排序
Stars: ✭ 93 (-12.26%)
Mutual labels:  article
Gophie
An Aggregator Engine for searching and downloading movies free - NO ADs!
Stars: ✭ 94 (-11.32%)
Mutual labels:  stream
Illuminati
This is a Platform that collects all the data accuring in your Application and shows the data in real time by using Kibana or other tools.
Stars: ✭ 106 (+0%)
Mutual labels:  stream

Understanding Streams in NodeJS

This is another installment in my series of articles where I try to demystify big words that are used in computer programming. And the next big word that I want to tackle is the Streams concept.

What is a Stream?

Before we move on to some code, let's answer the basic questions:

What is a Stream? A stream is an "infinite" flow of data that is sent in small chunks - period. For example, it's the opposite of an array, which will have a predefined size. You can add new elements to an array, but you can always ask how many items there are. With a stream, you don't know when the data will stop flowing - in a network environment, that is. In C, for example, you could use .fseek() to find the total length of a open file. But in NodeJS, this is not the case - at least, I haven't discovered a method to check the file size if the file was opened as a Stream.

I'm reading a file as a stream, so I know the size - right?

True, a file can be read as a stream or load it in to memory, but the point is that if you open it as a stream, there won't be a way to determine how big the file is.

A socket is purely a stream...You don't know how much data you are going to get; the data will be buffered by the system or network card until it reaches a point where the system will pass it to your code, so you can do something with it. The amount of data will depend on the network card, operating system, how fast the data is coming in, etc.

This means that you work with just a small subset of all of the information. It's important to understand that your data will be split into pieces, and it will be up to you to recombine it into something that makes sense for your situation.

For example, let's say that you want to display a full sentence: "I love the articles that David writes." But your code will get the sentence in the following way:

  1. I love the
  2. articles that Dav
  3. id writes.

Your job as a programmer will be to concatenate all of the data until you detect the . sign. And only then can you display the whole sentence (Read more about sockets here).

The Pipe concept

The Unix | pipe is nothing new. It was created in the 70s by Douglas McIlroy while he worked at Bell Labs. The job of a pipe is to get the output of one program and pass it as input to another one.

In NodeJS, we use Pipes inside the code to pass the result of one function to the next one. This is seen in Example 3, where we open a file, compress it, and save the result to a new file.

raw_file.pipe(gz).pipe(to_compressed_file);

Combining Pipes and Streams

By combining these two concepts, we can chain together chunks of code to manipulate the data in a very specific way, and pass it to the next piece of code.

The best use case of a Stream in NodeJS

Imagine that you have two hard drives, one with a 100GB file, the other with enough space to hold the output. Let's assume that the file is a log file, where we want to extract some useful data.

Loading 100GB into memory on your laptop would not be feasible, but we can solve the problem with Streams. Because instead of loading the whole file into memory, the system will load the log file in chunks. This allows your app to use a constant amount of RAM.

Basically, your laptop is just a proxy that manipulates the data and dumps the result in another place, thus making it possible to do work that would otherwise be impossible.

Code Break Down

Each folder in this repository contains a self-contained piece of code that just works. Take the time to read the README.md in each example, and don't forget to go over all of my comments. All of this in combination should give you a good understanding of what is going on.

The End

If you enjoyed this project, please consider giving it a 🌟. And check out my GitHub account, where you'll find additional resources you might find useful or interesting.

Sponsor 🎊

This project is brought to you by 0x4447 LLC, a software company specializing in building custom solutions on top of AWS. Follow this link to learn more: https://0x4447.com. Alternatively, send an email to [email protected].

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].