freeCodeCamp

5.3 KiB

Raw Blame History

title
Streams

Streams

Streams are available in Node.js core API as objects that allow the data to read or write in a continuous way. Basically, a stream does that in chunks in comparison to buffer which does its bit by bit, thus making it a slow process.

There are four types of streams available:

Readable (streams from which data is read)
Writable (streams to which data is written)
Duplex (streams that are both Readable and Writable)
Transform (Duplex Streams that can modify data as it is read and written)

Each available type has several methods associated. Some of the common ones are:

data (this runs when data is available)
end (this gets triggered when there is no data left to read)
error (this runs when there is an error either receiving or writing data)

Pipe

In programming, the concept of pipe is not new. Unix based systems have been pragmatically using it since the 1970s. What does a Pipe do? A pipe generally connects the source and the destination. It passes the output of one function as the input of another function.

In Node.js, pipe is used the same way, to pair inputs and outputs of different operations. pipe() is available as a function that takes a readable source stream and attaches the output to a destination stream. The general syntax can be represented as:

src.pipe(dest);

Multiple pipe() functions can also be chained together.

a.pipe(b).pipe(c);

// which is equivalent to

a.pipe(b);
b.pipe(c);

Readable Streams

Streams that produce data that can be attached as the input to a writable stream is known as a Readable stream. To create a readable stream:

const { Readable } = require('stream');

const readable = new Readable();

readable.on('data', chunk => {
  console.log(`Received ${chunk.length} bytes of data.`);
});
readable.on('end', () => {
  console.log('There will be no more data.');
});

Writable Stream

This is the type of a stream that you can pipe() the data to from a readable source. To create a writable stream, we have a constructor approach. We create an object from it and pass a number of options. The method takes three arguments:

chunk: a buffer
encoding: to convert data to human readable form
callback: a function that is called when the data is done processing from the chunk

const { Writable } = require('stream');
const writable = new Writable({
  write(chunk, encoding, callback) {
    console.log(chunk.toString());
    callback();
  }
});

process.stdin.pipe(writable);

Duplex Streams

Duplex streams help us to implement both readable and writable streams at the same time.

const { Duplex } = require('stream');

const inoutStream = new Duplex({
  write(chunk, encoding, callback) {
    console.log(chunk.toString());
    callback();
  },

  read(size) {
    this.push(String.fromCharCode(this.currentCharCode++));
    if (this.currentCharCode > 90) {
      this.push(null);
    }
  }
});

inoutStream.currentCharCode = 65;
process.stdin.pipe(inoutStream).pipe(process.stdout);

The stdin stream pipes the readable data into the duplex stream. The stdout helps us to see the data. The readable and writable parts of a duplex stream operate completely independent of each other.

Transform Stream

This type of stream is more of an advanced version of the duplex stream.

const { Transform } = require('stream');

const upperCaseTr = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

process.stdin.pipe(upperCaseTr).pipe(process.stdout);

The data we are consuming is same as the previous example of the duplex stream. The thing to notice here is that transform() does not require implementation of read or write methods. It combines both the methods itself.

Why use Streams?

Since Node.js is asynchronous so interacting by passing callbacks to functions with disk and network. An example given below reads the data from a file on the disk and responds it to over the network request from client.

const http = require('http');
const fs = require('fs');

const server = http.createServer((req, res) => {
  fs.readFile('data.txt', (err, data) => {
    res.end(data);
  });
});
server.listen(8000);

The above snippet of code will work but the entire data from the file will first go into the memory for every request before writing the result back to the client request. If the file we are reading is too large, this can become a very heavy and expensive server call as it will consume a lot of memory for the process to advance. The user experience on the client side will also suffer from delay.

In this case, if we use streams, the data will be send to the client request as one chunk at a time as soon as they received from the disk.

const http = require('http');
const fs = require('fs');

const server = http.createServer((req, res) => {
  const stream = fs.createReadStream('data.txt');
  stream.pipe(res);
});
server.listen(8000);

The pipe() here takes care of writing or in our case, sending the data with response object and once all the data is read from the file, to close the connection.

Note: process.stdin and process.stdout are build in streams in the global process object provided by Node.js API.

5.3 KiB Raw Blame History