Streams

Streams are abstract interfaces for working with streaming data. They allow you to handle data in small chunks, making it possible to process large files or data sources efficiently. Streams are the most important APIs in Node, and knowing how to handle them is essential to building efficient servers.

Node supports four basic stream types:

  1. Readable streams from which data can be read.
  2. Writable streams to which data can be written.
  3. Duplex streams that are both Readable and Writable.
  4. Transform streams are Duplex streams that can modify data as it is written and read.

All streams are instances of EventEmitter. Bu default, streams read and write Buffer objects.

Internal buffers

Readable streams read data from a source, while Writable streams write data to a destination. Consequently, every stream has two ends: an input (or source) and an output (or destination).

The challenge with stream-based APIs is that the two ends of the stream usually operate at different speeds. For example, the code reading from a stream might process data faster than the data is being written into the stream.

To manage this discrepancy, stream implementations typically include an internal buffer. This buffer holds data that has been written but not yet read, ensuring data is available for reading when needed and providing space to store data as it is written.

The size of this internal buffer is determined by the highWaterMark property of the options object when the stream is created. The default value is 65536 (64 KB), or 16 for objectMode streams.

Promise API

The stream/promises API provides an alternative set of asynchronous utility functions for streams that return Promise objects rather than using callbacks. This approach manages backpressure automatically and improves readability comparing to callback variants.

import { createReadStream, createWriteStream } from "fs";
import { pipeline } from "stream/promises";
import { createGzip } from "zlib";

const readable = createReadStream("data/alphabet.txt");
const transform = createGzip();
const writable = createWriteStream("data/output.txt.gz");

pipeline(readable, transform, writable)
  .then(() => {
    console.log("Done");
  })
  .catch((err) => {
    console.error(err);
  });