Streams: Readable, Writable and Transform

Sip, do not gulp

Imagine you need to move a swimming pool of water to another pool. You could try to lift all the water at once (impossible), or you could connect a hose and let it flow a bit at a time.

A stream is that hose for data. Instead of loading a whole file into memory, you handle it in small pieces called chunks, one after another.

With readFile, a 4 GB video needs 4 GB of memory before you can touch the first byte. With a stream, Node reads 64 KB, hands it to you, reads the next 64 KB, and so on. Memory stays small, and you can start working straight away.

Where you already meet streams

Streams are everywhere in Node:

• Reading and writing files: fs.createReadStream, fs.createWriteStream.
• HTTP: an incoming request is a readable stream; the response is a writable stream.
• The terminal: process.stdin is readable; process.stdout and process.stderr are writable.
• Compression: zlib.createGzip() is a transform stream.

Every stream is also an EventEmitter, so everything from the EventEmitter lesson applies, including the rule about 'error' listeners.

The four kinds of stream

You will mostly use streams that Node gives you, and sometimes build small Transforms of your own.

Reading a file with a Readable stream

For the examples, imagine orders.txt holds 5,000 lines like Order 1: 2 x coffee for Asha, about 155 KB in total.

With events

A Readable stream emits 'data' for every chunk and 'end' when there is no more:

// read-events.mjs
import { createReadStream } from 'node:fs';

const stream = createReadStream('orders.txt');
let chunks = 0;

stream.on('data', (chunk) => {
  chunks++;
  console.log(`chunk ${chunks}: ${chunk.length} bytes`);
});
stream.on('end', () => console.log('Done'));
stream.on('error', (err) => console.error('Read failed:', err.code));

chunk 1: 65536 bytes
chunk 2: 65536 bytes
chunk 3: 27821 bytes
Done

Each chunk is a Buffer (see the Buffers lesson). createReadStream reads 64 KB (65,536 bytes) at a time by default.

With for await (recommended)

Readable streams are async iterable, so you can loop over them with for await. This is the easiest and safest way to read a stream:

// read-loop.mjs
import { createReadStream } from 'node:fs';

const stream = createReadStream('orders.txt', {
  encoding: 'utf8',         // give me strings, not Buffers
  highWaterMark: 16 * 1024, // read 16 KB at a time
});

for await (const chunk of stream) {
  console.log(typeof chunk, chunk.length);
}

string 16384
string 16384
...
string 11437

• encoding: 'utf8' turns chunks into strings and never cuts a character in half.
• highWaterMark sets the chunk size. You will learn what else it controls in the Pipelines lesson.

Errors are thrown inside the loop, so a normal try/catch works:

try {
  for await (const chunk of createReadStream('missing.txt')) {}
} catch (err) {
  console.log('Read failed:', err.code); // prints: Read failed: ENOENT
}

Reading line by line

Chunks are cut by size, not by line. To process a file one line at a time, wrap the stream with node:readline:

// lines.mjs
import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const lines = createInterface({
  input: createReadStream('orders.txt'),
  crlfDelay: Infinity, // treat \r\n (Windows) as one line break
});

let count = 0;
for await (const line of lines) {
  count++;
  if (count <= 2) console.log(line);
}
console.log('Total lines:', count);

Order 1: 2 x coffee for Asha
Order 2: 2 x coffee for Asha
Total lines: 5000

This works on files far bigger than your memory.

Writing with a Writable stream

A Writable stream has two main methods:

• write(chunk): send a piece of data.
• end(lastChunk?): say “that is all”. You can pass one last chunk.

When everything has been written, it emits 'finish'.

// write.mjs
import { createWriteStream } from 'node:fs';

const out = createWriteStream('shopping.txt');

out.write('Milk\n');
out.write('Bread\n');
out.end('Eggs\n');

out.on('finish', () => console.log('Saved!'));
out.on('error', (err) => console.error('Write failed:', err.code));

Saved!

shopping.txt now holds the three lines. To wait for 'finish' with await, use events.once:

import { once } from 'node:events';

out.end();
await once(out, 'finish');

Note: write() returns true or false. A false means “slow down, my buffer is full”. Ignoring it is fine for small amounts of data, but not for large ones. That is called backpressure, and it is the main topic of the next lesson.

Readable.from: a stream from anything

Readable.from() turns an array, a string, or a generator into a Readable stream. It is perfect for tests and for feeding your own data into stream code.

// from.mjs
import { Readable } from 'node:stream';

for await (const name of Readable.from(['Asha', 'Ravi', 'Maya'])) {
  console.log('name:', name);
}

name: Asha
name: Ravi
name: Maya

A generator is a function marked with * that hands out values one at a time with yield. An async function* can await between values. Streams made from generators produce data only when asked:

// countdown.mjs
import { Readable } from 'node:stream';

async function* countdown() {
  for (let i = 3; i > 0; i--) yield `${i}...`;
  yield 'Lift off!';
}

for await (const line of Readable.from(countdown())) {
  console.log(line);
}

3...
2...
1...
Lift off!

Tip: Readable.from('hello') sends the whole string as one chunk. It does not split it into characters.

Object mode

Normally, streams carry bytes (Buffers) or strings. In object mode, each chunk can be any JavaScript value: an object, a number, an array.

Streams made by Readable.from are in object mode by default. For streams you build yourself, pass objectMode: true.

In object mode, highWaterMark counts objects (16 by default) instead of bytes.

Building your own streams

You build a stream by passing an options object with one key method.

A custom Readable

Pass a read() method. Node calls it whenever the stream wants more data. Inside it, call this.push(value) to hand data out, and this.push(null) to say “the end”. In practice, Readable.from(generator) is usually simpler, so reach for that first.

A custom Writable

The write(chunk, encoding, callback) method receives each chunk. Call callback() when you have finished with it, or callback(error) if it failed:

// logger.mjs
import { Writable } from 'node:stream';

const logger = new Writable({
  write(chunk, encoding, callback) {
    console.log('Writing:', chunk.toString());
    callback(); // ready for the next chunk
  },
});

logger.on('finish', () => console.log('All written'));
logger.write('first');
logger.write('second');
logger.end('last');

Writing: first
Writing: second
Writing: last
All written

Calling callback() is essential. Until you do, the stream will not give you the next chunk.

A custom Transform

A Transform takes chunks in, changes them, and pushes them out. Its transform(chunk, encoding, callback) method calls callback(null, output) to send a result:

// upper.mjs
import { Transform } from 'node:stream';

const upper = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  },
});

upper.on('data', (chunk) => console.log('out:', chunk.toString()));
upper.write('hello ');
upper.write('asha');
upper.end();

out: HELLO 
out: ASHA

A Transform can also skip a chunk (call callback() with nothing), push several results with this.push(), and run a flush(callback) method at the very end to push anything left over. The playground at the bottom of this page uses the skip trick.

To connect streams, every Readable has a .pipe(destination) method. source.pipe(upper) sends everything from source into upper. Piping has some traps around errors, and the Pipelines lesson shows the safer pipeline() function.

Common mistakes

No ‘error’ listener

// Wrong: if the file is missing, the process crashes
createReadStream('missing.txt').on('data', (chunk) => console.log(chunk));

Add an 'error' listener, or use for await inside try/catch, or use pipeline() from the next lesson.

Forgetting the callback in a custom stream

If your write or transform method never calls callback, the stream silently stops after the first chunk. Every path through the method must call it exactly once.

Quick recap

• A stream handles data in chunks, so memory stays small even for huge files.
• Four kinds: Readable, Writable, Duplex and Transform.
• Read with for await...of (simplest), or with 'data' and 'end' events.
• Write with write() and finish with end(), then wait for 'finish'.
• Readable.from() makes a stream from an array, string or generator.
• Object mode lets chunks be any JavaScript value.
• Custom streams: read() + push(), write() + callback(), transform() + callback(null, out).
• Always handle errors: an 'error' listener, try/catch around for await, or pipeline().