Skip to main content

JSONL Performance

Benchmarks, metrics, and optimization strategies for high-performance JSONL processing

Performance Advantages

O(1) Memory

Constant memory usage regardless of file size. Process 100GB files with 10MB RAM.

Instant Start

Begin processing immediately without loading entire file. 0ms time-to-first-record.

Append Speed

Add records in O(1) time. No file rewriting required unlike JSON arrays.

Parse Speed Benchmarks

Python: JSON vs JSONL Parsing

Test setup: 100,000 records, ~50MB file, Python 3.11, MacBook Pro M1

Format Library Parse Time Memory Peak Time to First Record
JSON Array json (stdlib) 1,450ms 280MB 1,450ms
JSONL json (stdlib) 1,380ms 8MB <1ms
JSONL orjson 410ms 8MB <1ms
JSONL.gz orjson + gzip 680ms 12MB 15ms

Key takeaways:

  • JSONL uses 35x less memory than JSON array
  • orjson provides 3.4x speedup over standard library
  • Compressed JSONL still faster than uncompressed JSON array
# Benchmark code
import json
import orjson
import gzip
import time
import tracemalloc

def benchmark_json_array():
    tracemalloc.start()
    start = time.time()
    with open('data.json', 'r') as f:
        data = json.load(f)  # Load entire array
        for record in data:
            process(record)
    elapsed = time.time() - start
    peak = tracemalloc.get_traced_memory()[1] / 1024 / 1024
    tracemalloc.stop()
    print(f"JSON Array: {elapsed*1000:.0f}ms, {peak:.0f}MB")

def benchmark_jsonl_orjson():
    tracemalloc.start()
    start = time.time()
    with open('data.jsonl', 'rb') as f:
        for line in f:
            record = orjson.loads(line)
            process(record)
    elapsed = time.time() - start
    peak = tracemalloc.get_traced_memory()[1] / 1024 / 1024
    tracemalloc.stop()
    print(f"JSONL (orjson): {elapsed*1000:.0f}ms, {peak:.0f}MB")

JavaScript/Node.js: Streaming Performance

Test setup: 100,000 records, ~50MB file, Node.js 20.x, MacBook Pro M1

Approach Parse Time Memory Peak Throughput
JSON.parse() entire file 2,100ms 350MB 47 rec/ms
readline + JSON.parse() 1,820ms 12MB 55 rec/ms
ndjson stream 1,240ms 10MB 81 rec/ms 
// Fast JSONL streaming with ndjson
const fs = require('fs');
const ndjson = require('ndjson');

const stream = fs.createReadStream('data.jsonl')
  .pipe(ndjson.parse())
  .on('data', (record) => {
    // Process each record as it arrives
    process(record);
  })
  .on('end', () => {
    console.log('Complete');
  });

Go: Concurrency Benchmark

Test setup: 1,000,000 records, ~500MB file, Go 1.21, MacBook Pro M1, 10 cores

Approach Parse Time Throughput CPU Usage
Single-threaded 3,200ms 313 rec/ms ~100% (1 core)
10 goroutines 420ms 2,381 rec/ms ~900% (9 cores)

7.6x speedup with parallel processing. JSONL's line-based format is trivially parallelizable. 

// Go: Parallel JSONL processing
package main

import (
    "bufio"
    "encoding/json"
    "os"
    "sync"
)

func processChunk(lines [][]byte, wg *sync.WaitGroup) {
    defer wg.Done()
    for _, line := range lines {
        var record map[string]interface{}
        json.Unmarshal(line, &record)
        // Process record...
    }
}

func main() {
    file, _ := os.Open("data.jsonl")
    defer file.Close()

    scanner := bufio.NewScanner(file)
    const chunkSize = 10000
    var chunk [][]byte
    var wg sync.WaitGroup

    for scanner.Scan() {
        chunk = append(chunk, append([]byte(nil), scanner.Bytes()...))

        if len(chunk) >= chunkSize {
            wg.Add(1)
            go processChunk(chunk, &wg)
            chunk = nil
        }
    }

    if len(chunk) > 0 {
        wg.Add(1)
        go processChunk(chunk, &wg)
    }

    wg.Wait()
}

Memory Efficiency

Memory Usage: JSON Array vs JSONL

Memory consumption for various file sizes (streaming JSONL vs loading JSON array):

File Size Records JSON Array Memory JSONL Memory Savings
10 MB 20,000 ~60 MB ~5 MB 92%
100 MB 200,000 ~550 MB ~8 MB 99%
1 GB 2,000,000 ~5.5 GB ~10 MB 99.8%
10 GB 20,000,000 ~55 GB (OOM) ~12 MB 99.98%

Why the difference?

  • JSON arrays load entire structure into memory (all records + array overhead)
  • JSONL streams one record at a time, discarding after processing
  • JSON parser allocates temporary objects during deserialization
  • Memory overhead typically 5-6x file size for JSON arrays

Real-World Memory Profiling

Measure memory usage in your own applications: 

# Python: Profile memory usage
import tracemalloc
import json

tracemalloc.start()

# Your processing code here
with open('data.jsonl', 'r') as f:
    for line in f:
        obj = json.loads(line)
        process(obj)

current, peak = tracemalloc.get_traced_memory()
print(f"Current: {current / 1024 / 1024:.1f} MB")
print(f"Peak: {peak / 1024 / 1024:.1f} MB")
tracemalloc.stop()

Compression Benchmarks

Compression Ratios

JSONL compresses extremely well due to repeated field names. Test file: 100MB JSONL with typical user records.

Compression Compressed Size Ratio Compress Time Decompress Time
None 100.0 MB - - -
gzip -6 12.4 MB 87.6% 3.2s 0.8s
gzip -9 11.8 MB 88.2% 8.1s 0.8s
bzip2 8.9 MB 91.1% 12.4s 4.2s
xz 7.2 MB 92.8% 28.6s 1.9s
zstd -3 11.2 MB 88.8% 0.6s 0.3s

Recommendations:

  • gzip -6 - Best balance of speed and compression (default choice)
  • zstd - Best for real-time pipelines (5x faster than gzip)
  • xz - Best for long-term archival (smallest size)

Streaming Compressed JSONL

Process compressed JSONL without decompressing entire file first:

Python

import gzip
import json

# Stream-decompress and process
with gzip.open('data.jsonl.gz', 'rt') as f:
    for line in f:
        obj = json.loads(line)
        # Memory stays constant!

Node.js

const fs = require('fs');
const zlib = require('zlib');
const readline = require('readline');

const stream = fs.createReadStream('data.jsonl.gz')
  .pipe(zlib.createGunzip())
  .pipe(readline.createInterface({ input: process.stdin }));

for await (const line of stream) {
  const obj = JSON.parse(line);
  // Process...
}

Command Line

# Decompress and process on-the-fly with jq
zcat data.jsonl.gz | jq '.name'

# Decompress, filter, compress again
zcat input.jsonl.gz | grep '"status":"active"' | gzip > filtered.jsonl.gz

Performance impact: Decompression adds 40-60% overhead, but file I/O reduction often makes it faster overall, especially with SSDs.

Streaming Efficiency

Time to First Record

How quickly can you start processing data? Test: 1GB file with 2M records.

JSON Array

12.4 seconds

Must parse entire file before accessing first record

JSONL

<1 millisecond

First record available immediately after reading first line

12,000x faster startup! Critical for real-time processing and interactive applications.

Processing Throughput

Records processed per second across different languages and approaches:

Language Library Records/sec Notes
Python json (stdlib) 72,000 Baseline
Python orjson 244,000 3.4x faster
Node.js JSON.parse() 55,000 Single-threaded
Node.js ndjson 81,000 Optimized streaming
Go encoding/json 313,000 Single goroutine
Go encoding/json (parallel) 2,381,000 10 goroutines
Rust serde_json 520,000 Zero-copy parsing
Command line jq 45,000 General purpose

Network Streaming Performance

Benchmark: HTTP endpoint returning 100,000 records over network.

JSON Array

18.2s

Client waits for full response

JSONL Streaming

0.15s

Time to first record

Total Transfer

18.5s

But processing starts immediately

Key benefit: User sees results in 150ms instead of 18 seconds, even though total transfer time is similar. Perceived performance is 120x better.

Real-World Performance Scenarios

Scenario 1: Log Processing Pipeline

Task: Process 50GB daily application logs (25M records), extract errors, write to database

JSON Array Approach

  • Parse time: 8+ minutes
  • Memory required: ~280GB
  • Result: Out of memory crash

JSONL Approach

  • Parse time: 4.2 minutes (streaming)
  • Memory required: 50MB
  • Result: Successfully completed

With gzip compression, file size reduced to 6GB, processing time 6.5 minutes.

Scenario 2: ML Training Data Preparation

Task: Transform 10M training examples for GPT fine-tuning

In-Memory Processing

  • Load all data: 45GB RAM
  • Transform time: 12 minutes
  • Write output: 8 minutes
  • Total: 20 minutes

Streaming Pipeline

  • Memory usage: 200MB
  • Stream + transform: 9 minutes
  • Simultaneous write: Included
  • Total: 9 minutes

2.2x faster, 225x less memory. Can run on small instance instead of memory-optimized.

Scenario 3: Real-Time Analytics Dashboard

Task: Display live events in web dashboard as they arrive

Polling JSON Endpoint

  • Poll every 5 seconds
  • Return full array (growing)
  • Client re-downloads all data
  • Latency: 5+ seconds

JSONL Streaming

  • Server-sent events (JSONL)
  • Push events as they occur
  • Client processes incrementally
  • Latency: <100ms

50x lower latency, 90% less bandwidth usage.

Performance Optimization Tips

Do

  • Use fast JSON libraries (orjson, simdjson, jsoniter)
  • Stream large files instead of loading into memory
  • Compress with gzip or zstd for storage
  • Parallelize processing across multiple cores
  • Use buffered I/O with large buffer sizes (1MB+)
  • Filter before parsing when possible
  • Build offset indexes for random access
  • Partition large datasets by date/category

Don't

  • Load entire file into array before processing
  • Use unbuffered file I/O
  • Parse every line if you only need subset
  • Store uncompressed JSONL in production
  • Read compressed files multiple times (cache if needed)
  • Ignore memory profiling in production
  • Process multi-GB files on single thread
  • Use JSONL for tiny datasets (<1MB)