Meridian
TB-Scale Reliable Data Transfer

Executive Summary

This document specifies the architecture, design decisions, and implementation details for Meridian, a production-grade system that transfers terabyte-scale datasets between nodes with zero data loss.

The system provides resumable transfers, content-defined deduplication, multi-path parallelism, end-to-end cryptographic integrity verification, and automatic failover — all while saturating available network bandwidth.

Requirements

Functional Requirements

• ▶
  Bulk Transfer: Transfer datasets from 1 GB to 50+ TB between two nodes (source and destination).
• ▶
  Resumability: If interrupted at any point (crash, network failure, reboot), resume from the point of interruption, not restart from scratch.
• ▶
  Incremental Sync: When transferring a previously-transferred dataset, only changed portions are re-sent.
• ▶
  Integrity Guarantee: The receiver cryptographically verifies that received data is bit-for-bit identical to the source. Silent corruption is detected.
• ▶
  Atomic Commit: A transfer is either fully committed or not committed at all. No partial states visible to consumers.
• ▶
  Multi-File Support: A single transfer job can include thousands of files in a directory tree, preserving structure on the destination.

Performance Targets

Reliability Targets

Security Requirements

Scalability

• • Datasets up to 50 TB in a single transfer job
• • Up to 10 million chunks per transfer (~40 TB at 4 MB average)
• • Up to 100 concurrent transfer jobs per node
• • Horizontal scaling via multiple sender/receiver pairs

Observability

• • Metrics: Prometheus-compatible on :9090 — throughput, chunk latency histogram, dedup ratio, error rate, WAL backlog
• • Tracing: OpenTelemetry distributed tracing with per-chunk spans. 100% sampling on failures, 10% on success
• • Alerting: Pre-configured rules for throughput drops, integrity failures, stalled transfers, disk space, certificate expiry

High-Level Architecture

The system consists of three planes: the data plane (chunking, transfer, reassembly), the control plane (coordination, state management, health monitoring), and the observability plane (metrics, tracing, alerting).

End-to-End Data Flow

Transfer State Machine

Component Responsibilities

Content-Defined Chunking

Algorithm: Buzhash

We use Buzhash (bitwise rolling hash) rather than Rabin fingerprinting:

We chose Buzhash because chunking is CPU-bound and runs inline with I/O. The 25% speed advantage keeps pace with NVMe read speeds (~3.5 GB/s). Rabin's better distribution doesn't matter because we rely on SHA-256 for content addressing, not the rolling hash.

Chunk Size Parameters

The 4 MB average is a three-way tension: smaller chunks (1 MB) give better dedup granularity but produce 10M chunks per 10 TB, requiring ~2.5 GB for the index. Larger chunks (16 MB) reduce index size but miss small edits. 4 MB is tunable per-environment.

The Chunk Manifest

ChunkMeta {
  chunk_index: u64,
  file_path:   String,
  offset:      u64,
  length:      u32,
  sha256:      [u8; 32]
}

The manifest records each chunk's position (byte offset within the file), size, and content hash. It's deterministic: given the same file content and CDC parameters, the same boundaries are always produced. For multi-file transfers, chunk indices are global across all files. The sender can transmit chunks in any order; the receiver uses the manifest to place each chunk correctly.

Deduplication Index

Transport Layer

Protocol Stack: gRPC over HTTP/2 over TCP

Why 8 Connections and 16 Streams?

What is Multiplexing?

Without multiplexing, sending 16 chunks simultaneously requires 16 TCP connections. With multiplexing, one TCP connection carries all 16 by interleaving. HTTP/2 chops each chunk's data into small frames (16 KB default), stamps each frame with a stream ID, and interleaves them on the wire. The receiver reads frames, sorts by stream ID, and reassembles each stream independently. This is purely a framing-layer concern — invisible to application code.

Why TCP and Not UDP?

QUIC Fallback

QUIC runs over UDP but builds its own per-stream reliability, eliminating TCP's head-of-line blocking. Used as a fallback when measured packet loss exceeds 1% (typically WAN/internet paths). Not the default because: (1) NIC hardware offloading only works for TCP; (2) BBR over TCP is more mature; (3) kernel bypass (io_uring) integrates better with TCP sockets.

Adaptive Compression

The first 64 KB of each chunk is sampled to estimate entropy. High-entropy data is sent uncompressed. Algorithm choice between LZ4/Zstd is based on the ratio of measured network throughput to available CPU cycles.

Zero-Copy I/O & Kernel Optimizations

Why a Single TCP Connection Doesn't Fill the Pipe

A 10 Gbps NIC can push 1.25 GB/s onto the wire. A single “normal” TCP connection achieves a fraction of this. Five independent bottlenecks compound:

Combined Effect

Data Integrity

Four layers of verification, each catching a different class of corruption at a different point in the pipeline:

Reliability & Fault Tolerance

Write-Ahead Log (WAL)

The WAL is an append-only file on disk — not a service, not a database, not a distributed system. It stores a sequence of binary records, each describing a state transition:

Why a WAL and Not a Database?

The WAL is append-only: every write is sequential I/O at the end of the file — the fastest possible disk operation. NVMe SSDs can do 500K+ sequential small writes per second. A database doing random I/O, maintaining B-tree indexes, and journaling is 10–50× slower for this access pattern. With 64 parallel workers recording state, a database's row-level locking creates contention. An append-only file has zero lock contention.

Crash Recovery Scenarios

Retry Strategy

delay = min(base × 2^attempt + random(0, jitter), max_delay)
base: 100ms, jitter: 50ms, max: 30s, max_attempts: 5

Per-connection retries: if a gRPC stream breaks, reconnect with fresh TLS handshake. All in-flight chunks on that stream are re-queued.

Chaos Engineering Tests

Security

Transport Security

All data in transit uses TLS 1.3 with cipher suites AES-256-GCM-SHA384 and ChaCha20-Poly1305-SHA256. Mutual TLS (mTLS) requires both sender and receiver to present X.509 certificates signed by the internal CA. Certificate rotation every 90 days.

Encryption at Rest — Key Hierarchy

Access Control & Audit

RBAC with OPA policy engine evaluates every transfer request against policies: allowed source/dest pairs, max transfer size, time-of-day windows, rate limits per-user. Immutable audit log records every API call, state transition, and access with who, what, when, from_ip, transfer_id, result. Retained for 2 years.

Observability

Key Metrics (Prometheus)

Distributed Tracing

Every transfer gets a root span. Child spans for each phase: scan_files, chunk_file, negotiate_dedup, transfer_chunks (with per-chunk sub-spans for compress/encrypt/send), verify_merkle, commit. Sampling: 100% for failed transfers, 10% for successful (adaptive based on volume).

Alerting Rules

Design Tradeoffs Summary

Every major decision involved rejecting viable alternatives:

Operations & Capacity Planning

Kernel Tuning

# Socket buffers
net.core.rmem_max = 134217728          # 128 MB receive buffer max
net.core.wmem_max = 134217728          # 128 MB send buffer max
net.ipv4.tcp_rmem = 4096 1048576 134217728
net.ipv4.tcp_wmem = 4096 1048576 134217728

# Congestion control
net.ipv4.tcp_congestion_control = bbr

# Network queue
net.core.netdev_max_backlog = 250000

# VM tuning
vm.dirty_ratio = 40
vm.swappiness = 10

# File descriptors
ulimit -n 1048576  # 1M open fds

Deployment Modes

Hardware Recommendations

Transfer Time Estimates

Assumes 80% link utilization, 0% dedup, Zstd level 3 (3.2× compression). Actual times significantly lower with dedup on incremental transfers.

When This System is Wrong