Trending Technology

Dynamic Write-Amplification Asynchrony: Compaction Cascades in Multi-Tenant NVMe Zoned Namespaces (ZNS)

By Arvind Rocky📚 4 Modules⏱ 36 min read🤖 AI-Generated

Module 1: Module 1: ZNS Architecture Fundamentals and Multi-Tenant Crosstalk Mechanics

Sub-module 1.1: NVMe ZNS Device Model—Zone State Machines, Sequential Write Constraints, and Hardware Zone-Reset Queue Internals+

NVMe Zoned Namespaces fundamentally restructure the write model that storage engines must respect. Unlike traditional block devices where any LBA can be written at any time, ZNS divides the namespace into fixed-size zones (typically 256 MB to 1 GB each), each enforcing a strict sequential-write-only constraint. This architectural choice directly maps to the physical erase-block structure of NAND flash, allowing the drive firmware to eliminate the garbage collection overhead that plagues conventional SSDs.

Zone State Machine Architecture

Every ZNS zone operates as a finite state machine with precisely defined transitions. A zone begins in the Empty state, where its write pointer (WP) is positioned at the zone's starting LBA. As sequential writes arrive, the zone transitions to Implicitly Opened, and the write pointer advances monotonically. Once the write pointer reaches the zone capacity, the zone enters Full state. To reuse a zone, the zone must be explicitly reset via a Zone Reset command, which atomically clears all data and repositions the write pointer to the zone start—transitioning back to Empty.

The critical states are Explicitly Opened and Implicitly Opened. A zone can have at most 128 explicitly opened zones across the entire namespace (this limit is hardware-specific but standardized in ZNS spec revision 1.1). When a write command arrives for a zone in Empty state, the firmware automatically transitions it to Implicitly Opened without consuming an explicit-open resource. However, if a tenant's workload requires interleaved writes across multiple zones or needs to maintain zone state across I/O bursts, explicit open commands become necessary, and exceeding the hardware limit causes command rejection.

Sequential Write Constraint Enforcement

The sequential write constraint means that if a write command targets LBA X in a zone, the zone's write pointer must equal X; otherwise, the command fails with a Zone Invalid Write error. Storage engines cannot tolerate this error—it indicates a fundamental protocol violation. Multi-tenant database systems must therefore track per-zone write pointers at the userspace level, or risk silent data corruption if a tenant's compaction routine miscalculates where the next write should land.

Consider a practical scenario: Tenant A's LSM tree has a level-0 file at zone 4, with a write pointer at offset 512 MB. Tenant B's compaction routine, running on the same host, flushes data to zone 4 (perhaps due to zone allocation fairness logic that round-robins assignments). If Tenant B's writes are not serialized with Tenant A's tracking logic, both could attempt to write at offset 512 MB, causing one to fail. The storage engine must implement a per-zone write-pointer cache and enforce mutual exclusion on zone transitions.

Hardware Zone-Reset Queue Internals

Zone reset operations are where multi-tenant crosstalk emerges. When a zone is full, the only way to reclaim it is to issue a Zone Reset command. Unlike writes, which can pipeline across multiple zones, zone reset is a destructive operation that must be serialized at the drive firmware level. The NVMe controller maintains an internal zone-reset queue, typically 32 to 64 entries deep (vendor-specific), that serializes all reset operations across all zones.

When Tenant A's compaction routine decides to reset 10 zones simultaneously (a common pattern in aggressive LSM compaction), those 10 reset commands enter the hardware queue. The drive firmware processes them sequentially, each reset taking 100–500 microseconds depending on NAND flash characteristics and internal cache state. During this window, no other zone can be reset, and critically, the drive's internal resource scheduler may deprioritize or stall write commands to other zones to ensure reset queue progress.

The zone-reset queue saturation creates a feedback loop: as resets back up, the firmware's I/O scheduler becomes conservative, reducing write throughput to zones not currently being reset. This manifests as elevated tail latencies for Tenant B's writes, even though Tenant B issued no reset commands. The crosstalk is invisible at the NVMe protocol level—no errors are returned—but the latency impact is measurable and severe (100–500 microsecond tail latency increases, sometimes spiking to milliseconds).

Real-world impact: A storage cluster running three database tenants on a single ZNS drive observed that when Tenant A triggered compaction (resetting 15 zones in rapid succession), Tenant B's p99 write latency increased from 200 microseconds to 2.1 milliseconds, violating its 1-millisecond SLA. The reset queue was the bottleneck, but standard monitoring tools (iostat, nvme-cli) showed no errors or queue depth anomalies—the crosstalk was purely latency-based.

Sub-module 1.2: LSM-Tree Compaction Burst Patterns and Write Amplification Cascades in Shared Physical Namespace Topologies+

Log-Structured Merge (LSM) trees are the foundational data structure for modern high-throughput databases (RocksDB, LevelDB, Cassandra). They achieve write efficiency by buffering writes in memory and periodically flushing sorted runs to disk. However, to maintain read performance, LSM trees must periodically compact overlapping runs, which generates massive write bursts that directly interact with ZNS zone allocation and reset patterns.

Compaction Cascade Mechanics

A typical LSM tree maintains multiple levels (L0, L1, L2, ...). Level-0 is where memtable flushes land; it contains unsorted runs. Compaction merges overlapping runs from adjacent levels, producing larger sorted runs at the next level. Each compaction pass reads data from two levels and writes the merged output to the next level, generating write amplification of 2–10x the input data size, depending on overlap ratios.

In a single-tenant ZNS setup, this write amplification is manageable: the storage engine allocates zones sequentially, fills them with compacted data, and resets them when full. However, in a multi-tenant environment with shared physical namespaces, compaction cascades from different tenants interleave at the zone level. Tenant A might be compacting L2→L3 (writing 500 MB), while Tenant B simultaneously flushes L0 (writing 100 MB). If both tenants' zone allocators are independent or poorly coordinated, they may allocate overlapping zone ranges, creating contention.

More critically, compaction bursts create zone-reset amplification. When Tenant A completes a compaction pass, it fills 2–3 zones with merged data. These zones are now full and cannot accept more writes. Tenant A's next compaction pass will reset these zones, but the reset queue serialization means those resets will stall other tenants' writes. If Tenant B is in the middle of a flush when Tenant A's resets arrive, Tenant B's write latency spikes.

Write Amplification Cascade Patterns

Write amplification in LSM trees follows predictable patterns. At low write rates (< 10 MB/s), level-0 compactions dominate, generating 2–3x write amplification. As write rates increase, higher-level compactions trigger, amplifying to 5–10x. In a multi-tenant system with shared zones, these cascades overlap:

Time T0–T100ms: Tenant A flushes L0 (100 MB input, 100 MB written to zone 0).
Time T100–T200ms: Tenant B flushes L0 (80 MB input, 80 MB written to zone 1).
Time T200–T500ms: Tenant A compacts L1→L2 (500 MB input, 1000 MB output, written to zones 2–4).
Time T300–T400ms (overlapping): Tenant B compacts L1→L2 (300 MB input, 600 MB output, written to zones 5–6, but zone allocator assigns zone 5 which is partially allocated to Tenant A's L0).

This interleaving creates zone fragmentation: zones contain data from multiple tenants or multiple compaction levels, reducing the efficiency of zone resets. When Tenant A resets zone 5, it must wait for Tenant B's writes to complete, or vice versa. The zone-reset queue serialization amplifies this: even though zones 0–6 are logically independent, the hardware reset queue processes all resets globally.

Shared Namespace Topology Constraints

A typical multi-tenant ZNS setup partitions the 256-zone namespace (example size) into tenant slices: Tenant A gets zones 0–85, Tenant B gets zones 86–170, Tenant C gets zones 171–255. This isolation is logical, not physical—all zones share the same hardware reset queue, the same NVMe command queue, and the same internal bandwidth.

The isolation breaks under compaction cascades because:

1. Zone-reset queue is global: All resets, regardless of tenant origin, compete for the single 32–64 entry queue.

2. Write bandwidth is shared: The drive's NAND flash bandwidth (typically 2–4 GB/s aggregate) is divided among all active writes. When Tenant A's compaction writes at 1.5 GB/s, Tenant B's concurrent writes are throttled to 0.5–1 GB/s.

3. Firmware I/O scheduler is conservative: To ensure reset queue progress, firmware may deprioritize write commands when resets are pending, causing write latency spikes for all tenants.

Real example: A 3-tenant RocksDB cluster on a single ZNS drive observed the following during peak load:

Tenant A (high write rate): 50 MB/s sustained, compaction every 30 seconds, resetting 4 zones per compaction.
Tenant B (medium write rate): 20 MB/s sustained, compaction every 60 seconds, resetting 2 zones per compaction.
Tenant C (low write rate): 5 MB/s sustained, compaction every 120 seconds, resetting 1 zone per compaction.

When all three compactions aligned (every 120 seconds), the zone-reset queue received 7 reset commands in rapid succession. The hardware queue (capacity 32) filled to 7 entries, but the firmware's internal scheduler became conservative, reducing write throughput for all tenants by 20–30%. Tenant C's p99 write latency spiked from 300 microseconds to 1.8 milliseconds, violating its 1-millisecond SLA, despite Tenant C issuing no reset commands.

Sub-module 1.3: Latency Crosstalk Root Cause Analysis—How Zone-Reset Queue Saturation Serializes Concurrent Tenant I/O Across Isolated Zones+

Multi-tenant latency crosstalk in ZNS systems is fundamentally a zone-reset queue saturation problem that manifests as write latency inflation for innocent tenants. The crosstalk is invisible at the NVMe protocol level (no errors, no queue-depth indicators), making it a silent SLA violator. Understanding the root cause requires tracing the interaction between the hardware reset queue, the firmware I/O scheduler, and tenant-level compaction patterns.

Zone-Reset Queue Saturation Mechanics

The NVMe ZNS specification mandates that zone reset operations are atomic and serialized at the device firmware level. The drive maintains an internal reset queue (typically 32–64 entries) that buffers incoming Zone Reset commands. When a reset command arrives, the firmware enqueues it and processes it in FIFO order. Each reset operation takes 100–500 microseconds (depending on zone size, NAND flash speed, and internal cache state). During this window, the firmware's I/O scheduler must decide: continue processing write commands to other zones, or deprioritize writes to ensure reset queue progress?

The firmware's choice is conservative: if the reset queue depth exceeds a threshold (typically 50% capacity), the firmware throttles write commands. This is a firmware-level optimization to prevent reset starvation, but it creates latency crosstalk. When Tenant A's compaction flushes 10 zone-reset commands into the queue within a 1-millisecond window, the queue depth jumps to 10 entries (out of 32). The firmware's scheduler detects this and begins throttling write commands to non-reset zones. Tenant B, running on a completely isolated zone, suddenly experiences elevated write latency because its commands are deprioritized relative to reset processing.

Latency Crosstalk Propagation Path

The crosstalk propagates through the following sequence:

1. Tenant A's Compaction Trigger (T0): Tenant A's LSM tree completes a compaction pass, filling zones 10–15. These zones are now full and must be reset before reuse.

2. Reset Command Burst (T0+1μs): Tenant A's storage engine issues 6 Zone Reset commands, one per zone, in rapid succession. These arrive at the NVMe controller's command queue within 10 microseconds.

3. Queue Depth Spike (T0+10μs): The firmware dequeues these 6 resets and enqueues them in the internal reset queue. Reset queue depth is now 6/32 (19%).

4. Scheduler Threshold Crossed (T0+50μs): If Tenant A continues issuing resets (which it does, as compaction cascades), the queue depth may reach 16/32 (50%), triggering the firmware's conservative throttling policy.

5. Write Throttling Activation (T0+100μs): The firmware begins deprioritizing write commands to non-reset zones. Tenant B's write commands, destined for zone 50 (completely isolated from Tenant A's zones 10–15), are now queued behind reset processing.

6. Tenant B Latency Spike (T0+500μs–T0+2ms): Tenant B's write latencies increase from baseline 200 microseconds to 1–2 milliseconds, as its commands wait for Tenant A's resets to drain from the queue.

Measurement and Detection of Crosstalk

Detecting this crosstalk requires instrumenting at multiple levels:

NVMe Driver Level: The Linux NVMe driver maintains per-queue statistics, but zone-reset queue depth is not exposed to userspace. Custom kernel modules or eBPF probes can trace `nvme_setup_cmd` and `nvme_complete_rq` to measure reset command latency and write command latency correlation.

Firmware Instrumentation: Vendor-specific firmware telemetry (available via NVMe Get Log Page commands) can expose reset queue depth, reset processing time, and write command throttling events. This telemetry is critical for root-cause analysis but is rarely enabled in production.

Application-Level Metrics: Storage engines can measure write latency percentiles and correlate them with compaction events. A spike in p99 write latency coinciding with a compaction trigger is a strong indicator of reset-queue-induced crosstalk.

Example measurement from a production RocksDB deployment:

```

Baseline (no compaction):

Write latency p50: 150 μs

Write latency p99: 280 μs

Write latency p999: 450 μs

During Tenant A Compaction (Tenant B isolated zone):

Write latency p50: 200 μs (33% increase)

Write latency p99: 1800 μs (543% increase)

Write latency p999: 3200 μs (611% increase)

Reset queue depth during compaction: 8–24 entries (25–75% capacity)

Firmware write throttling active: Yes (inferred from latency spike timing)

```

Firmware I/O Scheduler Behavior Under Reset Pressure

The firmware's I/O scheduler implements a weighted round-robin policy that prioritizes reset commands when queue depth is high. Under normal operation (reset queue depth < 50%), writes and resets are interleaved, allowing concurrent write progress. However, once reset queue depth exceeds the threshold, the scheduler switches to a reset-priority mode where:

Reset commands are dequeued and processed with minimal delay (< 10 microseconds between resets).
Write commands are queued but deferred, waiting for reset queue depth to drop below the threshold.
The firmware may also reduce write command acceptance rate (flow control), causing the NVMe command queue to back up at the driver level.

This behavior is invisible to the host because the NVMe protocol does not expose reset queue depth or scheduler state. The host only observes elevated write latency and assumes it is due to NAND flash contention or normal I/O load—not realizing that another tenant's resets are the culprit.

Multi-Tenant Isolation Failure

The fundamental failure of multi-tenant isolation is that zone-level logical isolation does not prevent hardware-level resource contention. Zones 10–15 (Tenant A) and zone 50 (Tenant B) are logically independent, but they share:

The NVMe command queue (typically 1024 entries, shared across all commands from all tenants).
The reset queue (32–64 entries, shared across all reset commands from all tenants).
The NAND flash bandwidth (2–4 GB/s aggregate, divided among all active operations).
The firmware's I/O scheduler logic (single scheduler for entire device).

When Tenant A saturates the reset queue, it indirectly throttles Tenant B's writes by influencing the firmware's scheduler priority. This is a covert channel for latency crosstalk: Tenant A cannot directly access Tenant B's zones, but it can degrade Tenant B's performance by controlling a shared hardware resource (the reset queue).

Quantifying Crosstalk Impact on SLA Compliance

In a production multi-tenant system with three tenants, each with a 1-millisecond write latency SLA:

Baseline SLA compliance: 99.9% (p999 latency < 1 ms).
During aligned compactions (all three tenants compacting simultaneously): 94.2% SLA compliance (p999 latency 2.1–3.5 ms).
SLA violation rate: 5.7% of operations, translating to 16 hours of SLA violations per month.

This violation is not due to hardware failure or capacity exhaustion—it is purely due to reset queue saturation and firmware scheduler conservatism. The solution requires userspace-level zone-reset throttling and fairness mechanisms, which are the subjects of subsequent modules.

Module 2: Module 2: Userspace I/O Scheduling Abstractions and Zone-Allocation Fairness Algorithms

Sub-module 2.1: Designing Zone-Aware I/O Schedulers in Userspace—Token Bucket Rate Limiting, Proportional Fair Queuing, and Tenant-Scoped Write Credits+

The fundamental challenge in multi-tenant NVMe ZNS environments is that the device's zone-reset queue—a hardware-level FIFO structure that manages sequential write pointer resets—operates as a global shared resource with no tenant isolation. When Tenant A's LSM compaction triggers aggressive zone resets, the hardware controller serializes these operations across all zones, creating microscopic but cascading latency spikes that affect Tenant B's unrelated I/O operations. The solution requires building userspace I/O scheduling abstractions that enforce tenant-level rate limits before requests ever reach the kernel or hardware.

Token Bucket Rate Limiting for Zone-Reset Budgets

Token bucket algorithms provide a mathematically elegant mechanism for controlling bursty workloads while permitting sustained throughput. In the ZNS context, the abstraction works by assigning each tenant a zone-reset token budget that replenishes at a configurable rate. Every zone reset operation consumes tokens; if a tenant exhausts its budget, subsequent reset requests queue locally in userspace rather than immediately hitting the hardware controller.

Architecture: Maintain a per-tenant token counter initialized to `max_tokens`. At each scheduling interval (typically 1-10 milliseconds), increment each tenant's counter by `refill_rate` tokens, capped at `max_tokens`. When a compaction thread requests a zone reset, atomically decrement the tenant's token counter. If the counter would go negative, enqueue the request in a local priority queue instead of submitting to the kernel.

Real-world tuning: Consider a ZNS drive with 128 zones and three tenants. If the hardware can safely handle 1,000 zone resets per second globally without creating crosstalk, allocate 300 tokens/sec to Tenant A (30% of zones), 300 tokens/sec to Tenant B, and 400 tokens/sec to Tenant C. Tenants cannot exceed their allocation in bursts because the token refill is time-bounded. If Tenant A's compaction thread generates 500 reset requests in 100ms (a sudden burst), only 30 requests execute immediately; the remaining 470 queue locally, smoothing the submission rate to the hardware.

Proportional Fair Queuing (PFQ) for Zone-Allocation Fairness

Token buckets alone don't address which tenant gets priority when multiple tenants are queued. Proportional Fair Queuing applies weighted scheduling to ensure that even under congestion, each tenant's I/O receives throughput proportional to its configured weight.

Core mechanism: Assign each tenant a weight (e.g., Tenant A = 1.0, Tenant B = 1.5, Tenant C = 0.5). Maintain per-tenant queues of pending zone-reset requests. At each scheduling decision, calculate the virtual finish time for each tenant's next request: `VFT = max(current_virtual_time, last_finish_time) + request_size / weight`. Schedule the request with the minimum VFT. This ensures that over time, each tenant receives throughput proportional to its weight.

Why this matters for ZNS: Zone resets are not uniform in cost. Resetting a zone with 1GB of data takes longer than resetting an empty zone. PFQ's size-aware scheduling prevents a tenant with many small zones from starving a tenant with fewer but larger zones. Additionally, PFQ naturally handles dynamic weight adjustment—if a tenant's SLA tier changes at runtime, you can adjust its weight, and the scheduler automatically rebalances fairness.

Tenant-Scoped Write Credits and Compaction Throttling

Beyond zone resets, compaction threads generate sustained write traffic that can saturate the ZNS drive's write bandwidth. Implementing per-tenant write credits extends token bucket logic to all write operations, not just resets.

Implementation: Each tenant receives a credit budget denominated in bytes per scheduling epoch (e.g., 10MB per 10ms). Compaction threads check their remaining credits before writing. If a write would exceed the budget, the thread blocks on a condition variable until the next epoch's credits are available. This is distinct from kernel-level I/O throttling because it operates entirely in userspace—the compaction thread itself becomes aware of its write quota and can make intelligent scheduling decisions.

Example scenario: Tenant A's compaction generates 50MB/sec of writes during a major level merge. Its credit budget is 5MB per 1ms epoch. The compaction thread writes 5MB, then stalls for 1ms waiting for the next epoch. Meanwhile, Tenant B's writes proceed uncontended. When Tenant A's epoch arrives, it writes another 5MB. The total throughput is preserved (50MB/sec over 10 epochs = 5MB per epoch), but the burst is eliminated.

Critical detail: Write credits must account for zone-write pointer position. A write that crosses a zone boundary (requiring a new zone to be opened) consumes additional hardware resources. Track this implicitly by making credit deductions proportional to the number of zones touched, not just bytes written.

Sub-module 2.2: Zone-Allocation Fairness Protocols—Dynamic Zone Partitioning, Weighted Zone Assignment, and Prevention of Single-Tenant Zone Monopolization+

Zone allocation is the spatial dimension of fairness. While token bucket and PFQ algorithms control the temporal dimension (when requests execute), zone allocation determines which physical zones each tenant can write to. Naive multi-tenant ZNS architectures assign zones statically—Tenant A gets zones 0-31, Tenant B gets zones 32-63—but this creates inflexible capacity bottlenecks and prevents dynamic workload adaptation.

Dynamic Zone Partitioning and Rebalancing

Static zone assignment fails because tenant workloads are non-uniform and time-varying. Tenant A might be in a heavy compaction phase using 90% of its allocated zones, while Tenant B is idle. The idle zones cannot be borrowed by Tenant A without manual reconfiguration.

Dynamic partitioning protocol: Maintain a global zone pool of unallocated zones. Each tenant maintains a zone lease specifying the zones it currently owns. When a tenant's active write set approaches its zone limit (e.g., 85% full), trigger a zone rebalancing event:

1. Calculate each tenant's zone utilization ratio: `active_zones / leased_zones`.

2. Identify tenants with low utilization (ratio < 0.3) and tenants with high utilization (ratio > 0.8).

3. Deallocate one zone from a low-utilization tenant, increment the global pool.

4. Allocate that zone to the high-utilization tenant.

Rebalancing frequency: Perform this calculation every 100-500ms. Too frequent rebalancing introduces overhead; too infrequent causes temporary stalls. The rebalancing decision itself must be non-blocking—update the zone mapping in a lock-free data structure (e.g., RCU-style read-copy-update) so compaction threads don't block waiting for zone allocation.

Real-world scenario: A three-tenant system initially allocates 43 zones each (128 total). Tenant A enters a major compaction, consuming 40 of its 43 zones within 2 seconds. Tenant B is idle, using 5 of its 43 zones. At the next rebalancing interval, the system deallocates 10 zones from Tenant B (leaving it with 33 zones, still above its 5-zone active usage) and allocates them to Tenant A (now with 53 zones). If Tenant A's compaction completes and it drops back to 5 active zones, a subsequent rebalancing returns zones to Tenant B.

Weighted Zone Assignment and Multi-Class Fairness

Not all tenants have equal importance. Enterprise SLAs often tier tenants into classes (e.g., Premium, Standard, Basic), where Premium tenants receive preferential resource allocation. Weighted zone assignment extends dynamic partitioning to support proportional fairness across tenant classes.

Weighting mechanism: Assign each tenant a zone weight (e.g., Premium = 2.0, Standard = 1.0, Basic = 0.5). Calculate each tenant's fair share of zones:

```

fair_share_zones = (tenant_weight / sum_of_all_weights) * total_zones

```

For a 128-zone drive with one Premium tenant (weight 2.0), two Standard tenants (weight 1.0 each), and one Basic tenant (weight 0.5):

Total weight = 2.0 + 1.0 + 1.0 + 0.5 = 4.5
Premium: (2.0 / 4.5) * 128 ≈ 56.9 zones → 57 zones
Standard (each): (1.0 / 4.5) * 128 ≈ 28.4 zones → 28 zones
Basic: (0.5 / 4.5) * 128 ≈ 14.2 zones → 14 zones

During rebalancing, ensure each tenant's leased zones approach their fair share. If a tenant exceeds its fair share and has low utilization, deallocate zones. If a tenant is below its fair share and has high utilization, allocate zones.

Dynamic weight adjustment: Weights can change at runtime (e.g., a customer upgrades from Standard to Premium). Implement weight changes as soft constraints—the scheduler gradually rebalances zones over 5-10 rebalancing cycles rather than immediately reassigning zones, preventing compaction disruptions.

Prevention of Single-Tenant Zone Monopolization

Even with weighted allocation, a pathological tenant can attempt to monopolize zones by:

1. Opening many zones and writing small amounts to each (preventing zone closure).

2. Triggering compactions that generate many zone resets, blocking other tenants.

3. Deliberately delaying zone resets to keep zones in a "reset-pending" state.

Monopolization prevention mechanisms:

Zone occupancy limits: Enforce a maximum open zones per tenant threshold. If a tenant has 20 zones open and attempts to open a 21st, the request blocks until it closes an existing zone. This forces tenants to manage their zone lifecycle responsibly.

Compaction rate capping: In addition to write credit limits, implement compaction operation limits. Each tenant receives a budget of compactions per minute (e.g., 100 compactions/min). Aggressive compactions that exceed this budget are throttled at the userspace scheduler level, preventing cascading zone resets.

Zone reset timeout enforcement: If a zone remains in a "reset-pending" state for more than 100ms (a configurable threshold), the scheduler forcibly completes the reset or reassigns the zone to another tenant. This prevents a malicious tenant from intentionally stalling the zone-reset queue.

Lease revocation: If a tenant consistently violates fair-share constraints (e.g., holding 80% of zones while using only 20%), the scheduler can revoke leases and force the tenant to close zones. This is a last-resort mechanism implemented as an administrative control.

Sub-module 2.3: Asynchronous Compaction Signaling and Backpressure Propagation—Communicating Zone Saturation State to LSM Compaction Threads Without Blocking+

The final piece of the puzzle is feedback control: how does the I/O scheduler communicate zone saturation and resource pressure back to compaction threads without introducing blocking calls that could deadlock the system? This is the bridge between the I/O scheduling layer and the LSM database engine.

Zone Saturation Signaling via Atomic State Variables

Compaction threads need real-time visibility into zone availability and I/O queue depth. Blocking system calls (e.g., `poll()`, `select()`) are unacceptable because they can cause priority inversion—a low-priority compaction thread blocks waiting for I/O, preventing a high-priority tenant's thread from running. Instead, implement non-blocking, atomic zone state signals.

State machine: Each zone maintains an atomic state variable:

`ZONE_OPEN`: Actively accepting writes.
`ZONE_NEARLY_FULL`: 80% capacity reached; new writes allowed but with increased latency.
`ZONE_FULL`: 100% capacity reached; no new writes accepted.
`ZONE_RESETTING`: Reset operation in progress; no writes or reads allowed.

Compaction threads poll this state without acquiring locks. When a thread attempts to write to a `ZONE_NEARLY_FULL` zone, it can make intelligent decisions: defer the write, trigger early compaction of a different level, or migrate data to an open zone in another tenant's allocation (with fairness constraints).

Implementation: Use a lock-free atomic structure:

```

struct zone_state {

atomic state;

atomic written_bytes;

atomic open_count;

};

```

Compaction threads perform load-acquire operations (`std::atomic::load(std::memory_order_acquire)`) to read state without barriers. The I/O scheduler performs store-release operations to update state, ensuring visibility across cores without explicit locks.

Backpressure Propagation via Shared Metrics Ring Buffer

Propagating only zone state is insufficient. Compaction threads also need visibility into aggregate I/O queue depth, zone-reset queue length, and per-tenant write credit availability. Communicating this via function calls or shared memory locks introduces contention. Instead, use a lock-free ring buffer shared between the I/O scheduler and compaction threads.

Architecture: The I/O scheduler writes metrics snapshots to a ring buffer every 1-10ms:

```

struct metrics_snapshot {

uint64_t timestamp_ns;

uint16_t global_zone_reset_queue_depth;

uint16_t tenant_pending_resets;

uint32_t tenant_write_credits_available;

uint8_t zone_states[128]; // state of each zone

uint16_t write_queue_depth_per_zone[128];

};

```

The ring buffer has 128 slots (enough for ~1 second of 10ms snapshots). Compaction threads read the latest snapshot without waiting:

```c

metrics_snapshot* latest = ring_buffer[ring_buffer_head.load()];

if (latest->tenant_write_credits_available < MIN_CREDIT_THRESHOLD) {

// Defer compaction or reduce write rate

}

```

This is completely non-blocking—there are no locks, no atomic increments in the read path, and no possibility of deadlock. The tradeoff is that compaction threads see slightly stale metrics (up to 10ms old), but this is acceptable because compaction decisions operate at sub-second granularity.

Adaptive Compaction Throttling Based on Zone Saturation

The metrics ring buffer enables adaptive throttling: compaction threads adjust their behavior based on observed zone saturation without explicit scheduler intervention.

Algorithm: Compaction threads implement a local throttle factor:

```

throttle_factor = 1.0 // Initially no throttling

if (metrics.global_zone_reset_queue_depth > QUEUE_DEPTH_HIGH) {

throttle_factor = 0.5; // Reduce compaction rate by 50%

} else if (metrics.global_zone_reset_queue_depth > QUEUE_DEPTH_MEDIUM) {

throttle_factor = 0.8; // Reduce by 20%

} else {

throttle_factor = 1.0; // No throttling

}

// Before starting a compaction:

if (rand() > throttle_factor) {

// Skip this compaction cycle

sleep_ms(10);

continue;

}

```

This implements probabilistic backoff: under high zone saturation, a compaction thread skips some compactions randomly, reducing aggregate load without explicit coordination.

Deadlock Prevention via Timeout-Based Backoff

The final risk is deadlock: if all compaction threads are blocked waiting for write credits or zone availability, and the I/O scheduler thread is blocked waiting for compactions to complete (to free zones), the system stalls.

Prevention mechanism: Implement timeout-based backoff in the userspace scheduler:

1. Track the time since last compaction completion per tenant.

2. If a tenant hasn't completed a compaction in 5 seconds, assume it's deadlocked.

3. Forcibly grant emergency write credits to the tenant (bypassing fairness temporarily).

4. If the tenant still doesn't progress in another 5 seconds, trigger a zone reset override: forcibly complete any pending zone resets, even if it disrupts fairness.

This is a last-resort mechanism, but it prevents total system failure. In practice, well-tuned token bucket rates and proportional fair queuing should make this unnecessary.

Monitoring and observability: Instrument the scheduler to log all backoff events, deadlock detections, and credit overrides. These are indicators that the token bucket refill rates or zone allocation weights need adjustment. A production system should alert operators when backoff is triggered more than once per hour.

Module 3: Module 3: Custom Zone-Throttling Mechanisms Within Storage Engines

Sub-module 3.1: Integrating Zone-Reset Queue Depth Telemetry into Compaction Decision Logic—Real-Time Monitoring, Predictive Stall Detection, and Adaptive Compaction Deferral+

The fundamental challenge in multi-tenant ZNS deployments is that zone-reset queue depth—the internal hardware metric tracking pending zone management operations—remains largely opaque to userspace storage engines. When Tenant A's LSM tree compaction triggers aggressive sequential writes, those writes saturate the NVMe controller's internal zone-reset queue (typically 32–256 slots depending on hardware generation). This queue saturation creates a hidden bottleneck: subsequent zone-reset or zone-finish operations from Tenant B's workload experience microsecond-scale delays that cascade into observable latency tail events, yet no explicit error signal alerts the storage engine that congestion exists.

Real-Time Monitoring Infrastructure

Effective zone-throttling begins with instrumenting the path between userspace and hardware to expose queue depth metrics. Modern NVMe devices expose zone state information via the Zone Descriptor Extension (ZDE) in the Identify Namespace command and through Zone Management Receive (ZMR) operations. A storage engine must poll or subscribe to these metrics at sub-millisecond granularity—typically 100–500 microsecond intervals—to detect queue saturation before it propagates stalls to other tenants.

The monitoring layer must track three distinct signals:

1. Active Zone Count: The number of zones currently in implicit-open or explicit-open state. ZNS devices enforce a maximum active zone limit (often 128 zones) to constrain controller resource consumption. When this limit is approached, subsequent zone-open operations block, creating backpressure.

2. Zone-Reset Queue Occupancy: The depth of pending zone-reset or zone-finish operations within the controller. This is not directly exposed via standard NVMe commands, but can be inferred through latency anomaly detection: measure the round-trip time (RTT) of zone management commands and flag when RTT exceeds the baseline by >50%.

3. Write Stall Duration: Track the elapsed time between when a tenant issues a write command and when the hardware acknowledges it. Spikes in write stall duration—especially when correlated with zone-reset queue activity—indicate contention.

Predictive Stall Detection Algorithms

Rather than reacting to stalls after they occur, a proactive storage engine should predict queue saturation before it impacts other tenants. This requires building a statistical model of zone-reset queue behavior specific to the workload and hardware.

The simplest predictive model is an exponential weighted moving average (EWMA) of zone-reset queue depth:

```

predicted_depth = α * current_depth + (1 - α) * predicted_depth

stall_risk = predicted_depth / queue_capacity_limit

```

Where α is typically 0.3–0.5 (favoring recent observations). When stall_risk exceeds a threshold (e.g., 0.7), the storage engine enters a "pre-stall" state and begins deferring non-critical compaction work.

A more sophisticated approach uses a time-series forecasting model (e.g., ARIMA or exponential smoothing) trained on historical zone-reset latency patterns. By correlating compaction intensity (bytes written per second, number of concurrent compaction threads) with observed queue depth, the engine can build a predictive function:

```

queue_depth(t+Δt) = f(compaction_intensity, active_zone_count, tenant_write_rate)

```

When this function predicts queue_depth will exceed 80% of capacity within the next 500 microseconds, compaction is deferred or throttled.

Adaptive Compaction Deferral

Once stall risk is detected or predicted, the storage engine must adapt its compaction schedule without violating LSM-tree invariants or causing unbounded write amplification.

Adaptive deferral operates at multiple levels:

Level-0 Deferral: Defer flushing memtables to Level-0 SSTables when zone-reset queue depth is high. This temporarily increases memory pressure but avoids triggering cascading compactions.

Partial Compaction Deferral: Rather than deferring entire compaction runs, defer only the write-heavy phases. For example, in a tiered compaction strategy, defer the merge phase but allow the read-heavy selection phase to proceed.

Tenant-Aware Deferral: If multiple tenants share the same physical zones, the storage engine must coordinate with a global scheduler to defer only the tenant(s) causing queue saturation, not all tenants equally.

A concrete example: Tenant A's compaction has written 5 GB to Zone 10 and is about to write another 3 GB. The zone-reset queue depth is at 92% capacity. The engine detects this via EWMA prediction. Instead of blocking the write, it:

1. Pauses the compaction thread after the next 512 MB write.

2. Yields CPU to allow other tenants' I/O to drain the zone-reset queue.

3. Resumes after queue depth drops below 60%.

This approach maintains fairness and prevents hard stalls while keeping compaction progress steady.

---

Sub-module 3.2: Implementing Per-Tenant Compaction Budgets and Write-Rate Governors—Sliding Window Enforcement, Burst Absorption Buffers, and Cross-Tenant Fairness Contracts+

Real-Time Monitoring Infrastructure

The monitoring layer must track three distinct signals:

Predictive Stall Detection Algorithms

The simplest predictive model is an exponential weighted moving average (EWMA) of zone-reset queue depth:

```

predicted_depth = α * current_depth + (1 - α) * predicted_depth

stall_risk = predicted_depth / queue_capacity_limit

```

queue_depth(t+Δt) = f(compaction_intensity, active_zone_count, tenant_write_rate)

```

When this function predicts queue_depth will exceed 80% of capacity within the next 500 microseconds, compaction is deferred or throttled.

Adaptive Compaction Deferral

Once stall risk is detected or predicted, the storage engine must adapt its compaction schedule without violating LSM-tree invariants or causing unbounded write amplification.

Adaptive deferral operates at multiple levels:

Level-0 Deferral: Defer flushing memtables to Level-0 SSTables when zone-reset queue depth is high. This temporarily increases memory pressure but avoids triggering cascading compactions.

Partial Compaction Deferral: Rather than deferring entire compaction runs, defer only the write-heavy phases. For example, in a tiered compaction strategy, defer the merge phase but allow the read-heavy selection phase to proceed.

Tenant-Aware Deferral: If multiple tenants share the same physical zones, the storage engine must coordinate with a global scheduler to defer only the tenant(s) causing queue saturation, not all tenants equally.

1. Pauses the compaction thread after the next 512 MB write.

2. Yields CPU to allow other tenants' I/O to drain the zone-reset queue.

3. Resumes after queue depth drops below 60%.

This approach maintains fairness and prevents hard stalls while keeping compaction progress steady.

---

Sub-module 3.3: Zone-Aware Compaction Strategy Selection—Tiered Compaction Policies, Partial Compaction Strategies, and Soft-Limit vs. Hard-Limit Enforcement Modes+

Real-Time Monitoring Infrastructure

The monitoring layer must track three distinct signals:

Predictive Stall Detection Algorithms

The simplest predictive model is an exponential weighted moving average (EWMA) of zone-reset queue depth:

```

predicted_depth = α * current_depth + (1 - α) * predicted_depth

stall_risk = predicted_depth / queue_capacity_limit

```

queue_depth(t+Δt) = f(compaction_intensity, active_zone_count, tenant_write_rate)

```

When this function predicts queue_depth will exceed 80% of capacity within the next 500 microseconds, compaction is deferred or throttled.

Adaptive Compaction Deferral

Once stall risk is detected or predicted, the storage engine must adapt its compaction schedule without violating LSM-tree invariants or causing unbounded write amplification.

Adaptive deferral operates at multiple levels:

Level-0 Deferral: Defer flushing memtables to Level-0 SSTables when zone-reset queue depth is high. This temporarily increases memory pressure but avoids triggering cascading compactions.

Partial Compaction Deferral: Rather than deferring entire compaction runs, defer only the write-heavy phases. For example, in a tiered compaction strategy, defer the merge phase but allow the read-heavy selection phase to proceed.

Tenant-Aware Deferral: If multiple tenants share the same physical zones, the storage engine must coordinate with a global scheduler to defer only the tenant(s) causing queue saturation, not all tenants equally.

1. Pauses the compaction thread after the next 512 MB write.

2. Yields CPU to allow other tenants' I/O to drain the zone-reset queue.

3. Resumes after queue depth drops below 60%.

This approach maintains fairness and prevents hard stalls while keeping compaction progress steady.

---

Module 4: Module 4: Observability, SLA Guarantees, and Production Deployment Patterns

Sub-module 4.1: Instrumentation for Multi-Tenant ZNS Systems—Zone-Reset Queue Depth Metrics, Per-Tenant Latency Attribution, Crosstalk Detection Heuristics, and Distributed Tracing Strategies+

Zone-Reset Queue Depth Metrics and Hardware Observability

The NVMe ZNS controller maintains an internal finite queue for zone-reset operations, typically bounded to 32–256 slots depending on the drive firmware. When Tenant A's LSM compaction triggers a full zone reset (clearing write pointers and invalidating all data in that zone), this operation consumes a controller queue slot. If compaction cascades occur—where multiple levels of the LSM tree compact simultaneously—the zone-reset queue saturates. Subsequent reset requests from Tenant B, even on physically isolated zones, must wait in software queues, introducing microsecond-scale tail latencies that violate SLA contracts.

Exposing this metric requires direct NVMe command instrumentation. Most cloud storage systems cannot rely on block-device layer abstractions (like `iostat` or `blktrace`) because these tools aggregate across all zones and tenants, obscuring the causality chain. Instead, you must instrument at the userspace NVMe driver level—typically via SPDK (Storage Performance Development Kit) or custom ioctl handlers—to track:

Zone-reset queue occupancy (current depth and historical percentiles: p50, p99, p99.9)
Reset latency per zone (time from submission to completion)
Reset queuing delay (time spent waiting in software queue before hardware acceptance)
Per-tenant reset operation counts (resets/second by tenant ID)

Modern observability stacks export these metrics to Prometheus or similar time-series databases. A real-world example: a production Cassandra cluster running on ZNS drives observed that during compaction, zone-reset queue depth climbed from ~2 to ~180 within 50 milliseconds, while p99 read latencies for other tenants spiked from 200 microseconds to 5+ milliseconds—a 25× degradation with no error logs.

Per-Tenant Latency Attribution and Causal Inference

Distinguishing whether a latency spike originates from a tenant's own I/O or from crosstalk requires sophisticated tracing. A naive approach—measuring end-to-end latency at the application layer—conflates local queueing delays with external interference. Instead, implement multi-layer latency decomposition:

1. Application layer: Measure time from query submission to response (includes all queueing).

2. Storage engine layer: Measure time from storage read/write call to return (includes compaction interference).

3. NVMe driver layer: Measure time from NVMe command submission to completion (hardware + controller queue time).

4. Zone allocation layer: Track which tenant owns which zones and which zones experienced reset contention.

By comparing layer-to-layer deltas, you can attribute latency to specific sources. If Tenant B observes a 2ms latency increase at the NVMe driver layer while its own compaction is idle, but Tenant A is actively compacting, the increase is clearly external crosstalk.

Implement this via request-scoped tracing contexts (similar to OpenTelemetry). Each I/O operation carries metadata: tenant ID, operation type (read, write, reset), zone ID, and timestamps at each layer. When an operation completes, emit a structured log entry or trace span. Aggregate these traces by tenant and zone to build a causal graph: "Tenant A's zone reset #4521 delayed Tenant B's read #89234 by 1.3ms because zone-reset queue depth reached 128."

Crosstalk Detection Heuristics

Detecting crosstalk requires statistical inference over noisy hardware metrics. A heuristic-based approach:

Baseline establishment: During a quiet period (no tenant compaction), measure the distribution of per-operation latencies for each tenant independently. Record p99 latency as the baseline.

Anomaly detection: Continuously monitor whether a tenant's p99 latency exceeds baseline by a threshold (e.g., 50%). If yes, and the tenant's own compaction queue is idle, flag a potential crosstalk event.

Correlation analysis: Cross-reference the anomaly timestamp with zone-reset queue depth and other tenants' compaction activity. If zone-reset queue depth > 50 and another tenant initiated compaction 10–100 milliseconds prior, assign high confidence to crosstalk.

False-positive filtering: Distinguish crosstalk from legitimate causes (e.g., disk thermal throttling, network jitter in the control plane). Correlate with hardware temperature sensors, network packet loss, and CPU scheduling delays.

A production heuristic might look like:

```

if (tenant_p99_latency > baseline_p99 * 1.5) AND

(tenant_compaction_queue_depth == 0) AND

(zone_reset_queue_depth > threshold) AND

(other_tenant_compaction_active == true) THEN

confidence = HIGH

crosstalk_event = TRUE

else

confidence = MEDIUM or LOW

```

Distributed Tracing Strategies for Multi-Tenant Workflows

Distributed tracing frameworks (Jaeger, Datadog) excel at visualizing request flows across service boundaries. Adapt this for storage: each I/O operation becomes a "span" with parent-child relationships reflecting dependencies (e.g., a compaction output write depends on a read from a lower LSM level).

Tenant-scoped trace sampling: To avoid overwhelming the tracing backend, sample traces per-tenant with adaptive rates. If Tenant A exhibits anomalous latencies, increase sampling rate for its traces. If Tenant B is quiescent, sample at 0.1%.

Zone-level trace aggregation: Group spans by zone to visualize zone-level contention patterns. A trace dashboard might show: "Zone 42 experienced 847 concurrent reset requests over 5 seconds; 12% were from Tenant A, 88% from Tenant B."

Cross-tenant dependency visibility: Emit "crosstalk links" in the trace graph when one tenant's I/O is delayed by another's. This creates a visual representation of the noisy-neighbor problem, invaluable for debugging SLA violations.

In production, this instrumentation overhead typically adds 2–5% CPU cost but provides 10–50× faster incident diagnosis compared to log-based analysis.

---

Sub-module 4.2: Defining and Enforcing Multi-Tenant SLA Contracts—Latency Tail Guarantees Under Compaction Load, Zone Availability Commitments, and Penalty-Based Rate Adjustment Mechanisms+

Latency Tail Guarantees Under Compaction Load

Traditional SLA contracts specify average latency (e.g., "mean read latency < 500 microseconds") or simple percentile bounds (e.g., "p99 latency < 10 milliseconds"). These metrics fail under compaction cascades because they average across quiescent and congested periods. A more robust SLA for multi-tenant ZNS systems must distinguish baseline latency (no compaction interference) from loaded latency (under peer compaction).

Define a two-tier SLA:

Tier 1: Baseline Guarantee: When no other tenant is compacting, your read latency shall not exceed p99 baseline + 5% margin. For a 200 microsecond baseline, this guarantees ≤ 210 microseconds.

Tier 2: Loaded Guarantee: When peer tenants are compacting, your read latency shall not exceed p99 baseline * 1.5 (or an absolute cap, whichever is lower). For a 200 microsecond baseline, this allows up to 300 microseconds under load—a realistic concession reflecting hardware contention.

This two-tier model acknowledges physical reality: perfect isolation is impossible, but bounded degradation is achievable.

Enforcement mechanism: The storage engine maintains a compaction state register for each tenant (IDLE, COMPACTING_L0, COMPACTING_L1, ..., COMPACTING_PEAK). Before admitting a read request, check the current compaction state. If IDLE, apply Tier 1 latency budget; if COMPACTING_*, apply Tier 2 budget. If the read is predicted to exceed its budget (based on current queue depth and zone-reset latency), either:

1. Queue and delay the read slightly to avoid SLA violation (adds fairness).

2. Reject the read with a retriable error, signaling backpressure to the client.

3. Throttle the peer compaction (discussed below) to free up zone-reset queue capacity.

A real production example: RocksDB instances on ZNS drives can encode compaction state into a shared memory region (e.g., `/dev/shm/tenant_compaction_state`). A userspace I/O scheduler reads this region and makes admission decisions, ensuring no tenant's reads violate their SLA tier.

Zone Availability Commitments

Beyond latency, SLAs must guarantee zone availability: the fraction of time a tenant's allocated zones are responsive and not stuck in reset-in-progress states. A zone stuck in reset for 50 milliseconds means a tenant cannot write to that zone for 50 milliseconds, potentially violating write throughput SLAs.

Define zone availability as:

```

Zone Availability = 1 - (sum of reset-stuck durations) / measurement_window

```

For a 1-hour measurement window, if zones are stuck in reset for 100 milliseconds total, zone availability is 99.9972%.

SLA contract example: "Each tenant's allocated zones shall maintain ≥ 99.9% availability over any 24-hour rolling window. Availability is measured as the fraction of time zones are not in reset-in-progress states."

Enforcement: Implement a zone availability monitor that tracks reset-in-progress durations per zone per tenant. If a tenant's 24-hour rolling availability drops below the SLA threshold, trigger remediation:

1. Immediate: Pause peer compactions to allow the tenant's zones to complete resets.

2. Short-term: Increase the tenant's zone allocation (if spare zones exist) to reduce reset frequency.

3. Medium-term: Adjust compaction rate limits for peers to reduce zone-reset queue contention.

Penalty-Based Rate Adjustment Mechanisms

Reactive SLA enforcement (pausing compactions after violations occur) is crude. Instead, implement proactive rate adjustment using penalty functions.

Each tenant has a compaction rate limit (e.g., 1000 compactions/second). This limit is dynamically adjusted based on:

1. Zone-reset queue depth: If depth > 75% capacity, reduce compaction rate by 10%.

2. Peer latency violations: If another tenant's p99 latency exceeds Tier 2 by > 20%, reduce the compacting tenant's rate by 5%.

3. Zone availability: If a tenant's zone availability drops below SLA, reduce all other tenants' compaction rates by 15%.

Implement this as a penalty accumulator:

```

penalty_score(tenant_i) =

0.4 * (zone_reset_queue_depth / max_depth) +

0.3 * max(0, (peer_p99_latency - tier2_limit) / tier2_limit) +

0.3 * max(0, (sla_threshold - zone_availability) / sla_threshold)

adjusted_rate_limit(tenant_i) =

base_rate_limit * (1 - penalty_score(tenant_i))

```

If `penalty_score` reaches 0.8, a tenant's compaction rate drops to 20% of baseline, effectively throttling it to prevent further crosstalk.

Fairness consideration: Penalty adjustments must not starve any tenant indefinitely. Implement a fairness floor: no tenant's rate can drop below 5% of baseline. If zone-reset queue is saturated, fairly distribute available queue capacity proportional to tenant allocations.

In practice, update penalty scores every 100 milliseconds and adjust rate limits every 500 milliseconds to avoid thrashing. A production system might use exponential moving averages to smooth noisy metrics:

```

smoothed_queue_depth = 0.8 * smoothed_queue_depth + 0.2 * current_queue_depth

penalty_score = compute_penalty(smoothed_queue_depth, ...)

```

Multi-Tenant SLA Coordination

When multiple tenants have SLA contracts, conflicts arise: Tenant A's Tier 2 latency guarantee might require throttling Tenant B, but Tenant B also has an SLA. Resolve this via SLA priority tiers:

Tier-1 SLA (Premium): Highest priority. Latency guarantees are enforced; other tenants are throttled if necessary.
Tier-2 SLA (Standard): Medium priority. Guarantees are best-effort; may be violated if Tier-1 SLAs are at risk.
Tier-3 SLA (Best-Effort): Lowest priority. No guarantees; can be heavily throttled.

When an SLA violation is imminent, the scheduler throttles lower-priority tenants first. This ensures contractual commitments to premium customers are honored.

---

Sub-module 4.3: Production Deployment Patterns—Gradual Rollout Strategies, A/B Testing Throttling Policies, Incident Response Runbooks for Compaction Cascade Detection, and Operational Tuning of Zone-Allocation Parameters+

Gradual Rollout Strategies and Canary Deployments

Deploying a new multi-tenant ZNS I/O scheduler is risky: a bug in rate-limiting logic or zone-allocation fairness can silently degrade performance for all tenants. Gradual rollout mitigates this via canary deployments.

Canary phase (Week 1): Deploy the new scheduler to 5% of clusters (or a single test cluster). Monitor:

Per-tenant p99 latency (both read and write)
Zone-reset queue depth and reset latency
Crosstalk detection events
Compaction completion time per tenant
Zone availability per tenant

If any metric degrades by > 10%, roll back immediately. If stable for 7 days, proceed to phase 2.

Early-adopter phase (Week 2–3): Deploy to 25% of clusters. Extend monitoring to include:

SLA violation rates per tenant
Penalty score distributions
Rate-limit adjustment frequency (should be smooth, not oscillating)
CPU overhead of the scheduler

If SLA violations increase by > 5%, roll back. If stable, proceed to phase 3.

Broad deployment (Week 4+): Roll out to all clusters, but maintain a kill switch: a feature flag that disables the new scheduler and reverts to the old one within 30 seconds. This allows rapid rollback if production issues emerge.

Operational best practice: Schedule deployments during low-traffic windows (e.g., 2–4 AM) when compaction activity is typically light. Monitor the first 2 hours intensively; if no anomalies, reduce monitoring frequency.

A/B Testing Throttling Policies

Before committing to a specific penalty function or rate-limit formula, A/B test competing policies on production workloads. This requires infrastructure to run multiple policies in parallel.

Experimental setup: Partition tenants into two cohorts (A and B). Cohort A uses the current production policy; Cohort B uses a new candidate policy. Run both for 1–2 weeks, measuring:

Latency tail percentiles: p50, p99, p99.9 for reads and writes.
Compaction throughput: GB/second compacted per tenant.
SLA compliance: Fraction of time each tenant's SLA is satisfied.
Crosstalk events: Detected via the heuristics from Sub-module 4.1.
Resource utilization: CPU, memory, NVMe queue depth.

Metrics for comparison:

1. Aggregate latency: Sum of all tenants' p99 latencies. A good policy minimizes this (improves average experience).

2. Fairness: Coefficient of variation in p99 latencies across tenants. A good policy keeps this low (prevents one tenant from suffering while others thrive).

3. Compaction lag: Time from LSM level reaching full to compaction completion. A good policy minimizes this (prevents write stalls).

Statistical rigor: Use a t-test or Mann-Whitney U test to determine if differences are significant. With 100+ tenants, even small differences become statistically significant, so focus on practically significant differences (e.g., > 5% improvement in p99 latency).

Real-world example: A production cluster tested two penalty functions:

Policy A (linear): `penalty = queue_depth / max_depth`
Policy B (quadratic): `penalty = (queue_depth / max_depth) ^ 2`

Policy B reduced p99.9 latency by 8% and decreased crosstalk events by 12%, but increased compaction lag by 3%. After discussion, the team chose Policy B because tail-latency guarantees were more critical than compaction speed for their SLAs.

Incident Response Runbooks for Compaction Cascade Detection

Despite careful design, compaction cascades will occur in production. Prepare runbooks to detect and mitigate them rapidly.

Automated detection:

1. Alert trigger: If zone-reset queue depth > 80% for > 30 seconds, fire alert: "Zone-reset queue saturation detected."

2. Escalation: If saturation persists for > 2 minutes, page on-call engineer.

3. Root cause analysis: Automatically emit a diagnostic report:

Which tenant(s) are compacting?
Which zones are experiencing reset contention?
Current penalty scores for all tenants.
Latency impact on non-compacting tenants.

Manual runbook (executed by on-call engineer):

Step 1: Verify the cascade

```

Query metrics: SELECT zone_reset_queue_depth, compacting_tenants, p99_latency_by_tenant

If queue_depth > 80% and p99_latency spike detected in non-compacting tenants:

Confirm cascade is real (not a metric collection artifact)

```

Step 2: Identify the culprit

```

Identify tenant with highest compaction_rate or largest LSM tree

Check if this tenant recently triggered a full compaction (e.g., after a bulk load or schema change)

```

Step 3: Immediate mitigation

```

Option A (Preferred): Throttle the culprit tenant

Set compaction_rate_limit = 50% of current for the culprit tenant

Monitor: Does zone-reset queue depth drop within 30 seconds?

If yes, gradually increase rate back to normal over 5 minutes

Option B (If Option A insufficient): Pause peer compactions

Temporarily set compaction_rate_limit = 0 for all non-critical tenants

This frees up zone-reset queue capacity for the culprit to finish faster

Risk: Delayed compaction for other tenants; use only as last resort

Option C (If both A and B fail): Failover to backup ZNS drive

If multi-drive setup exists, migrate affected tenants to a different drive

This is a last-resort measure; coordinate with platform team

```

Step 4: Post-incident investigation

```

After cascade resolves:

1. Examine the trigger: Did a tenant perform an unexpected bulk load?

2. Review compaction policy: Should we have predicted this cascade?

3. Adjust parameters: Increase zone allocation for this tenant? Reduce rate limit?

4. Update runbook: Document this specific scenario for future reference

```

Runbook automation: Implement these steps as a Kubernetes operator or custom daemon. When the automated detection fires, the daemon can immediately execute Step 3 (throttling) without human intervention, reducing MTTR (mean time to resolution) from minutes to seconds.

Operational Tuning of Zone-Allocation Parameters

Zone allocation is the foundational parameter that determines contention risk. Tune it based on production observability.

Zone-allocation parameters:

Zones per tenant: How many zones does each tenant own? More zones = lower reset frequency per zone = lower contention risk.
Zone size: On ZNS drives, zones are typically 256 MB to 1 GB. Smaller zones = more frequent resets = higher contention risk.
Compaction concurrency: How many compactions can a tenant run in parallel? Higher concurrency = more simultaneous zone resets = higher contention risk.

Tuning procedure:

Week 1: Baseline measurement

```

For each tenant, measure:

Compaction rate (compactions/second)
Zone-reset frequency (resets/second)
Zone-reset queue depth during peak compaction
p99 latency for other tenants during this tenant's compaction

Plot: compaction_rate vs zone_reset_queue_depth

Identify: At what compaction rate does queue depth exceed 50%?

```

Week 2–3: Targeted adjustments

```

For tenants with high compaction rates and high queue depth:

Increase zone allocation by 20%

Reduce compaction concurrency by 1 level

Re-measure after 1 week

For tenants with low compaction rates and low queue depth:

Decrease zone allocation by 10% (reclaim zones for other tenants)

Increase compaction concurrency by 1 level (improve compaction throughput)

```

Week 4+: Equilibrium

```

Iterate until:

All tenants' zone-reset queue depth stays < 50% during peak compaction
All tenants' p99 latencies stay within SLA bounds
Zone utilization is balanced (no tenant has excessive idle zones)

```

Real-world tuning example: A production cluster initially allocated 16 zones per tenant (64 tenants × 16 zones = 1024 zones, fully utilizing a 1 TB ZNS drive). During peak hours, zone-reset queue depth reached 90%, causing p99 latency to spike to 15 milliseconds (violating a 10 millisecond SLA).

Solution: Reduce tenant density from 64 to 48 tenants (allocate 21–22 zones per tenant). This decreased zone-reset queue depth to 40% during peak, and p99 latency dropped to 8 milliseconds. Trade-off: 25% fewer tenants per drive, but SLA compliance improved from 94% to 99.8%.

Automated tuning: Implement a feedback controller that continuously adjusts zone allocation based on observed metrics:

```

error = target_queue_depth - current_queue_depth

adjustment = Kp * error + Ki * integral(error) + Kd * derivative(error)

new_zone_allocation = current_zone_allocation + adjustment

```

Use PID (proportional-integral-derivative) control tuning to find Kp, Ki, Kd values that stabilize queue depth without oscillation. This allows the system to self-tune as workloads change.