Repository navigation
Support log payload compression in IoTConsensus V1 - #19244
Open
hongzhi-gao wants to merge 7 commits into
Open
hongzhi-gao wants to merge 7 commits into
hongzhi-gao wants to merge 7 commits into
Conversation
Contributor
Author
|
@Caideyipi @jt2594838 PTAL |
jt2594838
reviewed
Nov 13, 2026
jt2594838
reviewed
Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
IoTConsensus V1 replicates uncompressed logs through
syncLogEntries. Large payloads increase inter-node traffic and can cause replication backlog.This PR adds configurable log payload compression while retaining existing batching, RPCs, application, and synchronization-index advancement. Deployment requires all nodes to run the same supporting version and compression configuration. Rolling-upgrade negotiation and per-peer fallback are outside this PR's scope.
Design and implementation
Preserve batching; encode individual buffers
The compression unit is each ByteBuffer in
TLogEntry.data; the existing Batch policy is unchanged. A queued consensus request normally produces one buffer. WAL backfill may collect multiple records associated with the same consensus index into one entry, so an entry can contain multiple buffers.Each buffer carries its actual codec and uncompressed length. Compressed and raw buffers may coexist; entries containing only raw buffers omit compression metadata.
flowchart LR A[Queue / WAL backfill] --> B[Existing batching] B --> C[Encode individual buffers and retain original Batch] C --> D[Existing syncLogEntries RPC] D --> E[Validate and decode the entire request] E --> F[Existing deserialization, application and ACK] F --> G[Advance the acknowledged batch prefix]Per-buffer encoding preserves existing boundaries, avoids whole-request concatenation and splitting, and supports individual benefit checks and workspace reservations. Its tradeoffs are descriptor overhead and the inability to exploit cross-buffer redundancy. This round did not compare two native implementations and does not establish that per-buffer compression always outperforms whole-request compression.
Algorithms, configuration, and raw fallback
New startup setting, disabled by default:
data_region_iot_log_compression=NONESupported values are
NONE,LZ4,SNAPPY, andZSTD. LZ4 is the suggested initial choice when enabling compression:data_region_iot_log_compression=LZ4Internal configuration reuses TsFile
CompressionTypeand its codec factories, with a Thriftbyte codeccarryingCompressionType.serialize()directly (UNCOMPRESSED=0, SNAPPY=1, LZ4=7, ZSTD=8). The custom RPC enum and mapping are removed. The receiver usesCompressionType.deserialize()and rejects unknown or unsupported codecs before allocating decoded output; using TsFile IDs does not enable additional algorithms.Zstd uses TsFile's default level, currently the maximum level in the pinned dependency. There is no specialized Zstd-1 path or direct consensus-module dependency on Zstd JNI. GZIP and LZMA2 are not exposed because their output-bound handling and internal decompression allocations require further adaptation.
Buffers are sent raw when:
For example, a 2 KiB buffer is sent unchanged even with LZ4 enabled. The 4 KiB cutoff controls whether compression is attempted; the post-encoding check separately requires net size reduction. No public minimum-size parameter or fixed minimum-savings percentage is added.
Memory, retries, and errors
The sender creates an encoded copy without changing the original Batch or shared ByteBuffer positions. Workspace is reserved per buffer; only the actual encoded results remain retained. RPC callbacks and synchronous client-borrow/send failures release the reservation. Retries encode again from the original Batch.
Original backlog retains its existing
memorySizeaccounting. Encoded copies use the consensus memory pool; native/internal codec workspace is not separately charged to that Java pool.Before constructing consensus requests or deserializing them, the receiver validates descriptors, codecs, lengths, and total decoded size, then decodes the entire request. Only a fully decoded request enters the original application path. Compression failures return
logCompressionErrorand cannot complete the Batch or advance the synchronization index. Later ACKs cannot advance past the failed prefix.A Batch that has sent compressed payloads no longer takes the existing
TApplicationExceptionskip path, preventing protocol errors from incorrectly acknowledging it.The decoded-size limit reuses the existing RPC frame configuration as a per-request bound, not a global receive-memory quota. Decoding remains available when local sending is configured as NONE. There is no independent CRC: modified content that still decodes to the declared length may not be detected by this layer.
WAL, TsFile, and snapshot formats are unchanged. Optional Thrift fields do not make compressed payloads safe for old receivers; version consistency remains an operational requirement. The new byte field is also incompatible with the earlier experimental enum/i32 metadata; nodes running those revisions must not be mixed.
Performance evaluation
Environment and method
Tests used iot-benchmark
62e0dc1and database revision434292b65c5, with CRC removed and all codecs using TsFile defaults. ZSTD is not Zstd-1. The subsequent master merge and codec-ID wire-format simplification were not rebenchmarked; the results below remain measurements of434292b65c5.Each fresh cluster received a separate 512-request mixed warm-up. Two topologies × four modes × three profiles × three repetitions produced 72 measured phases. Mode and profile order were randomized; other jobs remained running on the shared servers. DataNodes ran native binaries without an experimental agent; a client-side observer recorded supplementary replication timings.
Official iot-benchmark results
The primary metrics come directly from the benchmark's reported throughput and request latency: throughput counts points/second, while latency measures a write request in milliseconds.
All completed successful repetitions are included, without excluding official scores because of WAL waits. Values are three-run arithmetic means with
[minimum, maximum]. P99 summaries are means of individual run P99 values, not percentiles recomputed from pooled requests.Write throughput: million points/second
Request P99: milliseconds
Server load and replication payload
These supplementary measurements cover the first client write through replica catch-up, rather than the official benchmark timing window. CPU-s is the three-run mean cumulative CPU consumption of all DataNodes. RSS is the largest sampled value for any individual DataNode across the three runs.
Payload ratios include raw fallbacks but exclude Thrift/TCP overhead. Lower RSS samples do not establish that compression saves memory; backlog and GC timing also contribute. Resource sampling and clock alignment introduce uncertainty, so small CPU differences are not treated as established improvements.
Findings
Validation and limitations
test-compile, covering codec round trips, real bidirectional asynchronous Thrift, whole-request rejection on invalid input, length limits, per-buffer memory fallback, cleanup on failures, and synchronization-prefix protection.WORKLOAD_BUFFER_SIZE=100input reuse. Short phases and shared-host variability make these fixed-workload regression comparisons, not sustained maximum-throughput measurements or universal production guarantees.