Repository navigation
test(e2e): hard and edge cases for compressed binlog transactions; refuse at start when the source session zone is one the driver cannot name (Spec 01.08 section 5, Spec 07.03 section 3.1.5) - #1532
Merged
Conversation
…fuse at start when the source session zone is one the driver cannot name (Spec 01.08 section 5, Spec 07.03 section 3.1.5) Re-ran this repository whole end-to-end corpus on the merged 2.11.0 head with binlog_transaction_compression=ON at the source (type suites, primary-key rebuild, replication-history modes, upgrade/downgrade/re-upgrade with GTID on and off, the DDL x DML matrix, DST fidelity, parts pressure, performance, the compression matrix) and added a hard-case harness for compression. No replication defect was found; two things came out of it. 1. tests/e2e/binlog_transaction_compression_hard.sh -- twelve hard cases, GTID on and off, value-level comparison, exit status is the verdict: SAVEPOINT / ROLLBACK TO SAVEPOINT in a multi-table transaction; a long transaction committing after short ones executed later; forty 1 MiB rows in one payload at zstd level 22; an InnoDB + MyISAM transaction (written uncompressed) followed by a compressed one; binlog_rows_query_log_events=ON; SET GLOBAL binlog_transaction_compression toggled at run time; keyless duplicates; a 1.2M-row single statement payload (~150 MB uncompressed, applied in 12-14 s on a 2 GiB heap); kill -9 while a second such payload is being applied (restart skipped 44 rows of the event in progress and re-delivered the rest; identical checksums); source max_allowed_packet lowered to 16 MiB under a 24 MiB incompressible payload (streamed regardless -- the dump connection is bounded by the replication packet limit); ADD / MODIFY / DROP COLUMN each followed by a compressed multi-statement transaction; kill -9 at a payload boundary with TRUNCATE and DROP TABLE while down. 49/49 with GTID on, 49/49 with GTID off. Spec 01.08 records the measurements. 2. ConnectionTimeZonePreflight (Spec 07.03 section 3.1.5). The temporal matrix default profile (both zone keys absent, source host in a DST zone) showed the engine failing every start with Connector/J "The server time zone value CDT is unrecognized or represents more than one time zone" and being restarted errors.max.retries times, each with a full stack trace, before stopping. The preflight reads @@GLOBAL.time_zone / @@GLOBAL.system_time_zone when database.connectionTimeZone is unset, blank or the driver keyword SERVER, resolves the effective zone with the driver own TimeUtil.getCanonicalTimeZone (so it agrees with the driver by construction), and refuses at start naming the property to set. A probe that cannot run warns and continues; non-MySQL sources are untouched. Verification: ConnectionTimeZonePreflightTest 10/10 (mutation-checked: reading time_zone instead of system_time_zone, and swallowing the driver exception, both go red), preflight family 32/32, CommitOrderVersionClampTest 18/18, validate_specs PASSED (73 specs, 850 citations), hard harness 49/49 x2 plus a standalone run from the repository path 49/49.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1531 on the 2.11.0 head: the whole end-to-end corpus of this repository was re-run with binlog_transaction_compression=ON at the source, a hard-case harness for compression was added, and the one defect the exercise surfaced is fixed. No replication defect was found.
Corpus re-run (compression on at the source in every harness)
Primary-key rebuild 26/26; base type suite 8/8; replication-history modes 148/148; upgrade/downgrade/re-upgrade GTID on 101/101 and GTID off 101/101; DDL x DML matrix 26/26; DST fidelity 15/15; performance, all five phases value-identical (51-65k rows/s); parts pressure peak 9 active parts, 0 TOO_MANY_PARTS; compression matrix GTID on 26/26, GTID off 26/26, compression-off control 26/26; temporal matrix profiles unchanged in their mismatch classes (the out-of-range clamp rows 1000-01-01 / 9999-12-31 that MySQL 8.0.46 now accepts are the documented Date32/DateTime64 range clamp of spec 07.03 section 3.3).
1. Hard-case harness: sink-connector-lightweight/tests/e2e/binlog_transaction_compression_hard.sh
Twelve cases, GTID on and off, value-level CRC per table, exit status is the verdict: SAVEPOINT / ROLLBACK TO SAVEPOINT in a multi-table transaction; a long transaction committing after short ones executed later; forty 1 MiB rows in one payload at zstd level 22; an InnoDB plus MyISAM transaction (written uncompressed by MySQL) followed by a compressed one on the same keys; binlog_rows_query_log_events=ON (Rows_query events inside the payload); SET GLOBAL binlog_transaction_compression toggled at run time with transactions on both sides; keyless duplicates; a 1.2M-row single-statement payload (about 150 MB uncompressed, applied in 12-14 s on a 2 GiB heap, RSS about 2.6 GB); kill -9 while a second such payload is being applied (the restart logs Skip 0 events / Skip 44 rows, re-delivers the rest, checksums identical); source max_allowed_packet lowered to 16 MiB under a 24 MiB incompressible payload (streamed regardless: the dump connection is bounded by the replication packet limit); ADD / MODIFY / DROP COLUMN each followed by a compressed multi-statement transaction; kill -9 at a payload boundary followed by TRUNCATE and DROP TABLE while down. Results: 49/49 with GTID on, 49/49 with GTID off, and 49/49 when run from the repository path. Spec 01.08 sections 3.1, 3.3 and 5 record the measurements.
2. ConnectionTimeZonePreflight (Spec 07.03 section 3.1.5)
The temporal matrix default profile (both zone keys absent, source host in a DST zone) fails every engine start with Connector/J: The server time zone value CDT is unrecognized or represents more than one time zone, and the engine is restarted errors.max.retries times (20, each with a full stack trace) before Replication is STOPPED. The new preflight runs after the compression preflight: when database.connectionTimeZone is unset, blank or the driver keyword SERVER, it reads @@GLOBAL.time_zone and @@GLOBAL.system_time_zone with one read-only SELECT, resolves the effective session zone with the driver own TimeUtil.getCanonicalTimeZone (so it agrees with the driver by construction: IANA ids, offsets and unambiguous abbreviations pass, ambiguous ones such as CDT do not) and refuses at start naming the property to set. A probe that cannot run warns and continues; non-MySQL sources are untouched. Live: the same default-profile run on this branch exits within seconds with the REFUSING TO START banner instead of the retry storm.
Verification
ConnectionTimeZonePreflightTest 10/10 (mutation-checked: reading time_zone instead of system_time_zone, and swallowing the driver exception, both go red), preflight family 32/32, CommitOrderVersionClampTest 18/18, scripts/validate_specs.py PASSED (73 specs, 850 citations).