Repository navigation
Tags
Tags
1.6.1: quiver mirror into another cluster under the copy's ids: both … …tables described as TiKV keeps them at the versions read and written (Info, from the meta keys; the status port no longer asked), keys and rows renumbered (Renumber: index ids in keys, column ids in rows of the second format re-sorted and in what indexes keep to restore, dropped columns' values and the row checksum left out, padding redone), keys of an index the source does not have public left behind, a row written before a column was added with a default refused (Missing) instead of read as NULL, a schema being changed refused; checked live between two playgrounds after dropped and added columns and indexes; the head of a chain that failed by itself tells the tail Leave before closing its side (it closed first, or raced the thread that took the failure, and the tail waited out the whole reconnect window); a row written before a column was added with a default gets that default written in, as TiDB writes it, instead of the copy being refused (Cell, every type checked byte for byte against rows TiDB rewrote; refused only for a JSON default other than null, a TIMESTAMP of TiDB before 3.0 and other character sets); a partitioned table copied partition by partition, each into the copy's partition of its name (Info reads the partitions and whether they are being rearranged, Renumber moves every key to its partition's id in the copy; one with a global index refused), checked live into another cluster and the same for RANGE, HASH, LIST COLUMNS and KEY tables after truncated, dropped and added partitions
1.6.0: TiDB 8.5 as a source and a target, and quiver mirror copying T… …iDB to TiDB as TiKV keys, within a cluster or into another; the README's TiDB section with the mirror against IMPORT INTO FROM SELECT drawn (assets/mirror.svg), the docs site naming TiDB
1.4.0: values masked as rows are read (--mask, --mask-secret, --mask-… …strict), a masking policy language of S-expressions with selectors, keep, strict, word lists and dates (--mask-script, expr:, quiver mask), maps in the quiverfile, the user's own statements after the load (--after-load), and documentation generated from the binary (quiver docs)
1.3.9: quiver builds the secondary keys it can order by default; enco… …ding test without border seeds - --tablespace-keys defaults to quiver: on the colleague's copy it was 15% faster than server on 1.3.5 and 30% on 1.3.7, every copy equal to its source. server keeps every key for ALTER. - The encoding controller's property also takes a rung whose next rung toward the bottleneck gains under 15%, a step the model's noise hides: CI of the 1.3.7 tag failed on seed 9757516 (a rung at 79.9% of the best, its neighbour 5.2% better). 300 seeds pass; no rollback, a step without a signal, the wrong direction and no doubling of silence are still caught. - README: keys quiver builds, the default, and --keyless-read.
1.3.8: a table without a primary key is read in one pass again, built… … on every renderer - On the colleague's server (1.3.7) reading through another index cost the source ~7x: 121-135 reader-seconds against 17 on 1.3.5, reading ended at 17.6-20.4 s instead of 12-14.5 s, so the shorter tail of table_049 (11 -> 7.1 s) bought nothing. Such a table is read whole again by default; --keyless-read ranges keeps the 1.3.6 ranges. - A part of a table whose runs number their own rows takes its batches into whichever run a renderer holds, as it renders: the lowest of 16 free slots, each with its run, closed every 32 MiB so secondary records sort while the table is still read. One reader's pass is no longer built on one thread. - After a dump or copy the CLI prints the five tables read last: parts, when their reading began and ended, reading and waiting. - cw on the scratch 8.4.11, keys built by quiver, 2 pairs: reading summed 1.7 s against 10 s with ranges, reading ended at 1.0 s against 1.4 s, the t049 tail 1.15 s against 1.3 s.
1.3.7: a table with a primary key is read in its order again; seconda… …ry merges split by whole records - 1.3.6 read a table whose primary key cannot bound chunks (a DATE column under --zero-dates keep) through FORCE INDEX of another index, so rows came out of clustered order and copy --ingest tablespace failed with Tablespace Out_of_order (a colleague's table: PRIMARY KEY (date, int), KEY (int, date)). Such a table is read whole again. A primary key with a usable UNIQUE index failed the same way since before 1.3.5 (UNIQUE ranges without ORDER BY) and is read whole too. The key choice is Provider.key, under a property test against a model of the choice. - A merge range now ends at a record of the index, not at a 14-byte key prefix, and ranges hold equal numbers of records. Equal prefixes could not be split before: in a text(10) key every https://www.… URL is the same value, told apart only by DB_ROW_ID, and all of them merged on one thread (1.5M rows, 16 threads: the largest range 1.39 → 1.03 of the mean). - A part's sorted records go to the scratch file in one piece, and the merge reads them up to 32 pages at a time, at most 64 MiB of buffers in all: 15k reads of 16 KiB → 512 on 3M rows. - bench/ibd builds and checks a table like the colleague's table_049 without a server. finish on 3M rows, 64 parts: 2 threads 0.40 → 0.365 s, 8 threads 0.27–0.35 → 0.16–0.23 s, 90% URLs on 8 threads 0.45–0.51 → 0.30–0.37 s.
1.3.6: a table without a primary or unique key is read in parallel th… …rough another index - A table with neither key was read by one query on one reader: on a colleague's copy a 10M-row table took ~ 8359 9 s while the other readers idled. Such a table is now chunked by ranges of the NOT NULL first column of another visible BTREE index (whole column, integer first, then the highest cardinality); ranges are read with FORCE INDEX and no ORDER BY. - The wire frame is v8: a table's key may be Indexed, so agents and collectors must both be 1.3.6.
1.3.5: copy builds the secondary indexes of tables without a primary … …key and text keys of _bin collations; deadlocks retried after pauses; built files checked on several threads - --tablespace-keys quiver now keeps in the file the plain keys and UNIQUE keys with a nullable column of a table without a primary key: their records end with the row's DB_ROW_ID, which the build writes. A UNIQUE key over NOT NULL columns, which InnoDB would make the clustered index, and the UNIQUE keys after it are still added by the server. - Text keys in utf8mb3_bin, utf8mb4_bin and latin1_bin (compared as if padded with spaces) and in utf8mb4_0900_bin (bytes) are built in the file; a text prefix counts characters, as the server. - A deadlock or lock wait timeout (1213, 1205) is retried up to 8 times after a random pause of up to 10 ms x 2^k, at most 1 s; before, 5 times at once (a run failed adding a foreign key). - A built file is checked on several threads, each level of each tree in slices. - The largest tables' tails go first, and the threads finishing and checking built files come from one budget for the whole copy (--build-threads).
1.3.4: tables without a primary key build again; a few threads instea… …d of one per table; 1.3.2's defaults, the new ways behind options - copy --ingest tablespace failed with Tablespace Out_of_order on a table without a primary key but with a UNIQUE key over NOT NULL columns: put back into the empty table, that key became its clustered index, and the rows are not read in its order. Such a table now gets no keys in the file, and a table InnoDB still clusters by a unique key goes through LOAD DATA (Clustered_by). - copy prepared each table, and imported each built table, on a thread of its own; declared all at once, 313 tables made some 350 threads, and every garbage collection stopped them all, which on a 512-CPU host took 7 times the CPU of 1.3.2. A fixed pool now prepares tables (up to 4 threads) and another imports them and adds keys (a thread per loader). - The defaults are 1.3.2's again: the server adds secondary keys after the import, the largest chunks are read first, a table is declared just before its rows. --tablespace-keys quiver, --read-order largest-tables and --declare-tables all-at-once turn on what 1.3.3 did by default, one by one.
PreviousNext