Repository navigation
[CSV 4/9] Word-at-a-time field seek and warp-aggregated validity and type counts in the CSV reader - #19
Conversation
The type detection and decode kernels process one row per thread, and
`seek_field_end` reads each row one character at a time. The loads of a
warp are therefore scattered over 32 rows, and each character costs a
separate memory transaction.
Add `seek_field_end_by_words`, which returns the same position as
`seek_field_end` (without escape characters) but loads the row in
aligned 8-byte words, and use it in both kernels:
- In a field that does not start with a quote, only a delimiter, a
terminator or a '\r' can end the field. The bytes of the word that
may hold one of them are flagged with the SWAR zero-byte test on the
word XOR the broadcast character. That test flags every occurrence
(it may also flag other bytes), and the flags are checked in
increasing order with the exact test of `seek_field_end` ('\r' only
ends a field before '\n'), so the first flag that passes is the
position `seek_field_end` returns, and false flags fail the test.
- The characters of a quoted field are run through the same quote
state machine as in `seek_field_end`, from the loaded word.
- Multi-character delimiters (delim_whitespace) keep using
`seek_field_end`.
Only words that lie entirely within [data begin, row end) are loaded,
whatever the alignment of the data, so no memory outside of the data
is read even for unaligned or unpadded buffers; the characters of the
row outside of such words are read one at a time. (The reader parses
its own aligned copy of the data for now, so the lower bound matters
for sources that would be parsed in place.) A word may start in
the previous row, whose bytes the decode kernel may be rewriting (in
place unescaping); those bytes are never used, as the documented
invariant of the decode kernel requires.
Median kernel times (nsys, RTX PRO 6000, 256MB benchmarks):
benchmark kernel before after
REALISTIC TAXI decode 5.64 ms 3.90 ms
REALISTIC LOGS decode 1.62 ms 1.25 ms
REALISTIC ANALYTICS decode 5.57 ms 4.12 ms
QUOTING QUOTE_0_PCT decode 9.38 ms 6.53 ms
QUOTING QUOTE_25_PCT decode 4.72 ms 3.02 ms
QUOTING QUOTE_100_PCT decode 1.91 ms 1.22 ms
INFERENCE ALL_INTEGRAL, 64 detection 11.36 ms 8.89 ms
INFERENCE ALL_FLOAT, 8 detection 6.61 ms 5.38 ms
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`batch_count_set_bits`, which backs `cudf::batch_null_count`, indexes the bitmasks with the second dimension of the grid, whose size is limited to 65535, so counting more bitmasks failed with cudaErrorInvalidValue. Launch the kernel once per batch of at most 65535 bitmasks. Nothing changes for fewer bitmasks. The CSV reader is about to count the nulls of its columns with this function, and CSV files can have more columns than that. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The decode kernel set the validity bit of each valid field with its own atomicOr, and counted the valid fields of each column with an atomicAdd per field. The 32 lanes of a warp decode 32 consecutive rows that start at a multiple of 32, so their bits of a column all lie in the same mask word and their atomics serialize on it; the counter of a column is a single address that every warp of the grid updates for every field. Group the lanes that mark the same column with __match_any_sync and let the lowest of them set all of their bits with one atomicOr (the bit of a row is its lane index). The lanes that are active together normally mark the same column, but that is not guaranteed after divergence, so they are grouped by column. The atomic is still required because different groups of the same warp can update the word at different times. Stop counting in the kernel: after decoding, count the nulls of the non-string columns from their masks with cudf::batch_null_count. This is exact because the masks are zero-initialized, bits are only set for the valid fields of rows in [0, num_records), and exactly that range is counted. String columns get their null counts when they are built from the decoded string pairs, as before, so their masks are not counted. The count replaces the copy of the kernel's counters to the host, so the number of synchronizations does not change. A test reads a table with more columns than the batched count can index in one grid (65535), which the previous commit enables. Median time of the decode kernel (nsys, RTX PRO 6000, 256MB benchmarks): benchmark before after INFERENCE ALL_INTEGRAL, 8 4.24 ms 2.26 ms INFERENCE ALL_INTEGRAL, 64 7.51 ms 4.36 ms INFERENCE MIXED, 8 3.83 ms 2.26 ms INFERENCE MIXED, 64 3.72 ms 2.97 ms QUOTING QUOTE_0_PCT 6.53 ms 4.33 ms QUOTING QUOTE_25_PCT 3.02 ms 2.31 ms REALISTIC TAXI 3.90 ms 3.39 ms REALISTIC LOGS 1.25 ms 1.10 ms Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The type detection kernel incremented a member of the column's column_type_histogram with a global atomicAdd in each branch of its classification. The counters of a column are a few addresses that every thread of the grid updates for each row, so these atomics serialize. Classify each field into a `field_class`, which indexes an array of `num_field_classes` counters per column, and increment the counter in one place. The lanes of a warp that increment the same counter at the same time are grouped with __match_any_sync, and the lowest of them adds their number with a single atomic. The counts are unchanged. Integers are classified by the new classify_integral_field, which holds the logic of infer_integral_field_counter; the latter is now a thin wrapper around it for the JSON type inference. The histograms are assembled on the host from the counters, so the device code no longer depends on the layout of column_type_histogram. A new test decides column types with single fields in later blocks of rows and with the number of NA fields, so that a lost or miscounted increment changes an inferred type. Median time of the type detection kernel (nsys, RTX PRO 6000, 256MB CSV_READER_TYPE_INFERENCE_NVBENCH with inference): profile, columns before after ALL_INTEGRAL, 8 2.77 ms 1.85 ms ALL_INTEGRAL, 64 8.00 ms 5.67 ms MIXED, 64 11.48 ms 5.92 ms ALL_STRING, 8 8.68 ms 1.15 ms ALL_STRING, 64 6.08 ms 2.03 ms ALL_FLOAT, 8 5.38 ms 5.73 ms (see the next commit) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Narrow tables have few type detection counters, and every warp of the grid updates them, so even warp-aggregated global atomics contend on them. This is worst when the lanes of a warp reach their increments at different times, as with variable-length floating point fields. For tables of up to 32 inferred columns, each block counts in shared memory, zeroed on entry, and adds its non-zero counts to the global counters at the end; threads without a row now stay until then instead of returning early. The counts are exact either way. The kernel is a template on where the counters are, rather than taking a runtime flag. With the counters known to be in shared memory, the compiler emits shared memory atomics (ATOMS in the SASS of this version); in a build that selected the counters with a runtime flag, the increment compiled to a generic atomic (ATOM.E.ADD), since the counter pointer can be in either address space. The global memory instance also keeps its early return and has no block barriers. The threshold only affects speed, and was measured on one GPU (RTX PRO 6000 Blackwell). Larger shared memory allocations per block made the kernel slower than counting in global memory (64 columns: 8.9 ms instead of 5.7 ms; 32 columns with the allocation padded to 2KB: 8.3 ms instead of 3.6 ms), likely because the kernel, which reads rows through the L1 cache, then gets a larger shared memory carveout. Wider tables spread their updates over enough counters that global memory atomics do not contend much on them. Median time of the type detection kernel (nsys, 256MB CSV_READER_TYPE_INFERENCE_NVBENCH with inference, num_cols overridden for 16 and 32): profile, columns global shared ALL_FLOAT, 8 5.6 ms 2.8 ms ALL_FLOAT, 32 4.9 ms 3.5 ms ALL_INTEGRAL, 16 6.4 ms 3.3 ms ALL_INTEGRAL, 32 5.5 ms 3.6 ms MIXED, 16 4.9 ms 3.2 ms ALL_INTEGRAL, 8 1.76 ms 1.88 ms Tables with 8 integer columns get slightly slower (ALL_INTEGRAL and MIXED: 2-3% end to end), which the gains above outweigh. The tests that decide column types with single fields in later blocks now also cover 32 and 33 columns, on both sides of the threshold, and fail if a block's shared counts are not added to the global counters. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Note Re-measured cleanly (2026-09-25); these numbers replace the earlier ones in this comment. The earlier numbers were taken while other validation jobs (tests, sanitizers, fuzzers) ran on the same host, against a baseline measured hours earlier, so the pinned-host, pageable-host and file results in particular could be skewed. Methodology (clean re-measurement). All ten builds of the stack (base and PRs 1–9) were measured in one session on the same GPU (host GPU 1), in two interleaved rounds (base, PR 1, …, PR 9, then again), so drift affects every build equally. Each benchmark run started only when no other process was using the GPU and no build/test/sanitizer/fuzzer was running on the host in our environment, and would have been re-run if anything appeared during it; all 160 runs were clean on the first attempt. Each number below is the best of the two rounds per side. CSV reader benchmarks: PR 4/9 (word-at-a-time seek, warp-aggregated validity and type counts)This PR Speedup vs base (cumulative), by input source
Speedup vs parent PR (this PR's own contribution)
Sources: the Full results: every CSV reader benchmark config
|
| Benchmark | Config | Base | Parent | This PR | vs base | vs parent |
|---|---|---|---|---|---|---|
| csv_read_column_selection | column_selection=ALL, num_chunks=1, row_selection=ALL | 2.63 GiB/s | 3.08 GiB/s | 3.44 GiB/s | 1.31× | 1.12× |
| csv_read_column_selection | column_selection=ALTERNATE, num_chunks=1, row_selection=ALL | 1.45 GiB/s | 1.63 GiB/s | 1.80 GiB/s | 1.24× | 1.11× |
| csv_read_column_selection | column_selection=FIRST_HALF, num_chunks=1, row_selection=ALL | 1.49 GiB/s | 1.69 GiB/s | 1.85 GiB/s | 1.24× | 1.10× |
| csv_read_column_selection | column_selection=SECOND_HALF, num_chunks=1, row_selection=ALL | 1.49 GiB/s | 1.69 GiB/s | 1.86 GiB/s | 1.24× | 1.10× |
| csv_read_data_type | data_type=DECIMAL, io=DEVICE_BUFFER | 15.33 GiB/s | 23.72 GiB/s | 34.56 GiB/s | 2.25× | 1.46× |
| csv_read_data_type | data_type=DURATION, io=DEVICE_BUFFER | 12.78 GiB/s | 18.73 GiB/s | 32.63 GiB/s | 2.55× | 1.74× |
| csv_read_data_type | data_type=FLOAT, io=DEVICE_BUFFER | 18.66 GiB/s | 23.27 GiB/s | 26.45 GiB/s | 1.42× | 1.14× |
| csv_read_data_type | data_type=INTEGRAL, io=DEVICE_BUFFER | 17.43 GiB/s | 23.30 GiB/s | 37.60 GiB/s | 2.16× | 1.61× |
| csv_read_data_type | data_type=STRING, io=DEVICE_BUFFER | 4.68 GiB/s | 25.79 GiB/s | 26.88 GiB/s | 5.74× | 1.04× |
| csv_read_data_type | data_type=TIMESTAMP, io=DEVICE_BUFFER | 12.37 GiB/s | 17.60 GiB/s | 26.20 GiB/s | 2.12× | 1.49× |
| csv_read_io | io=FILEPATH | 4.55 GiB/s | 6.47 GiB/s | 7.09 GiB/s | 1.56× | 1.10× |
| csv_read_io | io=HOST_BUFFER | 3.38 GiB/s | 3.87 GiB/s | 3.85 GiB/s | 1.14× | 0.99× |
| csv_read_row_selection | column_selection=ALL, num_chunks=1, row_selection=BYTE_RANGE | 2.56 GiB/s | 2.86 GiB/s | 3.19 GiB/s | 1.25× | 1.11× |
| csv_read_row_selection | column_selection=ALL, num_chunks=1, row_selection=NROWS | 2.54 GiB/s | 2.87 GiB/s | 3.32 GiB/s | 1.30× | 1.16× |
| csv_read_row_selection | column_selection=ALL, num_chunks=1, row_selection=SKIPFOOTER | 2.67 GiB/s | 3.10 GiB/s | 3.61 GiB/s | 1.35× | 1.17× |
| csv_read_row_selection | column_selection=ALL, num_chunks=8, row_selection=BYTE_RANGE | 1.93 GiB/s | 2.48 GiB/s | 2.66 GiB/s | 1.38× | 1.07× |
| csv_read_row_selection | column_selection=ALL, num_chunks=8, row_selection=NROWS | 0.71 GiB/s | 0.76 GiB/s | 0.78 GiB/s | 1.09× | 1.02× |
| csv_read_row_selection | column_selection=ALL, num_chunks=8, row_selection=SKIPFOOTER | 0.45 GiB/s | 0.47 GiB/s | 0.47 GiB/s | 1.05× | 1.00× |
CSV_READER_QUOTING_IO_NVBENCH (12 configs)
| Benchmark | Config | Base | Parent | This PR | vs base | vs parent |
|---|---|---|---|---|---|---|
| csv_read_quoting_io | data_size_mb=256, io=DEVICE_BUFFER, quote_pct=QUOTE_0_PCT | 17.11 GiB/s | 23.33 GiB/s | 37.03 GiB/s | 2.16× | 1.59× |
| csv_read_quoting_io | data_size_mb=256, io=DEVICE_BUFFER, quote_pct=QUOTE_100_PCT | 4.87 GiB/s | 24.59 GiB/s | 25.96 GiB/s | 5.33× | 1.06× |
| csv_read_quoting_io | data_size_mb=256, io=DEVICE_BUFFER, quote_pct=QUOTE_25_PCT | 12.75 GiB/s | 32.74 GiB/s | 42.03 GiB/s | 3.30× | 1.28× |
| csv_read_quoting_io | data_size_mb=256, io=FILEPATH, quote_pct=QUOTE_0_PCT | 5.25 GiB/s | 6.93 GiB/s | 7.38 GiB/s | 1.41× | 1.06× |
| csv_read_quoting_io | data_size_mb=256, io=FILEPATH, quote_pct=QUOTE_100_PCT | 4.69 GiB/s | 13.53 GiB/s | 13.32 GiB/s | 2.84× | 0.98× |
| csv_read_quoting_io | data_size_mb=256, io=FILEPATH, quote_pct=QUOTE_25_PCT | 6.47 GiB/s | 11.03 GiB/s | 11.40 GiB/s | 1.76× | 1.03× |
| csv_read_quoting_io | data_size_mb=256, io=HOST_BUFFER, quote_pct=QUOTE_0_PCT | 3.80 GiB/s | 4.03 GiB/s | 4.27 GiB/s | 1.12× | 1.06× |
| csv_read_quoting_io | data_size_mb=256, io=HOST_BUFFER, quote_pct=QUOTE_100_PCT | 4.18 GiB/s | 9.08 GiB/s | 9.28 GiB/s | 2.22× | 1.02× |
| csv_read_quoting_io | data_size_mb=256, io=HOST_BUFFER, quote_pct=QUOTE_25_PCT | 4.99 GiB/s | 6.82 GiB/s | 6.85 GiB/s | 1.37× | 1.00× |
| csv_read_quoting_io | data_size_mb=256, io=PINNED_BUFFER, quote_pct=QUOTE_0_PCT | 9.57 GiB/s | 11.45 GiB/s | 13.90 GiB/s | 1.45× | 1.21× |
| csv_read_quoting_io | data_size_mb=256, io=PINNED_BUFFER, quote_pct=QUOTE_100_PCT | 5.40 GiB/s | 18.59 GiB/s | 19.45 GiB/s | 3.60× | 1.05× |
| csv_read_quoting_io | data_size_mb=256, io=PINNED_BUFFER, quote_pct=QUOTE_25_PCT | 9.77 GiB/s | 17.88 GiB/s | 20.29 GiB/s | 2.08× | 1.13× |
CSV_READER_QUOTING_NVBENCH (6 configs)
| Benchmark | Config | Base | Parent | This PR | vs base | vs parent |
|---|---|---|---|---|---|---|
| csv_read_quoting | data_size_mb=256, quote_pct=QUOTE_0_PCT | 17.51 GiB/s | 24.55 GiB/s | 38.28 GiB/s | 2.19× | 1.56× |
| csv_read_quoting | data_size_mb=256, quote_pct=QUOTE_100_PCT | 4.80 GiB/s | 25.28 GiB/s | 27.23 GiB/s | 5.68× | 1.08× |
| csv_read_quoting | data_size_mb=256, quote_pct=QUOTE_25_PCT | 13.88 GiB/s | 33.07 GiB/s | 42.54 GiB/s | 3.06× | 1.29× |
| csv_read_quoting | data_size_mb=64, quote_pct=QUOTE_0_PCT | 14.10 GiB/s | 18.42 GiB/s | 26.77 GiB/s | 1.90× | 1.45× |
| csv_read_quoting | data_size_mb=64, quote_pct=QUOTE_100_PCT | 1.37 GiB/s | 8.34 GiB/s | 8.61 GiB/s | 6.30× | 1.03× |
| csv_read_quoting | data_size_mb=64, quote_pct=QUOTE_25_PCT | 4.57 GiB/s | 20.47 GiB/s | 21.62 GiB/s | 4.73× | 1.06× |
CSV_READER_REALISTIC_IO_NVBENCH (12 configs)
| Benchmark | Config | Base | Parent | This PR | vs base | vs parent |
|---|---|---|---|---|---|---|
| csv_read_realistic_io | data_size_mb=256, io=DEVICE_BUFFER, profile=ANALYTICS | 25.28 GiB/s | 37.00 GiB/s | 43.83 GiB/s | 1.73× | 1.18× |
| csv_read_realistic_io | data_size_mb=256, io=DEVICE_BUFFER, profile=LOGS | 27.19 GiB/s | 60.08 GiB/s | 68.82 GiB/s | 2.53× | 1.15× |
| csv_read_realistic_io | data_size_mb=256, io=DEVICE_BUFFER, profile=TAXI | 18.91 GiB/s | 34.85 GiB/s | 45.69 GiB/s | 2.42× | 1.31× |
| csv_read_realistic_io | data_size_mb=256, io=FILEPATH, profile=ANALYTICS | 6.63 GiB/s | 8.25 GiB/s | 8.96 GiB/s | 1.35× | 1.09× |
| csv_read_realistic_io | data_size_mb=256, io=FILEPATH, profile=LOGS | 9.85 GiB/s | 14.16 GiB/s | 15.48 GiB/s | 1.57× | 1.09× |
| csv_read_realistic_io | data_size_mb=256, io=FILEPATH, profile=TAXI | 6.37 GiB/s | 8.55 GiB/s | 9.73 GiB/s | 1.53× | 1.14× |
| csv_read_realistic_io | data_size_mb=256, io=HOST_BUFFER, profile=ANALYTICS | 4.83 GiB/s | 4.79 GiB/s | 4.92 GiB/s | 1.02× | 1.03× |
| csv_read_realistic_io | data_size_mb=256, io=HOST_BUFFER, profile=LOGS | 7.16 GiB/s | 7.98 GiB/s | 8.78 GiB/s | 1.23× | 1.10× |
| csv_read_realistic_io | data_size_mb=256, io=HOST_BUFFER, profile=TAXI | 4.77 GiB/s | 5.07 GiB/s | 5.43 GiB/s | 1.14× | 1.07× |
| csv_read_realistic_io | data_size_mb=256, io=PINNED_BUFFER, profile=ANALYTICS | 12.58 GiB/s | 14.93 GiB/s | 16.09 GiB/s | 1.28× | 1.08× |
| csv_read_realistic_io | data_size_mb=256, io=PINNED_BUFFER, profile=LOGS | 17.05 GiB/s | 25.59 GiB/s | 26.92 GiB/s | 1.58× | 1.05× |
| csv_read_realistic_io | data_size_mb=256, io=PINNED_BUFFER, profile=TAXI | 11.20 GiB/s | 15.19 GiB/s | 16.91 GiB/s | 1.51× | 1.11× |
CSV_READER_REALISTIC_NVBENCH (9 configs)
| Benchmark | Config | Base | Parent | This PR | vs base | vs parent |
|---|---|---|---|---|---|---|
| csv_read_realistic | data_size_mb=1024, profile=ANALYTICS | 25.11 GiB/s | 40.80 GiB/s | 49.03 GiB/s | 1.95× | 1.20× |
| csv_read_realistic | data_size_mb=1024, profile=LOGS | 31.88 GiB/s | 69.84 GiB/s | 81.58 GiB/s | 2.56× | 1.17× |
| csv_read_realistic | data_size_mb=1024, profile=TAXI | 23.43 GiB/s | 39.91 GiB/s | 52.73 GiB/s | 2.25× | 1.32× |
| csv_read_realistic | data_size_mb=256, profile=ANALYTICS | 25.24 GiB/s | 37.14 GiB/s | 43.96 GiB/s | 1.74× | 1.18× |
| csv_read_realistic | data_size_mb=256, profile=LOGS | 27.37 GiB/s | 60.37 GiB/s | 69.43 GiB/s | 2.54× | 1.15× |
| csv_read_realistic | data_size_mb=256, profile=TAXI | 19.01 GiB/s | 35.43 GiB/s | 45.63 GiB/s | 2.40× | 1.29× |
| csv_read_realistic | data_size_mb=512, profile=ANALYTICS | 25.64 GiB/s | 39.44 GiB/s | 47.23 GiB/s | 1.84× | 1.20× |
| csv_read_realistic | data_size_mb=512, profile=LOGS | 31.05 GiB/s | 67.20 GiB/s | 77.72 GiB/s | 2.50× | 1.16× |
| csv_read_realistic | data_size_mb=512, profile=TAXI | 22.88 GiB/s | 38.22 GiB/s | 50.45 GiB/s | 2.20× | 1.32× |
CSV_READER_SCALE_NVBENCH (5 configs)
| Benchmark | Config | Base | Parent | This PR | vs base | vs parent |
|---|---|---|---|---|---|---|
| csv_read_scale | data_size_mb=1024 | 3.54 GiB/s | 3.66 GiB/s | 4.05 GiB/s | 1.14× | 1.11× |
| csv_read_scale | data_size_mb=2048 | 3.52 GiB/s | 3.85 GiB/s | 4.06 GiB/s | 1.15× | 1.05× |
| csv_read_scale | data_size_mb=256 | 3.59 GiB/s | 3.62 GiB/s | 3.99 GiB/s | 1.11× | 1.10× |
| csv_read_scale | data_size_mb=4096 | 3.38 GiB/s | 3.71 GiB/s | 3.95 GiB/s | 1.17× | 1.06× |
| csv_read_scale | data_size_mb=512 | 3.60 GiB/s | 3.65 GiB/s | 4.03 GiB/s | 1.12× | 1.11× |
CSV_READER_TYPE_INFERENCE_IO_NVBENCH (64 configs)
| Benchmark | Config | Base | Parent | This PR | vs base | vs parent |
|---|---|---|---|---|---|---|
| csv_read_with_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=64, type_profile=ALL_FLOAT | 10.11 GiB/s | 13.29 GiB/s | 17.94 GiB/s | 1.77× | 1.35× |
| csv_read_with_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=64, type_profile=ALL_INTEGRAL | 9.02 GiB/s | 11.63 GiB/s | 21.21 GiB/s | 2.35× | 1.82× |
| csv_read_with_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=64, type_profile=ALL_STRING | 4.40 GiB/s | 15.06 GiB/s | 22.05 GiB/s | 5.01× | 1.46× |
| csv_read_with_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=64, type_profile=MIXED | 6.59 GiB/s | 9.82 GiB/s | 17.18 GiB/s | 2.61× | 1.75× |
| csv_read_with_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=8, type_profile=ALL_FLOAT | 12.31 GiB/s | 16.49 GiB/s | 22.54 GiB/s | 1.83× | 1.37× |
| csv_read_with_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=8, type_profile=ALL_INTEGRAL | 21.14 GiB/s | 28.36 GiB/s | 39.96 GiB/s | 1.89× | 1.41× |
| csv_read_with_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=8, type_profile=ALL_STRING | 11.91 GiB/s | 18.97 GiB/s | 50.17 GiB/s | 4.21× | 2.65× |
| csv_read_with_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=8, type_profile=MIXED | 21.15 GiB/s | 28.38 GiB/s | 39.92 GiB/s | 1.89× | 1.41× |
| csv_read_with_inference_io | data_size_mb=256, io=FILEPATH, num_cols=64, type_profile=ALL_FLOAT | 4.03 GiB/s | 5.20 GiB/s | 5.80 GiB/s | 1.44× | 1.11× |
| csv_read_with_inference_io | data_size_mb=256, io=FILEPATH, num_cols=64, type_profile=ALL_INTEGRAL | 4.12 GiB/s | 5.01 GiB/s | 6.37 GiB/s | 1.55× | 1.27× |
| csv_read_with_inference_io | data_size_mb=256, io=FILEPATH, num_cols=64, type_profile=ALL_STRING | 3.94 GiB/s | 9.98 GiB/s | 12.28 GiB/s | 3.12× | 1.23× |
| csv_read_with_inference_io | data_size_mb=256, io=FILEPATH, num_cols=64, type_profile=MIXED | 3.16 GiB/s | 4.53 GiB/s | 5.80 GiB/s | 1.84× | 1.28× |
| csv_read_with_inference_io | data_size_mb=256, io=FILEPATH, num_cols=8, type_profile=ALL_FLOAT | 4.37 GiB/s | 5.73 GiB/s | 6.14 GiB/s | 1.41× | 1.07× |
| csv_read_with_inference_io | data_size_mb=256, io=FILEPATH, num_cols=8, type_profile=ALL_INTEGRAL | 5.98 GiB/s | 7.10 GiB/s | 7.96 GiB/s | 1.33× | 1.12× |
| csv_read_with_inference_io | data_size_mb=256, io=FILEPATH, num_cols=8, type_profile=ALL_STRING | 7.49 GiB/s | 11.38 GiB/s | 17.26 GiB/s | 2.31× | 1.52× |
| csv_read_with_inference_io | data_size_mb=256, io=FILEPATH, num_cols=8, type_profile=MIXED | 5.79 GiB/s | 7.47 GiB/s | 8.34 GiB/s | 1.44× | 1.12× |
| csv_read_with_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=64, type_profile=ALL_FLOAT | 3.07 GiB/s | 3.31 GiB/s | 3.53 GiB/s | 1.15× | 1.07× |
| csv_read_with_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=64, type_profile=ALL_INTEGRAL | 3.21 GiB/s | 3.26 GiB/s | 3.90 GiB/s | 1.22× | 1.20× |
| csv_read_with_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=64, type_profile=ALL_STRING | 3.50 GiB/s | 7.23 GiB/s | 8.63 GiB/s | 2.47× | 1.19× |
| csv_read_with_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=64, type_profile=MIXED | 2.60 GiB/s | 3.00 GiB/s | 3.59 GiB/s | 1.38× | 1.20× |
| csv_read_with_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=8, type_profile=ALL_FLOAT | 3.30 GiB/s | 3.39 GiB/s | 3.54 GiB/s | 1.07× | 1.05× |
| csv_read_with_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=8, type_profile=ALL_INTEGRAL | 4.04 GiB/s | 4.25 GiB/s | 4.38 GiB/s | 1.08× | 1.03× |
| csv_read_with_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=8, type_profile=ALL_STRING | 6.36 GiB/s | 7.83 GiB/s | 10.91 GiB/s | 1.72× | 1.39× |
| csv_read_with_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=8, type_profile=MIXED | 4.06 GiB/s | 4.17 GiB/s | 4.58 GiB/s | 1.13× | 1.10× |
| csv_read_with_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=64, type_profile=ALL_FLOAT | 6.80 GiB/s | 8.01 GiB/s | 9.64 GiB/s | 1.42× | 1.20× |
| csv_read_with_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=64, type_profile=ALL_INTEGRAL | 6.43 GiB/s | 7.37 GiB/s | 10.86 GiB/s | 1.69× | 1.47× |
| csv_read_with_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=64, type_profile=ALL_STRING | 4.36 GiB/s | 12.45 GiB/s | 16.85 GiB/s | 3.86× | 1.35× |
| csv_read_with_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=64, type_profile=MIXED | 4.90 GiB/s | 6.37 GiB/s | 9.20 GiB/s | 1.88× | 1.44× |
| csv_read_with_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=8, type_profile=ALL_FLOAT | 7.73 GiB/s | 9.12 GiB/s | 10.79 GiB/s | 1.39× | 1.18× |
| csv_read_with_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=8, type_profile=ALL_INTEGRAL | 10.97 GiB/s | 12.56 GiB/s | 14.63 GiB/s | 1.33× | 1.16× |
| csv_read_with_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=8, type_profile=ALL_STRING | 10.30 GiB/s | 14.79 GiB/s | 28.79 GiB/s | 2.79× | 1.95× |
| csv_read_with_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=8, type_profile=MIXED | 10.97 GiB/s | 12.61 GiB/s | 14.70 GiB/s | 1.34× | 1.17× |
| csv_read_without_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=64, type_profile=ALL_FLOAT | 18.66 GiB/s | 23.22 GiB/s | 26.36 GiB/s | 1.41× | 1.14× |
| csv_read_without_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=64, type_profile=ALL_INTEGRAL | 17.43 GiB/s | 24.48 GiB/s | 38.11 GiB/s | 2.19× | 1.56× |
| csv_read_without_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=64, type_profile=ALL_STRING | 4.91 GiB/s | 24.50 GiB/s | 25.98 GiB/s | 5.29× | 1.06× |
| csv_read_without_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=64, type_profile=MIXED | 14.50 GiB/s | 23.39 GiB/s | 34.86 GiB/s | 2.40× | 1.49× |
| csv_read_without_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=8, type_profile=ALL_FLOAT | 21.45 GiB/s | 26.91 GiB/s | 27.76 GiB/s | 1.29× | 1.03× |
| csv_read_without_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=8, type_profile=ALL_INTEGRAL | 28.99 GiB/s | 41.64 GiB/s | 58.82 GiB/s | 2.03× | 1.41× |
| csv_read_without_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=8, type_profile=ALL_STRING | 20.79 GiB/s | 59.74 GiB/s | 65.06 GiB/s | 3.13× | 1.09× |
| csv_read_without_inference_io | data_size_mb=256, io=DEVICE_BUFFER, num_cols=8, type_profile=MIXED | 29.16 GiB/s | 42.00 GiB/s | 58.35 GiB/s | 2.00× | 1.39× |
| csv_read_without_inference_io | data_size_mb=256, io=FILEPATH, num_cols=64, type_profile=ALL_FLOAT | 5.12 GiB/s | 6.62 GiB/s | 6.37 GiB/s | 1.24× | 0.96× |
| csv_read_without_inference_io | data_size_mb=256, io=FILEPATH, num_cols=64, type_profile=ALL_INTEGRAL | 5.02 GiB/s | 7.09 GiB/s | 7.68 GiB/s | 1.53× | 1.08× |
| csv_read_without_inference_io | data_size_mb=256, io=FILEPATH, num_cols=64, type_profile=ALL_STRING | 4.78 GiB/s | 13.46 GiB/s | 13.38 GiB/s | 2.80× | 0.99× |
| csv_read_without_inference_io | data_size_mb=256, io=FILEPATH, num_cols=64, type_profile=MIXED | 4.60 GiB/s | 6.50 GiB/s | 6.61 GiB/s | 1.44× | 1.02× |
| csv_read_without_inference_io | data_size_mb=256, io=FILEPATH, num_cols=8, type_profile=ALL_FLOAT | 5.29 GiB/s | 6.94 GiB/s | 6.49 GiB/s | 1.23× | 0.94× |
| csv_read_without_inference_io | data_size_mb=256, io=FILEPATH, num_cols=8, type_profile=ALL_INTEGRAL | 6.01 GiB/s | 8.31 GiB/s | 8.65 GiB/s | 1.44× | 1.04× |
| csv_read_without_inference_io | data_size_mb=256, io=FILEPATH, num_cols=8, type_profile=ALL_STRING | 11.19 GiB/s | 19.09 GiB/s | 18.49 GiB/s | 1.65× | 0.97× |
| csv_read_without_inference_io | data_size_mb=256, io=FILEPATH, num_cols=8, type_profile=MIXED | 6.39 GiB/s | 8.32 GiB/s | 7.96 GiB/s | 1.25× | 0.96× |
| csv_read_without_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=64, type_profile=ALL_FLOAT | 3.75 GiB/s | 3.90 GiB/s | 3.80 GiB/s | 1.01× | 0.97× |
| csv_read_without_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=64, type_profile=ALL_INTEGRAL | 3.79 GiB/s | 4.21 GiB/s | 4.45 GiB/s | 1.17× | 1.06× |
| csv_read_without_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=64, type_profile=ALL_STRING | 4.25 GiB/s | 9.12 GiB/s | 9.14 GiB/s | 2.15× | 1.00× |
| csv_read_without_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=64, type_profile=MIXED | 3.33 GiB/s | 3.68 GiB/s | 3.89 GiB/s | 1.17× | 1.05× |
| csv_read_without_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=8, type_profile=ALL_FLOAT | 3.90 GiB/s | 4.01 GiB/s | 3.90 GiB/s | 1.00× | 0.97× |
| csv_read_without_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=8, type_profile=ALL_INTEGRAL | 4.30 GiB/s | 4.64 GiB/s | 4.75 GiB/s | 1.10× | 1.02× |
| csv_read_without_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=8, type_profile=ALL_STRING | 8.50 GiB/s | 11.04 GiB/s | 11.32 GiB/s | 1.33× | 1.03× |
| csv_read_without_inference_io | data_size_mb=256, io=HOST_BUFFER, num_cols=8, type_profile=MIXED | 4.49 GiB/s | 4.66 GiB/s | 4.64 GiB/s | 1.03× | 1.00× |
| csv_read_without_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=64, type_profile=ALL_FLOAT | 9.60 GiB/s | 10.89 GiB/s | 11.44 GiB/s | 1.19× | 1.05× |
| csv_read_without_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=64, type_profile=ALL_INTEGRAL | 9.67 GiB/s | 11.47 GiB/s | 14.00 GiB/s | 1.45× | 1.22× |
| csv_read_without_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=64, type_profile=ALL_STRING | 5.47 GiB/s | 18.53 GiB/s | 19.51 GiB/s | 3.57× | 1.05× |
| csv_read_without_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=64, type_profile=MIXED | 7.96 GiB/s | 10.71 GiB/s | 12.72 GiB/s | 1.60× | 1.19× |
| csv_read_without_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=8, type_profile=ALL_FLOAT | 10.51 GiB/s | 11.67 GiB/s | 11.73 GiB/s | 1.12× | 1.01× |
| csv_read_without_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=8, type_profile=ALL_INTEGRAL | 12.94 GiB/s | 14.77 GiB/s | 16.66 GiB/s | 1.29× | 1.13× |
| csv_read_without_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=8, type_profile=ALL_STRING | 17.06 GiB/s | 31.61 GiB/s | 33.26 GiB/s | 1.95× | 1.05× |
| csv_read_without_inference_io | data_size_mb=256, io=PINNED_BUFFER, num_cols=8, type_profile=MIXED | 12.86 GiB/s | 14.78 GiB/s | 16.47 GiB/s | 1.28× | 1.11× |
CSV_READER_TYPE_INFERENCE_NVBENCH (32 configs)
| Benchmark | Config | Base | Parent | This PR | vs base | vs parent |
|---|---|---|---|---|---|---|
| csv_read_with_inference | data_size_mb=256, num_cols=64, type_profile=ALL_FLOAT | 10.18 GiB/s | 13.44 GiB/s | 18.06 GiB/s | 1.77× | 1.34× |
| csv_read_with_inference | data_size_mb=256, num_cols=64, type_profile=ALL_INTEGRAL | 9.10 GiB/s | 12.01 GiB/s | 21.53 GiB/s | 2.37× | 1.79× |
| csv_read_with_inference | data_size_mb=256, num_cols=64, type_profile=ALL_STRING | 4.40 GiB/s | 16.23 GiB/s | 22.80 GiB/s | 5.18× | 1.40× |
| csv_read_with_inference | data_size_mb=256, num_cols=64, type_profile=MIXED | 6.80 GiB/s | 9.82 GiB/s | 17.33 GiB/s | 2.55× | 1.76× |
| csv_read_with_inference | data_size_mb=256, num_cols=8, type_profile=ALL_FLOAT | 12.32 GiB/s | 17.01 GiB/s | 22.61 GiB/s | 1.84× | 1.33× |
| csv_read_with_inference | data_size_mb=256, num_cols=8, type_profile=ALL_INTEGRAL | 21.29 GiB/s | 28.43 GiB/s | 40.00 GiB/s | 1.88× | 1.41× |
| csv_read_with_inference | data_size_mb=256, num_cols=8, type_profile=ALL_STRING | 12.00 GiB/s | 20.59 GiB/s | 50.31 GiB/s | 4.19× | 2.44× |
| csv_read_with_inference | data_size_mb=256, num_cols=8, type_profile=MIXED | 21.64 GiB/s | 28.95 GiB/s | 40.71 GiB/s | 1.88× | 1.41× |
| csv_read_with_inference | data_size_mb=64, num_cols=64, type_profile=ALL_FLOAT | 10.15 GiB/s | 10.41 GiB/s | 13.54 GiB/s | 1.33× | 1.30× |
| csv_read_with_inference | data_size_mb=64, num_cols=64, type_profile=ALL_INTEGRAL | 8.33 GiB/s | 10.47 GiB/s | 15.68 GiB/s | 1.88× | 1.50× |
| csv_read_with_inference | data_size_mb=64, num_cols=64, type_profile=ALL_STRING | 1.42 GiB/s | 7.10 GiB/s | 7.84 GiB/s | 5.51× | 1.10× |
| csv_read_with_inference | data_size_mb=64, num_cols=64, type_profile=MIXED | 4.05 GiB/s | 7.30 GiB/s | 10.58 GiB/s | 2.61× | 1.45× |
| csv_read_with_inference | data_size_mb=64, num_cols=8, type_profile=ALL_FLOAT | 11.10 GiB/s | 14.46 GiB/s | 18.60 GiB/s | 1.68× | 1.29× |
| csv_read_with_inference | data_size_mb=64, num_cols=8, type_profile=ALL_INTEGRAL | 18.41 GiB/s | 23.81 GiB/s | 30.59 GiB/s | 1.66× | 1.28× |
| csv_read_with_inference | data_size_mb=64, num_cols=8, type_profile=ALL_STRING | 5.52 GiB/s | 16.80 GiB/s | 31.08 GiB/s | 5.63× | 1.85× |
| csv_read_with_inference | data_size_mb=64, num_cols=8, type_profile=MIXED | 19.86 GiB/s | 25.03 GiB/s | 32.60 GiB/s | 1.64× | 1.30× |
| csv_read_without_inference | data_size_mb=256, num_cols=64, type_profile=ALL_FLOAT | 18.10 GiB/s | 23.49 GiB/s | 26.40 GiB/s | 1.46× | 1.12× |
| csv_read_without_inference | data_size_mb=256, num_cols=64, type_profile=ALL_INTEGRAL | 17.51 GiB/s | 24.68 GiB/s | 38.37 GiB/s | 2.19× | 1.55× |
| csv_read_without_inference | data_size_mb=256, num_cols=64, type_profile=ALL_STRING | 4.70 GiB/s | 25.21 GiB/s | 27.15 GiB/s | 5.78× | 1.08× |
| csv_read_without_inference | data_size_mb=256, num_cols=64, type_profile=MIXED | 14.57 GiB/s | 23.41 GiB/s | 35.16 GiB/s | 2.41× | 1.50× |
| csv_read_without_inference | data_size_mb=256, num_cols=8, type_profile=ALL_FLOAT | 21.55 GiB/s | 26.93 GiB/s | 27.92 GiB/s | 1.30× | 1.04× |
| csv_read_without_inference | data_size_mb=256, num_cols=8, type_profile=ALL_INTEGRAL | 29.30 GiB/s | 42.93 GiB/s | 58.48 GiB/s | 2.00× | 1.36× |
| csv_read_without_inference | data_size_mb=256, num_cols=8, type_profile=ALL_STRING | 21.92 GiB/s | 63.01 GiB/s | 64.91 GiB/s | 2.96× | 1.03× |
| csv_read_without_inference | data_size_mb=256, num_cols=8, type_profile=MIXED | 29.24 GiB/s | 42.16 GiB/s | 59.39 GiB/s | 2.03× | 1.41× |
| csv_read_without_inference | data_size_mb=64, num_cols=64, type_profile=ALL_FLOAT | 15.91 GiB/s | 16.00 GiB/s | 18.68 GiB/s | 1.17× | 1.17× |
| csv_read_without_inference | data_size_mb=64, num_cols=64, type_profile=ALL_INTEGRAL | 14.41 GiB/s | 18.52 GiB/s | 26.43 GiB/s | 1.83× | 1.43× |
| csv_read_without_inference | data_size_mb=64, num_cols=64, type_profile=ALL_STRING | 1.44 GiB/s | 8.51 GiB/s | 8.69 GiB/s | 6.05× | 1.02× |
| csv_read_without_inference | data_size_mb=64, num_cols=64, type_profile=MIXED | 9.73 GiB/s | 17.03 GiB/s | 23.28 GiB/s | 2.39× | 1.37× |
| csv_read_without_inference | data_size_mb=64, num_cols=8, type_profile=ALL_FLOAT | 18.58 GiB/s | 21.68 GiB/s | 23.24 GiB/s | 1.25× | 1.07× |
| csv_read_without_inference | data_size_mb=64, num_cols=8, type_profile=ALL_INTEGRAL | 25.71 GiB/s | 35.53 GiB/s | 43.52 GiB/s | 1.69× | 1.23× |
| csv_read_without_inference | data_size_mb=64, num_cols=8, type_profile=ALL_STRING | 7.32 GiB/s | 40.94 GiB/s | 37.78 GiB/s | 5.16× | 0.92× |
| csv_read_without_inference | data_size_mb=64, num_cols=8, type_profile=MIXED | 25.23 GiB/s | 35.45 GiB/s | 43.45 GiB/s | 1.72× | 1.23× |
Summary
The CSV type-detection and decode kernels process one row per thread. This PR cuts their memory transactions and atomic contention:
atomicOrper group of lanes on the same column. Null counts come from the finished masks viacudf::batch_null_count, replacing a same-addressatomicAddper field.cudf::batch_null_countnow supports more than 65,535 bitmasks. This is a libcudf-wide fix; see Notes.Motivation
With one thread per row, a warp's loads are spread over 32 rows, so every character costs a separate memory transaction. The per-column counters (valid counts and type histograms) are a few addresses that every thread updates, so their atomics serialize.
Design
seek_field_end_by_words: returns exactly whatseek_field_endreturns without escape characters.delim_whitespacekeeps the old seek.__match_any_syncand set their bits with oneatomicOr. Null counts are computed from the zeroed masks over exactly [0, num_records), which adds no synchronization.field_classthat indexes the counters, and increments are warp-aggregated.classify_integral_fieldholds the integer logic and is shared with the JSON type inference. Up to 32 columns use per-block shared-memory counters (a separate template variant, which compiles to shared-memory atomicsATOMS; a runtime-flag build compiled the increment to a genericATOM.E.ADD); wider tables use global counters. The cutoff was measured on one GPU and only affects speed.Testing
\x01,\xFE, CR-only, and CRLF inside quotes; CRLF input including withdelim_whitespace; validity at mask-word edges with short rows anduse_cols; types decided by single fields in later blocks (8, 32, 33 and 40 columns); wide-table inference; more than 65,535 columns end to end plus a 70,000-maskbatch_null_counttest.--rmm_mode=cuda) and racecheck are clean; the 70,000-column test allocates from its own pool so it stays fast under compute-sanitizer. Mutation checks on the counting paths are caught by the new tests.Performance (local, RTX PRO 6000, 256 MB, minimum of 3 interleaved rounds; official best-of-2 numbers for all sources are in the benchmark comment below)
Notes for reviewers
cudf::batch_null_count(src/bitmask/null_mask.cu) for all of libcudf. Behaviour is unchanged for 65,535 masks or fewer. Owners of that utility should review it; it could also land separately first.This is PR 4 of a stack of draft PRs extracted from an automated (CORAL) optimization run of the CSV reader, reviewed by a separate principal-engineer review pass; it builds on #18.
🤖 Generated with Claude Code