8000 GitHub - RRZE-HPC/WINIC: Multi-Platform Microbenchmarking Tool · GitHub
[go: up one dir, main page]

Skip to content
RRZE-HPCPublic

About

Multi-Platform Microbenchmarking Tool

Resources

Stars

4 stars

Watchers

2 watching

Forks

Repository files navigation

Introduction

WINIC is a platform-independent automated microbenchmarking tool. It currently supports x86, ARM and RISC-V on Linux. WINIC can automatically determine latency and throughput values for most instructions the given CPU supports.

Limitations

WINIC currently cannot measure:

  • latencies on parts of memory operands e.g. base and index registers
  • gather and scatter instructions
  • branches, returns, system calls
  • privileged instructions

Benchmarking strategy

For a given instruction form, WINIC either generates a kernel that allows the CPU to parallelize execution for measuring throughput, or one that forces sequential execution for measuring latency. The kernel generated is executed in a loop and the total runtime is measured. The cycles needed per executed instruction is calculated using only the clock frequency and the number of times the instruction was executed. Therefore WINIC does not need access to hardware performance counters.

Download and Build

WINIC uses LLVM and clang to generate and assemble benchmarks. Use setup.sh after cloning this repository to automatically download and build LLVM and WINIC. To manage multiple builds e.g. for multiple platforms in an HPC context specify --dir <buildName> to build a version of LLVM into ./llvm-build-buildName and WINIC into ./build-buildName.

Usage

To calculate the elapsed clock cycles per instruction WINIC needs the clock-frequency to be fixed e.g. by using likwid-setFrequencies. Once the frequency is fixed use WINIC as follows:

./winic [options] MODE -f <frequency> [mode-specific-options]

For more consistent results it is strongly recommended to pin the application to a single core e.g. with likwid-pin.

likwid-pin -c 2 ./winic [options] MODE -f <frequency> [mode-specific-options]

Global Options

Option Description Default
-d,--debug Enable debug output false
-c,--cpu CPU model, needed if LLVM cannot detect it, or to use the INFO mode on a non-native architecture -
-m,--march Architecture, needed if LLVM cannot detect it, or to use the INFO mode on a non-native architecture -
-v,--version Show version -

Available modes:

TP/LAT:

Measure latencies or inverse throughputs. By default WINIC measures all available instructions and generates a .yaml file with the results. Additionally a report_{MODE}_{TIMESTAMP} is generated providing additional information about how the values were obtained and warnings about unusual results. The runtime of a full run strongly depends on the architecture, here some rough estimates:

Architecture Runtime TP Runtime LAT
x86 1h - 2h 3h - 9h
AArch64 5 min 30 min
RISCV 30 min 1h

TP/LAT Options

Option Description Default
-f,--frequency Frequency in GHz (required) -
-i,--instruction Measure specific instructions by LLVM instruction name. This option can take multiple names or regular expressions. For example, to measure all variants of SSE/AVX fused-multiply-add instructions use -i VFMADD.* -
-o,--output Path to the .yaml file to save the results to. If the file already exists it will be updated (see updating a database) . If no file is specified, a timestamped one will be generated. If set to /dev/null no file will be generated. -
--register-init-value V Load V into all registers used before every benchmark run. Accepts decimal, octal with prefix 0, hexadecimal with prefix 0x, or floating point e.g. 7.0 (positive only) 7.0
--immediate-value V Set all immediates to V. Accepts decimal, octal with prefix 0 or hexadecimal with prefix 0x 7
--runs N Repeat each measurement N times and take the minimum runtime 4
--loop-iterations N Number of loop iterations of the kernel. Higher values increase precision but also increase runtime. 1000000
--loop-size N Target Number of instructions in the Loop body. Shorter bodies might be generated if there are not enough registers available 12
--no-report Don't generate report file false
--output-asm Write the generated asm files to asm/ (clears the directory at the start of the run) false
--memory Enable (all), disable (none) or filter (only) for instructions accessing memory all
--x87-fp Enable, disable or filter for x87 floating point instructions. Default is none, as they are deprecated and consume a lot of time on architectures that emulate them. none
--runtime-limit Set a limit for the time spent on an instruction in cycles. If a run takes longer than the limit the instruction is skipped 300
--run-in-subprocess Run every benchmark in a subprocess. Always leave this enabled in production runs true
--keep-empty-entries Include instructions in the output even if they do not have any values false
--min-opcode Minimum LLVM opcode number to measure (this is mostly useful for development) 0
--max-opcode Maximum LLVM opcode number to measure (this is mostly useful for development) max opcode

MAN

In manual mode, WINIC can execute arbitrary benchmark functions. To run a function called tp from file.s and calculate the cycles per instruction assuming the loop has 12 instructions do

winic MAN -f <frequency> --path file.s --func-name tp --num-instructions 12

MAN Options

Option Description Default
-f,--frequency Frequency in GHz (required) -
-p,--path Assembly file path (required) -
--func-name Function to benchmark (required) -
-n,--num-instructions Number of instructions in loop (required) -
--init-name Initialization function -
--runs N Repeat each measurement N times and take the minimum runtime 4

There are always cases where WINIC doesn't produce correct data. To do a custom benchmark for an instruction, first run WINIC in TP or LAT mode with -i <LLVM_INSTRUCTION_NAME> --output-asm. This will output all .s files generated for the benchmark to asm/ and generate an assembler_out.log. The .s files can then be modified and executed using the MAN-mode.

INFO

In info mode, WINIC can provide metadata to any instruction it supports. This is useful if one wants to find the LLVM name for a specific instruction variant (find all "add" variants using -i ADD.*) and for development. The INFO mode can be used no matter the architecture WINIC is running on. To get metadata for another architecture, use the -c and -m options.

INFO Options

Option Description Default
-i,--instruction Specify the instruction name. This option can take multiple names or regular expressions. -
--internal Display the native LLVM operand types instead of WINICs abstraction -

Updating a database

By default TP and LAT mode generate a db_{TIMESTAMP}.yaml file with the results. Use -o/--output <file.yaml> to specify a custom path instead. If the file already exists the values obtained during the run will overwrite the existing ones according to the following rules:

  • all new non-null values will overwrite existing values
  • new null values will not overwrite existing values
  • all existing values will be left unchanged if no new value was obtained

Updating the database works with single instructions as well as full TP/LAT runs. A standard workflow therefore would be to do a TP run generating a database and then a LAT run updating it.

Helper instructions

WINIC automatically uses helper instructions to:

  • break dependencies between instructions to measure throughput
  • introduce dependencies between instructions to measure latency

All uses of helper instructions are logged in report_{MODE}_{TIMESTAMP}.
If an instruction would need a helper but none can be found, WINIC will fail and report "ERROR_NO_HELPER".
WINIC can only use instructions as helper if they were measured in the current run which is a problem when trying to measure single instructions. The solution is to first do a full run and look up the dependencies of the instruction in the report, then the measurement can be reproduced by supplying all dependencies alongside the instruction using the -i <LLVM_INSTRUCTION_NAME> option.
Note that currently --output does not load the values into the internal working databases so the information read from there can not be used as helpers.

Analysis/Reference files

The analysis/ python module can be used to compare the results with various other sources, or to generate useful reference files which contain comprehensive information about instructions, operands, registers etc. from LLVM. For more details refer to analysis/README.md.

About

Multi-Platform Microbenchmarking Tool

Resources

Stars

4 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages

0