diff --git a/README.md b/README.md index 625ecbfd71b938eae7e375614a10752b25551083..9345cc6f693d731d6bf01957441338a90f65a55c 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,10 @@ Please visit the [official Parasoft website](http://www.parasoft.com) for more i - [Quick start](#quick-start-sa) - [Example Pipelines](#example-pipelines-sa) - [Reviewing Analysis Results](#reviewing-analysis-results-sa) +- [Unit Tests](#unit-tests) + - [Quick start](#quick-start-ut) + - [Example Pipelines](#example-pipelines-ut) + - [Reviewing Test Results](#reviewing-test-results-ut) - [Code Coverage](#code-coverage) - [Quick start](#quick-start-cc) - [Example Pipelines](#example-pipelines-cc) @@ -226,6 +230,162 @@ You can define a merge request policy for your integration branch that will bloc 1. In the **Policy details** section, define a rule for Static Application Security Testing (select “IF SAST…”). Configure additional options, if needed. +## Unit Tests +### Quick start +To collect test executions results for your code with Parasoft C/C++test and review test results in GitLab, you need to customize your pipeline to include a job that will: + - build your project with unit tests instrumentation enabled. + - execute the instrumented application. + - generate C/C++test unit tests reports. + - convert C/C++test unit tests report to xUnit format (using Saxon). + - upload the xUnit report. + +#### Prerequisites +* This extension requires Parasoft C/C++test 2021.2 (or newer) with a valid Parasoft license. +* We recommend that you execute the pipeline on a GitLab runner with the following components installed and configured on the runner: + - C/C++ build toolchain + - Parasoft C/C++test 2021.2 (or newer) + - On Windows, we recommend that you use PowerShell Core 6 or later. If you use Windows PowerShell 5.1, you must ensure the correct file encoding - see the example pipelines for details. + +* To support xUnit format, you need the following files to be accessible on a GitLab runner: + - Saxon-HE, which you can copy the folder from [here](https://gitlab.com/parasoft/cpptest-gitlab/-/blob/master/xsl/saxon) or download from [Saxonica](https://www.saxonica.com/download/java.xml). + - [XSLT file](xsl/cpptest-professional-xunit/xunit.xsl) for transforming C/C++test Professional unit tests report to xUnit report. + - [XSLT file](xsl/cpptest-standard-xunit/xunit.xsl) for transforming C/C++test Standard unit tests report to xUnit report. + +### Example Pipelines +The following examples show simple pipelines for Make and CMake-based projects. The examples assume that C/C++test is run on a GitLab runner and the path to the `cpptestcli` executable is available on `$PATH`. + + +##### Run C/C++test Standard with CMake project +For details on how to configure project for unit tests, see [Integrating with CppUnit and CppUtest](https://docs.parasoft.com/display/CPPTEST20232/Integrating+with+CppUnit+and+CppUtest). +See also the example [.gitlab-ci.yml](pipelines/unit/cpptest-standard-cmake/.gitlab-ci.yml) file. +```yaml +# This is a basic pipeline to help you get started with C/C++test integration to collect unit tests results for a CMake-based project. + +# Be sure to configure variables below. +variables: + CMAKE_APP_NAME: "app-executable" + CPPTEST_UT_LOG_FILE: "path/to/cpptest-ut-log-file" + CPPTEST_INSTALL_DIR: "path/to/cpptest" + CPPTEST_XSL_DIR: "path/to/xsl" + CPPTEST_SAXON_DIR: "path/to/saxon" + +stages: + - test + +# Runs unit tests with C/C++test. +cpptest-unit-tests: + variables: + CPPTEST_REPORTS_DIR: "path/to/reports" + + stage: test + + script: + # When running on Windows with PowerShell 5.1, be sure to enforce the default file encoding: + # - $PSDefaultParameterValues['Out-File:Encoding'] = 'default' + + # Configures your CMake project. + - echo "Configuring project..." + - cmake -S . -B build + + # Builds your CMake project. + - echo "Building project" + - cmake --build build + + # Run the instrumented application to generate unit tests log file (.utlog). + - echo "Run the instrumented application" + - build/$CMAKE_APP_NAME + + # Generate unit tests report + - echo "Generating unit tests report" + - cpptestcli -config "builtin://Unit Testing" -input $CPPTEST_UT_LOG_FILE -report $CPPTEST_REPORTS_DIR + + # Converts the unit tests report to xUnit format. + # + # To use Saxon for report transformation, a Java executable is required. + # C/C++test includes Java which can be used for this purpose. + - echo "Generating xUnit report..." + - $CPPTEST_INSTALL_DIR/bin/jre/bin/java -jar "$CPPTEST_SAXON_DIR/saxon-he-12.2.jar" -xsl:"$CPPTEST_XSL_DIR/xunit.xsl" -s:"$CPPTEST_REPORTS_DIR/report.xml" -o:"$CPPTEST_REPORTS_DIR/report-xunit.xml" + + artifacts: + # Uploads test results in the xUnit format, so that they are displayed in GitLab + reports: + junit: $CPPTEST_REPORTS_DIR/report-xunit.xml +``` + +##### Run C/C++test Professional with Make project +For details on how to configure project for unit tests, see [Testing from the Command Line Interface](https://docs.parasoft.com/display/CPPTESTPROVS20232/Testing+from+the+Command+Line+Interface). +See also the example [.gitlab-ci.yml](pipelines/unit/cpptest-professional-make/.gitlab-ci.yml) file. + +```yaml +# This is a basic pipeline to help you get started with C/C++test integration to collect unit tests results for a Make-based project. + +# Be sure to configure variables below. +variables: + CPPTEST_XSL_DIR: "path/to/xsl" + CPPTEST_SAXON_DIR: "path/to/saxon" + CPPTEST_INSTALL_DIR: "path/to/cpptest" + +stages: + - build + - test + +build-make: + stage: build + script: + # Builds your Make project using 'cpptesttrace' to collect input data for handle test executions results. + # Be sure 'cpptesttrace' is available on $PATH. + - echo "Building project..." + - cpptesttrace make clean all + artifacts: + # Archives cpptestscan.bdf so that it can be used in the 'test' stage. + paths: + - cpptestscan.bdf + +# Runs unit test with C/C++test. +cpptest-unit-tests: + variables: + CPPTEST_REPORTS_DIR: "path/to/reports" + RESOURCE_PROJECT_FOLDER_NAME: "resource project folder name" + LOCALSETTINGS_FILE_PATH: "path/to/localsettings/properties/file" + + stage: test + script: + # When running on Windows with PowerShell 5.1, be sure to enforce the default file encoding: + # - $PSDefaultParameterValues['Out-File:Encoding'] = 'default' + + # Launches C/C++test. + - echo "Running C/C++test..." + # Generate unit tests + - echo "Generate unit tests" + - cpptestcli -config "builtin://Generate Unit Tests" -resource $RESOURCE_PROJECT_FOLDER_NAME -localsettings $LOCALSETTINGS_FILE_PATH -data $CI_BUILDS_DIR/cpptest-workspace-$CI_PIPELINE_ID -bdf cpptestscan.bdf + # Run unit tests + - echo "Run unit tests" + # Use the following command line when using cpptest professional prior to v2024.1 + # - cpptestcli -config "builtin://Run Unit Tests" -resource $RESOURCE_PROJECT_FOLDER_NAME -localsettings $LOCALSETTINGS_FILE_PATH -data $CI_BUILDS_DIR/cpptest-workspace-$CI_PIPELINE_ID -bdf cpptestscan.bdf -report $CPPTEST_REPORTS_DIR + + # Use the following command line when using cpptest professional v2024.1 + - cpptestcli -config "builtin://Run Unit Tests" -resource $RESOURCE_PROJECT_FOLDER_NAME -localsettings $LOCALSETTINGS_FILE_PATH -data $CI_BUILDS_DIR/cpptest-workspace-$CI_PIPELINE_ID -bdf cpptestscan.bdf -property report.additional.report.dir=$CPPTEST_REPORTS_DIR + + # Converts the unit tests report to xUnit format. + # + # To use Saxon for report transformation, a Java executable is required. + # C/C++test includes Java which can be used for this purpose. + - echo "Generating xUnit report..." + - $CPPTEST_INSTALL_DIR/bin/jre/bin/java -jar "$CPPTEST_SAXON_DIR/saxon-he-12.2.jar" -xsl:"$CPPTEST_XSL_DIR/xunit.xsl" -s:"$CPPTEST_REPORTS_DIR/report.xml" -o:"$CPPTEST_REPORTS_DIR/report-xunit.xml" + after_script: + # Removes the workspace folder. + - rm -rf $CI_BUILDS_DIR/cpptest-workspace-$CI_PIPELINE_ID + + artifacts: + # Uploads test results in the xUnit format, so that they are displayed in GitLab + reports: + junit: $CPPTEST_REPORTS_DIR/report-xunit.xml +``` + +### Reviewing Test Results +When the pipeline completes, you can review the test results collected by C/C++test in the *Tests* tab of the GitLab pipeline. + + ## Code Coverage ### Quick start @@ -245,7 +405,7 @@ To collect code coverage with Parasoft C/C++test Standard and review coverage re - Parasoft C/C++test Standard 2021.2 (or newer) - On Windows, we recommend that you use PowerShell Core 6 or later. If you use Windows PowerShell 5.1, you must ensure the correct file encoding - see the example pipelines for details. * To support Cobertura format, you need the following files to be accessible on a GitLab runner: - - Saxon-HE, which you can copy from [here](https://gitlab.com/parasoft/cpptest-gitlab/-/blob/master/xsl/saxon) or download from [Saxonica](https://www.saxonica.com/download/java.xml). + - Saxon-HE, which you can copy the folder from [here](https://gitlab.com/parasoft/cpptest-gitlab/-/blob/master/xsl/saxon) or download from [Saxonica](https://www.saxonica.com/download/java.xml). - [XSLT file](https://gitlab.com/parasoft/cpptest-gitlab/-/blob/master/xsl/cpptest-standard-cobertura/cobertura.xsl) for transforming C/C++test coverage report to Cobertura report. ### Example Pipelines diff --git a/pipelines/unit/cpptest-professional-make/.gitlab-ci.yml b/pipelines/unit/cpptest-professional-make/.gitlab-ci.yml new file mode 100644 index 0000000000000000000000000000000000000000..71770e5eb389bb82ac92271aeb2b4bf4f6182d38 --- /dev/null +++ b/pipelines/unit/cpptest-professional-make/.gitlab-ci.yml @@ -0,0 +1,62 @@ +# This is a basic pipeline to help you get started with C/C++test integration to collect unit tests results for a Make-based project. + +# Be sure to configure variables below. +variables: + CPPTEST_XSL_DIR: "path/to/xsl" + CPPTEST_SAXON_DIR: "path/to/saxon" + CPPTEST_INSTALL_DIR: "path/to/cpptest" + +stages: + - build + - test + +build-make: + stage: build + script: + # Builds your Make project using 'cpptesttrace' to collect input data for handle test executions results. + # Be sure 'cpptesttrace' is available on $PATH. + - echo "Building project..." + - cpptesttrace make clean all + artifacts: + # Archives cpptestscan.bdf so that it can be used in the 'test' stage. + paths: + - cpptestscan.bdf + +# Runs unit test with C/C++test. +cpptest-unit-tests: + variables: + CPPTEST_REPORTS_DIR: "path/to/reports" + RESOURCE_PROJECT_FOLDER_NAME: "resource project folder name" + LOCALSETTINGS_FILE_PATH: "path/to/localsettings/properties/file" + + stage: test + script: + # When running on Windows with PowerShell 5.1, be sure to enforce the default file encoding: + # - $PSDefaultParameterValues['Out-File:Encoding'] = 'default' + + # Launches C/C++test. + - echo "Running C/C++test..." + # Generate unit tests + - echo "Generate unit tests" + - cpptestcli -config "builtin://Generate Unit Tests" -resource $RESOURCE_PROJECT_FOLDER_NAME -localsettings $LOCALSETTINGS_FILE_PATH -data $CI_BUILDS_DIR/cpptest-workspace-$CI_PIPELINE_ID -bdf cpptestscan.bdf + # Run unit tests + - echo "Run unit tests" + # Use the following command line when using cpptest professional prior to v2024.1 + # - cpptestcli -config "builtin://Run Unit Tests" -resource $RESOURCE_PROJECT_FOLDER_NAME -localsettings $LOCALSETTINGS_FILE_PATH -data $CI_BUILDS_DIR/cpptest-workspace-$CI_PIPELINE_ID -bdf cpptestscan.bdf -report $CPPTEST_REPORTS_DIR + # Use the following command line when using cpptest professional v2024.1 + - cpptestcli -config "builtin://Run Unit Tests" -resource $RESOURCE_PROJECT_FOLDER_NAME -localsettings $LOCALSETTINGS_FILE_PATH -data $CI_BUILDS_DIR/cpptest-workspace-$CI_PIPELINE_ID -bdf cpptestscan.bdf -property report.additional.report.dir=$CPPTEST_REPORTS_DIR + + # Converts the unit tests report to xUnit format. + # + # To use Saxon for report transformation, a Java executable is required. + # C/C++test includes Java which can be used for this purpose. + - echo "Generating xUnit report..." + - $CPPTEST_INSTALL_DIR/bin/jre/bin/java -jar "$CPPTEST_SAXON_DIR/saxon-he-12.2.jar" -xsl:"$CPPTEST_XSL_DIR/xunit.xsl" -s:"$CPPTEST_REPORTS_DIR/report.xml" -o:"$CPPTEST_REPORTS_DIR/report-xunit.xml" + after_script: + # Removes the workspace folder. + - rm -rf $CI_BUILDS_DIR/cpptest-workspace-$CI_PIPELINE_ID + + artifacts: + # Uploads test results in the xUnit format, so that they are displayed in GitLab + reports: + junit: $CPPTEST_REPORTS_DIR/report-xunit.xml \ No newline at end of file diff --git a/pipelines/unit/cpptest-standard-cmake/.gitlab-ci.yml b/pipelines/unit/cpptest-standard-cmake/.gitlab-ci.yml new file mode 100644 index 0000000000000000000000000000000000000000..b431df68b246fb1ca1d686cf29fa62db9e1e95ff --- /dev/null +++ b/pipelines/unit/cpptest-standard-cmake/.gitlab-ci.yml @@ -0,0 +1,51 @@ +# This is a basic pipeline to help you get started with C/C++test integration to collect unit tests results for a CMake-based project. + +# Be sure to configure variables below. +variables: + CMAKE_APP_NAME: "app-executable" + CPPTEST_UT_LOG_FILE: "path/to/cpptest-ut-log-file" + CPPTEST_INSTALL_DIR: "path/to/cpptest" + CPPTEST_XSL_DIR: "path/to/xsl" + CPPTEST_SAXON_DIR: "path/to/saxon" + +stages: + - test + +# Runs unit tests with C/C++test. +cpptest-unit-tests: + variables: + CPPTEST_REPORTS_DIR: "path/to/reports" + + stage: test + + script: + # When running on Windows with PowerShell 5.1, be sure to enforce the default file encoding: + # - $PSDefaultParameterValues['Out-File:Encoding'] = 'default' + + # Configures your CMake project. + - echo "Configuring project..." + - cmake -S . -B build + + # Builds your CMake project. + - echo "Building project" + - cmake --build build + + # Run the instrumented application to generate unit tests log file (.utlog). + - echo "Run the instrumented application" + - build/$CMAKE_APP_NAME + + # Generate unit tests report + - echo "Generating unit tests report" + - cpptestcli -config "builtin://Unit Testing" -input $CPPTEST_UT_LOG_FILE -report $CPPTEST_REPORTS_DIR + + # Converts the unit tests report to xUnit format. + # + # To use Saxon for report transformation, a Java executable is required. + # C/C++test includes Java which can be used for this purpose. + - echo "Generating xUnit report..." + - $CPPTEST_INSTALL_DIR/bin/jre/bin/java -jar "$CPPTEST_SAXON_DIR/saxon-he-12.2.jar" -xsl:"$CPPTEST_XSL_DIR/xunit.xsl" -s:"$CPPTEST_REPORTS_DIR/report.xml" -o:"$CPPTEST_REPORTS_DIR/report-xunit.xml" + + artifacts: + # Uploads test results in the xUnit format, so that they are displayed in GitLab + reports: + junit: $CPPTEST_REPORTS_DIR/report-xunit.xml \ No newline at end of file diff --git a/xsl/cpptest-professional-xunit/README.md b/xsl/cpptest-professional-xunit/README.md new file mode 100644 index 0000000000000000000000000000000000000000..fb4049a3390c413c8bd70fcf2dae49d0dd9f0ea1 --- /dev/null +++ b/xsl/cpptest-professional-xunit/README.md @@ -0,0 +1,23 @@ +# Generating a Xunit report with C/C++test Professional + +To report unit tests results using xUnit format: + +1. Copy the [`xunit.xsl`](xunit.xsl) file into a local directory (``). +2. Copy [`Saxon`](https://gitlab.com/parasoft/cpptest-gitlab/-/blob/master/xsl/saxon) files into a local directory (``). +3. Update your GitLab pipeline to convert `/report.xml` report into `/report-xunit.xml`: + +```yaml + ... + # Converts the unit tests report to xUnit format. + # + # To use Saxon for report transformation, a Java executable is required. + # C/C++test includes Java which can be used for this purpose. + - echo "Generating xUnit report..." + - $CPPTEST_INSTALL_DIR/bin/jre/bin/java -jar "$CPPTEST_SAXON_DIR/saxon-he-12.2.jar" -xsl:"$CPPTEST_XSL_DIR/xunit.xsl" -s:"$CPPTEST_REPORTS_DIR/report.xml" -o:"$CPPTEST_REPORTS_DIR/report-xunit.xml" + + artifacts: + # Uploads test results in the xUnit format, so that they are displayed in GitLab + reports: + junit: $CPPTEST_REPORTS_DIR/report-xunit.xml +``` +4. Run your GitLab pipeline. diff --git a/xsl/cpptest-professional-xunit/xunit.xsl b/xsl/cpptest-professional-xunit/xunit.xsl new file mode 100644 index 0000000000000000000000000000000000000000..b79de62990ca6e0a12f53a5bcf6f1e407b92cd41 --- /dev/null +++ b/xsl/cpptest-professional-xunit/xunit.xsl @@ -0,0 +1,598 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Caused by: + + + + + + at + + + + + + + + + + + + + + + + + + + + + Multiple errors reported + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + err + fail + pass + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Caused by: + + + + + + at + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + + + + + + + + + + . + + + / + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + 0 + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/xsl/cpptest-standard-xunit/README.md b/xsl/cpptest-standard-xunit/README.md new file mode 100644 index 0000000000000000000000000000000000000000..53176ddb69164bda0298a47255b8f66eb5c72482 --- /dev/null +++ b/xsl/cpptest-standard-xunit/README.md @@ -0,0 +1,23 @@ +# Generating a xUnit report with C/C++test Standard + +To report unit tests results using xUnit format: + +1. Copy the [`xunit.xsl`](xunit.xsl) file into a local directory (``). +2. Copy [`Saxon`](https://gitlab.com/parasoft/cpptest-gitlab/-/blob/master/xsl/saxon) files into a local directory (``). +3. Update your GitLab pipeline to convert `/report.xml` report into `/report-xunit.xml`: + +```yaml + ... + # Converts the unit tests report to xUnit format. + # + # To use Saxon for report transformation, a Java executable is required. + # C/C++test includes Java which can be used for this purpose. + - echo "Generating xUnit report..." + - $CPPTEST_INSTALL_DIR/bin/jre/bin/java -jar "$CPPTEST_SAXON_DIR/saxon-he-12.2.jar" -xsl:"$CPPTEST_XSL_DIR/xunit.xsl" -s:"$CPPTEST_REPORTS_DIR/report.xml" -o:"$CPPTEST_REPORTS_DIR/report-xunit.xml" + + artifacts: + # Uploads test results in the xUnit format, so that they are displayed in GitLab + reports: + junit: $CPPTEST_REPORTS_DIR/report-xunit.xml +``` +4. Run your GitLab pipeline. diff --git a/xsl/cpptest-standard-xunit/xunit.xsl b/xsl/cpptest-standard-xunit/xunit.xsl new file mode 100644 index 0000000000000000000000000000000000000000..b79de62990ca6e0a12f53a5bcf6f1e407b92cd41 --- /dev/null +++ b/xsl/cpptest-standard-xunit/xunit.xsl @@ -0,0 +1,598 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Caused by: + + + + + + at + + + + + + + + + + + + + + + + + + + + + Multiple errors reported + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + err + fail + pass + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Caused by: + + + + + + at + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + + + + + + + + + + + + + + . + + + / + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + 0 + + + + + + + + + 0 + + + + + + + + + + + + + + + + + + + \ No newline at end of file