This document describes how OpenShift developers should interact with the OpenShift command-line integration test suite.
The script to run the entire suite lives in hack/test-cmd.sh. All of the test suites that make up
the parent suite live in test/cmd, and are divided by functional area.
To run the full test suite, use:
$ hack/test-cmd.shTo run a single test suite, use:
$ hack/test-cmd.sh <name>To run a set of suites matching some regex, use:
$ hack/test-cmd.sh <regex>Any test suite can also be run if an OpenShift instance is running and you are the cluster admin by running the scripts inside of
test/cmd. The scripts will use the
8000
current project. All scripts assume cluster-admin privilege.
Most new tests belong in a specific suite under test/cmd. The only tests that belong in the parent suite (hack/test-cmd.sh) are
those that test functionality that does not depend on being logged in and having a project. For instance, some of the tests that live
there now are creating configuration files, logging in and logging out, and starting the master as an instance of Atomic Enterprise.
New suites can be added by placing scripts in test/cmd.
The os::cmd namespace provides a number of utility functions for writing CLI integration tests. All tests in all CLI test suites
must use these functions, except for some exceptions mentioned later.
The utility functions have two major functions - expecting a specific exit code from the command to be tested, and expecting something
about the output of that command to stdout and/or stderr. There are three classes of utility functions - those that expect "success"
or "failure", those that expect a specific exit code, and those that re-try their command until either some condition is met or some
time passes. The latter type of function is useful when waiting for some component to update, be it an imagestream, the project cache,
etc.
All utility functions that expect something about the output of the command to stdout or stderr use grep -Eq to run their test,
so the functions can accept either text literals or regex compliant with grep -E for input.
The utility functions use eval to run the commands passed to them, and do so in a sub-shell. In order to pass a command into a utility
function, it must be quoted. Therefore, if there is a literal string ('text') in your command, you must use double-quotes ("there is the text: 'text'") to ensure that when the command is passed to eval, the text that you wanted to be a literal string remains so
and does not get interpreted as a command itself.
Furthermore, variables can be passed in either surrounded by single quotes ('$var') or double quotes ("$var"). It is best practice
to use double-quotes in your test scripts when passing variables to the utility functions, as this will allow the test to see the
expanded variable and display your command exactly as it will be run, instead of displaying the fact that there is a variable that has
yet to be expanded.
In some cases, you may want to pass a string in to the wrapper functions that contains what looks like a bash variable but is not.
In this case, you must escape the dollar-sign when passing it in, for example: "\$notavar".
bash variable assignments done inside of a command passed to a utility function are not visible to the shell running your test.
Therefore, if your test uses bash variables and you would like to do an assignment, you should not use the os::cmd wrapper functions.
The utility functions contingent on command success or failure are:
expect_success takes one argument, the command to be run, and runs it. If the command succeeds (its return code is 0), the utility
function returns 0. Otherwise, the utility function returns 1.
In order to test that a command succeeds, pass it to os::cmd::expect_success like:
$ os::cmd::expect_success 'openshift admin config'expect_failure takes one argument, the command to be run, and runs it. If the command fails (its return code is not 0), the utility
function returns 0. Otherwise, the utility function returns 1.
In order to test that a command fails, pass it to os::cmd::expect_failure like:
$ os::cmd::expect_failure 'openshift admin policy TYPO'expect_success_and_text takes two arguments, the command to be run and the text that is expected, and runs the command. If the command
succeeds (its return code is 0) and stdout or stderr contain the expected text, the utility function returns 0. Otherwise, the
utility function returns 1.
In order to test that a command succeeds and outputs some text, pass it to os::cmd::expect_success_and_text like:
$ os::cmd::expect_success_and_text 'oadm create-master-certs -h' 'Create keys and certificates'In order to test that a command succeeds and outputs some text matching a regular expression, pass it to os::cmd::expect_success_and_text like:
$ os::cmd::expect_success_and_text "oc get imageStreams wildfly --template='{{index .metadata.annotations \"openshift.io/image.dockerRepositoryCheck\"}}'" '[0-9]{4}\-[0-9]{2}\-[0-9]{2}' # expect a date like YYYY-MM-DDexpect_failure_and_text takes two arguments, the command to be run and the text that is expected, and runs the command. If the command
fails (its return code is not 0) and stdout or stderr contain the expected text, the utility function returns 0. Otherwise, the
utility function returns 1.
In order to test that a command fails and outputs some text, pass it to os::cmd::expect_failure_and_text like:
$ os::cmd::expect_failure_and_text 'oc login --certificate-authority=/path/to/invalid' 'no such file or directory'expect_success_and_not_text takes two arguments, the command to be run and the text that is not expected, and runs the command. If the
command succeeds (its return code is 0) and stdout or stderr do not contain the text, the utility function returns 0. Otherwise,
the utility function returns 1.
In order to test that a command succeeds and does not output some text, pass it to os::cmd::expect_success_and_not_text like:
$ os::cmd::expect_success_and_not_text 'openshift' 'Atomic'expect_failure_and_not_text takes two arguments, the command to be run and the text that is not expected, and runs the command. If the
command fails (its return code is not 0) and stdout or stderr do not contain the text, the utility function returns 0. Otherwise,
the utility function returns 1.
In order to test that a command fails and does not output some text, pass it to os::cmd::expect_failure_and_not_text like:
$ os::cmd::expect_failure_and_not_text 'oc get' 'NAME'The utility functions that re-try the command until a condition is satisified or some time passes all default to trying the command once every 200ms and time-out after one minute. The functions are:
try_until_success expects at least one argument, the command to be run, but will accept a second argument setting the timeout and a third
setting the command re-try interval. try_until_success will run the given command once every interval until the timeout, expecting it to
succeed (its exit code is 0). If that occurs, the function will return 0. Otherwise, the utility function will return 1.
In order to re-try a command until it succeeds, pass it to os::cmd::try_until_success like:
$ os::cmd::try_until_success 'oc get imagestreamTags mysql:5.5'try_until_failure expects at least one argument, the command to be run, but will accept a second argument setting the timeout and a third
setting the command re-try interval. try_until_failure will run the given command once every interval until the timeout, expecting it to fail
(its exit code is not 0). If that occurs, the function will return 0. Otherwise, the utility function will return 1.
In order to re-try a command until it fails, pass it to os::cmd::try_until_failure like:
$ os::cmd::expect_success 'oc delete project recreated-project'
$ os::cmd::try_until_failure 'oc get project recreated-project'try_until_text expects at least two arguments, the command to be run and the expected text, but will accept a third argument setting the
timeout and a fourth setting the command re-try interval. try_until_text will run the given command once every interval until the timeout,
expecting stdout and/or stderr to contain the expected text. If that occurs, the function will return 0. Otherwise, the utility function
will return 1.
In order to re-try a command until it outputs a certain text, without regard to its exit code, pass it to os::cmd::try_until_text like:
$ os::cmd::try_until_text 'oc get projects' 'ui-test-project'The utility functions that allow a developer to expect a specific exit code are:
expect_code takes two arguments, the command to be run and the code to be expected from it, and runs the command. If the command returns the
expected code, the utility function returns 0. Otherwise, the utility function returns 1.
expect_code_and_text takes three arguments, the command to be run, the code and the text to be expected from it, and runs the command. If the
command returns the expected code and stdout or stderr contain the expected text, the utility function returns 0. Otherwise, the utility
function returns 1.
expect_code_and_not_text takes three arguments, the command to be run, the code to be expected and the text not to be expected from it, and
runs the command. If the command returns the expected code and stdout or stderr do not contain the expected text, the utility function
returns 0. Otherwise, the utility function returns 1.
In order to pass in a command that doesn't contain any quoted text, quote your command with single- or double-quotes:
$ os::cmd::expect_success 'oc get routes'
$ os::cmd::expect_success "oc get routes"In order to pass in a command that contains literal text, use double quotes around the command and single-quotes around your text literal:
$ os::cmd::expect_success "oc get dc/ruby-hello-world --template='{{ .spec.replicas }}'"In order to pass in a command that contains a bash variable you would like to be expanded, double-quote your command:
$ imagename="isimage/mysql@${name:0:15}"
$ os::cmd::expect_success "oc describe ${imagename}"In order to pass in a command that contains something that looks like a bash variable, but isn't, escape the $ with a forward-slash:
$ os::cmd::expect_success "oc new-build --dockerfile=\$'FROM centos:7\nRUN yum install -y httpd'"