selftests: allow runners to override the timeout

The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.

To help with all this:

  o Enhance documentation to avoid future increases of insane timeouts
  o Add the option to allow overriding the default timeout with test
    runners with a command line option

Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>

diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index 12b575b76b20b1ded2f33b28746f32244a16ac44..dd214af7b7ff28f0f585c581a978704bdcfd051d 100644 (file)
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times::
 
 For other features see the script usage output, seen with the `-h` option.
 
+Timeout for selftests
+=====================
+
+Selftests are designed to be quick and so a default timeout is used of 45
+seconds for each test. Tests can override the default timeout by adding
+a settings file in their directory and set a timeout variable there to the
+configured a desired upper timeout for the test. Only a few tests override
+the timeout with a value higher than 45 seconds, selftests strives to keep
+it that way. Timeouts in selftests are not considered fatal because the
+system under which a test runs may change and this can also modify the
+expected time it takes to run a test. If you have control over the systems
+which will run the tests you can configure a test runner on those systems to
+use a greater or lower timeout on the command line as with the `-o` or
+the `--override-timeout` argument. For example to use 165 seconds instead
+one would use:
+
+   $ ./run_kselftest.sh --override-timeout 165
+
+You can look at the TAP output to see if you ran into the timeout. Test
+runners which know a test must run under a specific time can then optionally
+treat these timeouts then as fatal.
+
 Packaging selftests
 ===================
 

diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
index 294619ade49fedcc12057cd04f8f6f1b134e0e47..1c952d1401d46bd8f69534330c8edd6915646b3d 100644 (file)
--- a/tools/testing/selftests/kselftest/runner.sh
+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -8,7 +8,8 @@ export logfile=/dev/stdout
 export per_test_logging=
 
 # Defaults for "settings" file fields:
-# "timeout" how many seconds to let each test run before failing.
+# "timeout" how many seconds to let each test run before running
+# over our soft timeout limit.
 export kselftest_default_timeout=45
 
 # There isn't a shell-agnostic way to find the path of a sourced file,
@@ -90,6 +91,14 @@ run_one()
                done < "$settings"
        fi
 
+       # Command line timeout overrides the settings file
+       if [ -n "$kselftest_override_timeout" ]; then
+               kselftest_timeout="$kselftest_override_timeout"
+               echo "# overriding timeout to $kselftest_timeout" >> "$logfile"
+       else
+               echo "# timeout set to $kselftest_timeout" >> "$logfile"
+       fi
+
        TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
        echo "# $TEST_HDR_MSG"
        if [ ! -e "$TEST" ]; then

diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
index 97165a83df632080a1203f733ef8c3c3d9a43c32..9a981b36bd7ff28e552d76149284c3e2a0a7d274 100755 (executable)
--- a/tools/testing/selftests/run_kselftest.sh
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -26,6 +26,7 @@ Usage: $0 [OPTIONS]
   -l | --list                  List the available collection:test entries
   -d | --dry-run               Don't actually run any tests
   -h | --help                  Show this usage info
+  -o | --override-timeout      Number of seconds after which we timeout
 EOF
        exit $1
 }
@@ -33,6 +34,7 @@ EOF
 COLLECTIONS=""
 TESTS=""
 dryrun=""
+kselftest_override_timeout=""
 while true; do
        case "$1" in
                -s | --summary)
@@ -51,6 +53,9 @@ while true; do
                -d | --dry-run)
                        dryrun="echo"
                        shift ;;
+               -o | --override-timeout)
+                       kselftest_override_timeout="$2"
+                       shift 2 ;;
                -h | --help)
                        usage 0 ;;
                "")