Uploaded image for project: 'Qt'
  1. Qt
  2. QTBUG-105576

Test stderr output for testlib selftests

    XMLWordPrintable

Details

    • Task
    • Resolution: Unresolved
    • P3: Somewhat important
    • None
    • 6.4
    • Testing: qtestlib
    • None
    • All

    Description

      At present, qtbase/tests/auto/testlib/selftests/tst_selftests.cpp's checkErrorOutput() does some checks for which tests are known to produce stderr output, exempting them from a check that requires there to be none. It does not verify that the error output is as expected, when error output is expected.

      This means that assorted error output that we generate (when we catch signals, for example) or that the operating system generates (when the signal is not stopped by our handler having reported it) is not checked. While we aren't hugely fussy about the operating system errors – they vary between operating systems and may vary from release to release – we should at least test our own output contains the information we think it should, that hopefully helps whoever is running the test to understand what happened.

      We could potentially add expected_*_error.* files (or, after QTBUG-105574 is addressed, expected_error.* files in each sub-test directory) that record the error output, for those tests that produce any. This may require (as for the existing crashes test's standard output) variant files for the cases where the output does depend on operating system – even some of testlib's own output varies.
      We may be able to limit sensitivity to system variations by accepting and ignoring actual error output lines that aren't present in the expected error output file; but this would require hand-editing of the generated output, or heuristic special-casing in the generate_expected_output.py script, to exclude operating system messages and only check for testlib's own output.

      Attachments

        Activity

          People

            cnn Qt Core & Network
            Eddy Edward Welbourne
            Vladimir Minenko Vladimir Minenko
            Alex Blasche Alex Blasche
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: