Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-244

Tracebench regression of June 2011 on mac and linux - false positives?

    XMLWordPrintable

Details

    • Bug
    • Resolution: Out of scope
    • P1: Critical
    • 2012q3
    • 2011q3
    • Performance tracking
    • None
    • Linux and Mac

    Description

      The BM tool indicates a severe regression in the tst_QTraceBench:trace(textedit) benchmark in qt/qt#4.7:

      http://bmc.test.qt.nokia.com:8080/bm2/analysis/tsbm.shtml?db=bm&host=oslo-bm-ubuntu104-64-1&platform=linux-g%2B%2B-64&branch=qt%2Fqt%234.7&sha11=3ec5f0c801645cf8beac52f7eab0c97190f9b46d&sha12=latest&sha1_sel=e9ab8493053a7dd56cde058e736f46feafb378d0&benchmark=tst_QTraceBench%3Atrace%28textedit%29&metric=WalltimeMilliseconds&difftol=1.1&durtolmin=3&durtolmax=10

      Jo was however unable to reproduce this regression manually on his Linux box (barbarella). This suggests that the regression is a false positive caused e.g. by some unintended modification of the test environment. The likelihood that this is not caused by a change in Qt is further supported by Samuel (the owner of the benchmark in question) not finding any obvious commits that could have caused this in the snapshot interval around the change.

      It should be investigated whether this is really a false positive and what could possibly have caused it.

      Interestingly, the same regression also appears to occur on mac, but for a snapshot that was uploaded around two days earlier:

      http://bmc.test.qt.nokia.com:8080/bm2/analysis/tsbm.shtml?db=bm&host=oslo-bm-mac-mini-64-1&platform=macx-g%2B%2B&branch=qt%2Fqt%234.7&sha11=earliest&sha12=latest&sha1_sel=597117790ad86a1e68526fa7ccd1a38fca2b55f3&benchmark=tst_QTraceBench%3Atrace%28textedit%29&metric=WalltimeMilliseconds&difftol=1.1&durtolmin=3&durtolmax=10

      It should be investigated whether this too is a false positive, and also why the change does not occur for the same snapshot as on Linux.

      Attachments

        Issue Links

          Activity

            People

              seahumad [BB] Sergio Ahumada (Inactive)
              bdo Jo Asplin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: