Uploaded image for project: 'Qt Quality Assurance Infrastructure'
  1. Qt Quality Assurance Infrastructure
  2. QTQAINFRA-1171

Race conditions between WebServer and Scheduler while cancelling integration

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • P3: Somewhat important
    • production
    • unversioned
    • Coin (obsolete)
    • None

    Description

      When cancelling an integration, the running integration task file is removed by the Scheduler. However, this creates a race condition between the Webserver and Scheduler, because the Webserver tries to read a symlink that might already been deleted by the Scheduler.

      To reproduce:
      1. In webui, add a new integration:
      2. In webui, cancel the integration.
      3. In tmux, compare scheduler and webserver pane outputs:

      2017-02-25 17:31:14,926 DEBUG:utils: Removed running symlink: /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin

      webserver:2017/02/25 17:31:14.927219 websocket.go:183: Error reading symlink: readlink /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin: no such file or directory /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin

      As we compare the timestamps, we can see that the Webserver tries to read the symlink that has already been destroyed by the Scheduler.

      Attachments

        Activity

          People

            aakeskimo Aapo Keskimölö
            aakeskimo Aapo Keskimölö
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: