Details
-
Bug
-
Resolution: Done
-
P3: Somewhat important
-
unversioned
-
None
Description
When cancelling an integration, the running integration task file is removed by the Scheduler. However, this creates a race condition between the Webserver and Scheduler, because the Webserver tries to read a symlink that might already been deleted by the Scheduler.
To reproduce:
1. In webui, add a new integration:
2. In webui, cancel the integration.
3. In tmux, compare scheduler and webserver pane outputs:
2017-02-25 17:31:14,926 DEBUG:utils: Removed running symlink: /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin
webserver:2017/02/25 17:31:14.927219 websocket.go:183: Error reading symlink: readlink /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin: no such file or directory /home/akeskimo/ci-working-dir/storage/running-integrations/run_builds_1488036666.thrift_bin
As we compare the timestamps, we can see that the Webserver tries to read the symlink that has already been destroyed by the Scheduler.