P2: Important Description
Extended queues for the VMs can cause the workitems to timeout. This leads to the VM not being terminated, unable to start. This results them blocking in the queue forever.
Jan 19 10:57:32 coin-master coin[58893]: Task exception was never retrieved Jan 19 10:57:32 coin-master coin[58893]: future: <Task finished name='Task-23382642' coro=<AgentLauncher.allocate_agent.<locals>.allocate() done, defined at /var/lib/coin/tqtc-coin-ci/src/opennebulahardwarepool.py:406> exception=TimeoutError('VM (id Jan 19 10:57:32 coin-master coin[58893]: Traceback (most recent call last): Jan 19 10:57:32 coin-master coin[58893]: File "/var/lib/coin/tqtc-coin-ci/src/opennebulahardwarepool.py", line 422, in allocate Jan 19 10:57:32 coin-master coin[58893]: await _block_until_vm_is_in_state(self.vm_id, one_vm_state.active, max_iterations=int(await self.waitingForHardwareTimeout / 10)) Jan 19 10:57:32 coin-master coin[58893]: File "/var/lib/coin/tqtc-coin-ci/src/opennebulahardwarepool.py", line 277, in _block_until_vm_is_in_state Jan 19 10:57:32 coin-master coin[58893]: await _block_until_object_is_in_state(getter, vm_id, state, "VM", ERROR_STATES, check_interval, max_iterations) Jan 19 10:57:32 coin-master coin[58893]: File "/var/lib/coin/tqtc-coin-ci/src/opennebulahardwarepool.py", line 259, in _block_until_object_is_in_state Jan 19 10:57:32 coin-master coin[58893]: raise TimeoutError(message % (type, id, current_state, state)) Jan 19 10:57:32 coin-master coin[58893]: concurrent.futures._base.TimeoutError: VM (id:235653) is still in a undesired state (1), while we wanted to see it in state: one_vm_state.active