QSR 1.2 Release Test Automation Development

This user story shall contain all the tasks needed for the QSR 1.2 release test automation development. 

Please create corresponding tasks for the wishlist implementation.

Wishlist for the project RTA: (to be groomed and prioritized)

1. Jenkins side considerations
   1. Please expand the test configuration coverage to meet the QSR 1.2 requirements. We need some 11 configs instead of the current three.
      1. Please see the indicative configuration list from QTQAINFRA-3905. At need, we can fine-tune it.
   2. Please support running the RTA for different QSR software branches (e.g., 1.1.x and 1.2.x)
      1. Currently, only the latest branch is supported.
      2. Check what the best approach with all other requirements is—parameter, copy of the test job, or something else.
   3. Please support running the RTA for different QSR software branches.
      1. The top priority for QSR 1.2 is Qt 5.15.1 and 5.12.9. You can test others if there is time.
      2. Currently, this is supported by the parameter.
   4. Please merge the smoke tests and detailed tests into one test job in Jenkins.
      1. No need for two separate jobs for these.
   5. Have test jobs for both staging and production.
2. Test script considerations
   1. Please rename the test cases in the scripts according to the final names of the test cases in the Test Rail
      1. We are going to include the data into the TÜV reports and customer deliverables. So we prefer unified naming across the test suites.
   2. Please refactor at the need to run the GUI tests separately in the QSR nightly test system
      1. The system will get the latest sources from the Git. We do not need to run the installation tests. We want to run the GUI tests automatically in that system. (Ubuntu 18.04)
      2. Check if we want to share the GUI tests with the customers?
   3. We need to indicate in the manual test suite, which tests have been automated. So we pick only the necessary ones to the manual test plan.
      1. And when there are more automated cases added, we need to sync these things to avoid overlapping work.
   4. In case that Windows 7 32-bit MinGW compilations cannot be done in Qt Creator with Squish, use only command-line interface for testing.
   5. Combine tests to scenarios for performance efficiency.
      1. Manual tests can be run separately, and they have dependencies, the automated tests can be run in a batch.
   6. Check what is the feasible approach to managing both release and debug builds
   7. Verify the QSR runtime screen in the host environment if not yet done
   8. Use the existing capacity tests (separate Python scripts) and create GUI tests for those.
3. Test Rail considerations
   1. Please organize the test suite similarly than for the other suites
      1. Include the needed descriptions and other data.
      2. The covered requirements must be stated.
      3. If the test script covers safety requirements, it must be stated in the title.
   2. Please use Test Rail integration for the QSR RTA results.
      1. Currently, a dedicated QSR RTA sandbox is in use. Leave it for debugging and developing the new tests etc.
      2. For the production-ready tests, we prefer to use the actual QSR project in the Test Rai. This approach helps us to get the actual results in one place without too much clutter.
   3. Please use the example template for the test plan.
      1. Assumingly the integration script requires one open test plan.
      2. Post-process the test plan information after execution. Please move the SHA information to the description, name suite properly, and mark it for the corresponding milestone.
      3. If possible, put the test results from different Qt versions under one test plan.
   4. Please consider forking the Test Rail integration Python script to QSR specific.
4. Test scope considerations
   1. Please automate the QSR 1.0, QSR 1.1, and QSR 1.2 related tests.
      1. The work is continuous despite the certification and release activities.
   2. Please consider the following test runs in the test plan
      1. Test run: Installer tests + content verification
      2. Test run: Tooling setup (binary compatibility, tool building)
      3. Test run: Build documentation
      4. Test run: Autotests including integration tests (command line)
      5. Test run: Qt Creator tooling verification
      6. Test run: Qt Design Studio tooling verification
      7. Test run: Runtime verification. (event sending, etc.)
      8. Test run (future improvement for QSR 1.3): Generate HW images
      9. Test run (future improvement for QSR 1.3): Deploy to the target hardware.
      10. Test run (future improvement for QSR 1.3): Run code coverage measurements
      11. Test run (future improvement for QSR 1.3): Run Valgding tools
      12. Test run (future improvement for QSR 1.3): Run fuzz testing.
5. Qt Design Studio related test considerations
   1. The existing Qt Design Studio (QDS) testing has behavior-driven development (BDD) test scripts in use.
      1. Consider if the QSR 1.2 QDS tests could have the same approach.
      2. Consider having all QSR tests refactored into BDD in the future releases.
6. Future improvement opportunities consideration
   1. The animation features and state supports generate QML code that is laborious to test manually.
      1. Consider deploying the froglogic Code Coverage measurements for the generated QML code.