Question & Answer
1. How can I open the failed and passed revisit sessions?
In the execution list, you can directly click on a revisit session name to access the session detail of any failed or passed revisit session. Somethimes, you will see the error icon on a revisit session and a revisit name is unable to click, which means there were some errors occur and leads to the cannot initiate the revisit. Therefore, there are no created sessions at that time.
On the other hand, you can access the session list and search by typing your preferred manual session. At a result, you will see the list of revisit sessions that related to the giving manual session.
2. Why did all of my revisit sessions fail but I was not able to see them on the sessions page?
ITA has a process called “exploring” the manual session and it will reproduce your Manual session step by step before executing revisit sessions, the purpose is to ensure that all the actions in the manual session are qualified for running ITA. So if you see all of your revisit sessions failed, it is mostly because the “exploring” process has not run successfully, you can hit the re-run icon to execute the "exploring" process again.
3. What can make the ITA “exploring” process failed?
The “exploring” process failed mostly because the user has violated ITA unsupported actions or encountered ITA limitations (you can access the Limitations section for more information). Below is the list of use cases that causing the failures:
Performe a test action too fast: Please take a waiting time about 3 seconds between 2 test actions, this will ensure the screen has loaded completely when you jump to a next test.
Tap on a button that is covered by the virtual keyboard: This may impact the element recognition ability of ITA. This is a case that we will handle in the future. For now, there is a way to go through this problem. Before you click on a button, if the virtual keyboard currently appears, please press “Back” button on the manual toolbar then click the button as normal (Android only).
Tap outside a pop-up to close it: This is an action that the user normally does when they want to close a pop-up. Unfortunately, ITA can not do this for now. You are recommended to close a pop-up by clicking on the buttons inside the pop-up properly.
Tap on dynamic elements: Buttons and texts that change over the time like news or weather forecast are the limitations of ITA. Don't worry, a team is addressing this!
Tap on buttons on device’s navigation bar (device's softkey)
- There are some problems with devices that are used for running ITA (device availability, network, etc.)
4. Can I trigger ITA on latest devices ?
Technically you can, but right now, there are some issues that currently makes ITA unstable on on latest devices like S10, S10+, etc. Some devices that we recommend for the manual:
- Galaxy Note 5
- Galaxy Note 8
- Galaxy Note 9
- Galaxy S7
- Galaxy S8
- Galaxy S8+
- Moto X Play
- Moto Z (2) Play
You can run any latest iOS devices like iPhone X, iPhone XS Max, iPhone 11, etc. with verion 11+. But please aware that right now ITA does not support Ipad or iPod.
5. Does ITA use the same version of the application as in the manual session when I have many versions of the app in the app repository?
Yes, ITA will use exact application version in the giving manual as well as any test actions, ensuring your test case will be playbacked properly on multiple devices.
6. Is there any way to select which Android versions/devices to run the tests on instead of selecting the recommended random 5/10/20 device options?
For now, we only provide a default bundle likes 5/10/20 devices bundles. In the near future, we will provide you more choices by giving an ability to custome your bundles based on your need.
7. why does a revisit session automatically has the scroll actions in my test case although I did not scroll or swipe in my manual session?
Don't worry, those are the scrolling actions that ITA performed. The reason is because an elements on any specific resolution might have different locations especially on the list view, therefore, to ensure an elements will be selected properly, ITA will try to find an element by scrolling a couple of times.
8. I selected Top 20 recommended but the tests were en-queued on 18 devices only.
This is because these devices are offline at that time. At a result, ITA does not always “revisit” on full of 20 devices. In the near future, we will provide you an ability to properly cover a range of devices for the revisit.
9. What is the difference between a Failed revisit and a Blocked revisit?
ITA uses deep learning to "know" which element in the Revisit session matches the element used in the Manual session. Unlike web testing where the HTML is standard across browsers, on mobile how the element is referenced differs by device type, manufacturer and, in some cases, test run. That's right, some apps dynamically generate the XML!
For the cases where the deep learning is unable to find an element, ITA presents a Blocker Remediation screen to allow the tester to "guide" the deep learning to ensure the continuation of the test case.
Blocked Test: The deep learning could not find an element on the screen during a Revisit session. You can “remediate” the blocker and the test will rerun without encounter an issue again.
Failed Test: This typically happens when one of the restricted user actions have been performed during the manual session, or the reason might come from the device issues like network, high CPU, etc. We are improving the user experience related to this and continuing to implement more gestures / user actions to eliminate these scenarios long-term.
10. Why my revisit failed?
Be aware of OS version that suitable your application (CPU architect): if there is a device in device bundle with version that your app doesn’t support (only support 32bit or 64bit), that session running in that device will be failed.
The revisit device went offline unexpectedlly
Network issue: The connection to the device sometimes is not stable, it could make ITA failed unexpectedly.
11. How long would it take for ITA to execute a manual session and how will I know when this task is completed? Will there be a notification?
It takes about an hour as avarage for ITA to run on 20 devices bundle with a manual session that has 20 steps. Either the when ITA was kicked-off or ITA was completed, you will receive an email for the notification.