Run Automation Test

1. Release Task Inputs

After adding Kobiton Automation Test Executor task to your pipeline, you are required to fill in some inputs. We are going to split them into 3 parts from top to bottom:

Part 1: Required inputs

• Display name: The Kobiton API Credentials that you created in the previous section. You can add more credentials if you want. For now let's pick one from the dropdown list.

• Kobiton executor server credentials: The Kobiton Executor Server credentials in your service connnection store.

• Git repository URL: This is the URL to your GIT repository that contains your test script. If the repository is private, please use the SSH URL.

• Git repository branch: The branch of the repository to execute automation test script.

• Git repository SSH key: The SSH Key to access the above Git repository if it is set as private. In this document, we will use Kobiton's public sample repository so we will leave this field empty.

  
  
  

Part 2: Desired Capabilities

• Upload New App Version:

  • By letting this checked, you are required to upload a new version of your existing app on Kobiton App Repository.

    • App file path: Select an apk/ipa file in your release artifact.

    
    
    

    • App ID: The App ID of your existing app on Kobiton App Repository. You can login to https://portal.kobiton.com/apps and check your app ID.

    

    

    As an example in the above picture, once the uploading process is successfully completed, a value of “4058” will be parsed to your KOBITON_SESSION_APPLICATION environment variable.

    
    
    

  • If you leave this field unchecked, you have the option to input your existing App ID or a URL to your apk file in App ID input. This value will be directly passed to your KOBITON_SESSION_APPLICATION env variable, so make sure to type in the correct format for “app” field in desiredCaps like below:

    
    
    

• Use Custom Device: This will check if you want to define the device related parameters in your desiredCaps via env variables.

  • If you let this unchecked, the extension will ignore the env variables.

  • If this is checked, you are required to type in 3 fields which are self-explanatory:

    • Kobiton Cloud Device Name: KOBITON_DEVICE_NAME

    • Device platform version: KOBITON_SESSION_PLATFORM_VERSION

    • Device platform: KOBITON_DEVICE_PLATFORM

    
    
    

Part 3: Execution inputs:

• Root Directory: The root directory is the path to git your repository.

• Commands: The command lines to execute your test script. These commands will be called at the root directory of your Git repository. The commands will depend on your test project. In this example, we will use the command “mvn test” to build the project and run automation test script.

• Wait for execution:

  • If this is unchecked, the release will continue to the next release task. The execution job ID of this task will be printed, but not the console logs or report URL.

  • If this is checked, the task will wait for the execution to be either COMPLETED (commands ran successfully) or FAILED (cannot clone git repository, server error,...) then print the console log and report URL (if it existed).

    • Log Type: User can pick a type of log to be printed. Combined means the output logs and error logs will be printed in chronological order. The Separated option allows you to choose one of the 2 types of logs.
      • It is recommended to log your test case result with appropriate log type. For example, in Java you can use System.err.println to print test case errors, this will be consumed in the error logs while System.out.println will return texts in output logs.
    • Terminate release pipeline on failed job: If this is checked, once the execution job status is FAILED, an error will be returned and the whole release is terminated (All the following release tasks below this task will be skipped).

    
    
    

2. Execute the release

• Once everything is set up, click on the Save button on the right.

  

• Leave everything blank for this time, then click OK.

  

• Click Create Release button on the right to create a new release.

  

• A new window will appear, leave everything blank and click Create.

  

• A notification will appear, click on Release-1 to view the release status.

  

• When the release in completed, you can check the release log by clicking on the Logs button right below the release Stage.

  

• Click on the Kobiton Automation Test Executor task to view its logs

  

  

  If you chose to Wait for execution, in happy cases (no server error, all test cases passed) you can see the test result, execution job ID and report URL

  

  

  The Job ID is formatted with prefix “rN” where N is the release ID of your Azure Devops project. Clicking on the Report URL will lead you to the report interface.
  

• Please contact us to be provided username, password and domain.

• After the session is created successfully, we can look up the session in the Kobiton Portal https://portal.kobiton.com/sessions .

  

3. Overall

Here is a simple case we made to demonstrate how to trigger the automation test script on Kobiton devices.

• Filling in the inputs

  

  

  • Our public git repoitory: https://github.com/kobiton/samples.git

  • In Root directory input of the execution task. You have to change directory to test scripts before run the test.

    ./java/java_testng_junit
    

  • In command input of the execution task to run the test.

    $ mvn test -Dtest=azureTestNG
    

  • Since this demo repository is public, you can leave the SSH Key field blank and use HTTPS URL in the Kobiton Azure Devops Release Task.

• Preparing the automation script

  Due to not using custom device, the script will set default value for these following capabilities:

  Environment	Description
  KOBITON_USERNAME	The kobiton Username you set in Azure Service conneciton
  KOBITON_API_KEY	The Kobiton API Key you set in Azure Service connection
  KOBITON_DEVICE_NAME	Value of the device name in Kobiton Extension Task
  KOBITON_SESSION_APPLICATION	Value of the app in Kobiton Extension Task