Appium AI: Natural language locators (Beta)

Overview

Appium AI enables automation scripts to locate UI elements using natural language instead of traditional selectors such as XPath or accessibility IDs.

Test authors describe elements in plain language while keeping existing Appium workflows. This improves test resilience in environments where selectors are brittle, unavailable, or frequently changing.

Availability

Appium AI (Natural Language Locators) is in Beta.

This feature requires access to a compatible LLM provider. Kobiton supports OpenAI, Azure OpenAI, Anthropic Claude, self-hosted OpenAI-compatible endpoints such as Ollama, and the Kobiton-hosted LLM.

Most providers require your own API credentials or a reachable local endpoint. The Kobiton-hosted LLM is the exception: it needs no third-party account and is available where the Mac mini host can reach llm.kobiton.com. It is unavailable on an air-gapped host or where the domain is blocked.

A ChatGPT subscription alone does not enable this feature. OpenAI, Azure OpenAI, and Anthropic Claude require API access.

This feature is available on Private and Local devices. Appium AI is not available on Public Cloud devices.

To set a default LLM provider for your organization, see Configure Appium AI settings for your organization.

Your lab administrator configures the LLM provider and model defaults for your environment. Contact them to change the host-wide setup.

Natural language locators

Appium AI extends existing Appium workflows by allowing findElement(…​) to accept natural language descriptions instead of traditional selectors.

python
element = driver.find_element("natural", "The login button at the bottom")
element.click()
For Appium java-client 10.x and above, extend the custom locator strategy to use Appium AI. See this guide for instructions.

Appium AI interprets the description, identifies the most relevant UI element, and returns an interactable element for use in the test.

Natural language locators identify elements only. Actions such as click(), send_keys(), or assertions must still be performed explicitly in the test script.

How Appium AI resolves elements

Appium AI supports two locator strategies:

Strategy Behavior

uitree

Reads the view hierarchy or accessibility attributes exposed by the application and returns a matching element by XPath or CSS selector. This is the default strategy.

visionvirtual

Analyzes a screenshot of the current view with a vision-capable LLM and returns a virtual element representing the matched region. Use this for canvas-based apps, games, and other graphics-heavy apps that lack UI metadata.

Appium AI resolves the strategy once when the session starts. You cannot switch strategies during a test run, and a single session cannot use both. To exercise both strategies, run separate sessions.

Before you begin

  • Your environment is configured to support natural language locators. If unsure, contact your lab administrator or Kobiton Support.

  • You are familiar with basic Appium commands such as findElement(…​).

  • You are using Private and Local devices (natural language locators are not available on Public Cloud devices).

Choose a locator strategy

Set the strategy per session with the kobiton:locatorStrategy capability:

python
capabilities = {
    # ...
    'kobiton:locatorStrategy': 'visionvirtual',
}

A capability passed in the session takes precedence over the host-wide default your administrator set.

visionvirtual uses the vision model already configured for Appium AI. You do not need to set kobiton:llmVisionModel to use it. To route vision requests through a different model or provider, see Override Appium AI settings per session.

Appium AI does not detect canvas-based apps. To use vision-based resolution, set kobiton:locatorStrategy to visionvirtual.

Use a natural language locator

Pass a descriptive string into findElement(…​) using the natural locator strategy.

python
element = driver.find_element("natural", "The login button at the bottom")
element.click()
java
WebElement el = driver.findElement(ByNatural.natural("the Accessibility button"));
el.click();
The Java example uses the custom ByNatural class to extend the default locator strategy. See this guide for instructions.

Write effective descriptions

Use clear, specific descriptions to help identify the correct element.

Good examples:

driver.findElement(ByNatural.natural("the Accessibility button"));
driver.findElement(ByNatural.natural("the Username field"));
driver.findElement(ByNatural.natural("the City dropdown list"));

Less effective examples:

driver.findElement(ByNatural.natural("Click button"));
driver.findElement(ByNatural.natural("Select item"));

When multiple similar elements are present, include additional context such as position, label, or surrounding UI to improve accuracy.

Expected behavior

Appium AI evaluates the description and returns the most relevant matching element using the locator strategy set for the session. uitree returns a selector resolved from the view hierarchy. visionvirtual returns a virtual element mapped to coordinates on the screenshot.

Results may not be deterministic. If multiple elements match the description, refine the description to improve accuracy.

When the visionvirtual strategy cannot find the element on the current screen, it returns a not-found result rather than a best-guess match. Scroll an off-screen element into view, or refine the description, if a visible element is missed.

Natural language locators work alongside traditional selector strategies and are best used in combination with them, rather than as a complete replacement.

Limitations and considerations

Natural language locators are not intended to replace all selector strategies. Use them where they provide value, and rely on traditional selectors when precision is required.

This approach may not be suitable when:

  • Stable and reliable selectors already exist

  • Exact element matching is required (for example, when multiple similar elements are present)

  • The environment does not support natural language locators (such as Basic Appium or unsupported configurations)

Current limitations:

  • Natural language locators identify elements only. They do not perform actions such as clicking or typing.

  • Script generation and full AI-driven test creation are not supported.

  • Automatic fallback from traditional locators to natural language is not supported.

Element identification quality depends on the locator strategy set for the session. uitree depends on the view hierarchy exposed by the application, and visionvirtual depends on the visual clarity and layout of the current screen. Clear, specific descriptions improve results for both strategies.

Troubleshooting

If an element is not found or the wrong element is returned:

  • If the app renders its UI on a canvas, set kobiton:locatorStrategy to visionvirtual. The default uitree strategy cannot locate elements in an app that exposes no view hierarchy.

  • Use more specific language in the description.

  • Include additional context such as position, labels, or surrounding elements.

  • Verify that the UI element is visible and accessible in the view hierarchy.

  • When using uitree, confirm that the application exposes sufficient UI metadata (view hierarchy or accessibility attributes) for element identification.

  • When using visionvirtual, ensure the target element is clearly visible on the current screen.

  • If your script sets kobiton:llmBaseUrl, confirm it also sets kobiton:llmApiKey. Sessions that override the base URL without providing a key fail on the first LLM request.

If natural language locators are not working as expected, contact your lab administrator to confirm that the LLM provider in use (OpenAI, Azure OpenAI, Anthropic Claude, Ollama, or the Kobiton-hosted LLM) is configured correctly.

If issues persist, contact Kobiton Support.

Next steps

To override Appium AI provider or model settings for a specific session, see Override Appium AI settings per session.