Finding sub-images on the screen (or in other images)

Overview¶

In more complex scenarios (typically involving multiple applications) it is sometimes not possible to automate all the processes involved.

In such situations it may be an acceptable workaround to check if a certain sub-image is on the screen to decide whether an activity failed or not, or to decide whether to take specific actions or not.

Here is an example of this which uses an external Java program called find_image_in_image (download find_image_in_image ) for the image search:

import os
import os.path
import subprocess

def main():
    sub_image_path = squishinfo.testCase + "/test_image_sub.png"
    results = image_coordinates_on_screen(sub_image_path)
    if results["FOUND"] == "YES":
        x = results["FOUND_AT_X"]
        y = results["FOUND_AT_Y"]
        test.passes("Image has been found at (x/y): %s/%s" % (x, y))
    else:
        test.fail("Image has not been found (or invalid parameters to external tool)")

    test.log("FIND TOOL RESULTS:")
    for name, value in results.iteritems():
        test.log("  %s: %s" % (name, value))

    return

def image_coordinates_on_screen(sub_image_path):
    find_tool = squishinfo.testCase + "/../find_image_in_image.bat"
    cmd = [find_tool, sub_image_path]
    output, returncode = os_capture(cmd, squishinfo.testCase)
    output = output.replace("\r\n", "\n")
    output = output.replace("\r", "\n")
    output = output.split("\n")
    results = {}
    for l in output:
        if len(l) == 0:
            continue
        name, value = l.split("=", 1)
        results[name] = value
    return results

def os_capture(args, cwd=None):
    if cwd is None:
        cwd = os.getcwd()
    proc = subprocess.Popen(
        args=args,
        cwd=cwd,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT)
    stdout = proc.communicate()[0]
    return stdout, proc.returncode

test.py

Example output if the sub-image has been found:

Pass Image has been found at (x/y): 465/859
Log  FIND TOOL RESULTS:
Log    SUB_IMAGE_TOTAL_PIXELS: 682
Log    IMAGE_TOTAL_PIXELS: 2304000
Log    RUN_TIME_MILLIS: 258
Log    FAILED_PIXEL_COUNT: 0
Log    ALLOWED_PIXEL_FAILS_COUNT: 0
Log    FAILED_PIXEL_PERCENT: 0.0
Log    FOUND: YES
Log    FOUND_AT_Y: 859
Log    FOUND_AT_X: 465

Example output if the sub-image has not been found:

Fail Image has not been found (or invalid parameters to external tool)
Log  FIND TOOL RESULTS:
Log    FOUND: NO
Log    SUB_IMAGE_TOTAL_PIXELS: 682
Log    ALLOWED_PIXEL_FAILS_COUNT: 0
Log    IMAGE_TOTAL_PIXELS: 2304000
Log    RUN_TIME_MILLIS: 620