Region¶
-
class
Region
¶
- Region is a rectangular area on a screen, which is defined by
- its upper left corner (x, y) as a distance relative to the upper left corner of the screen (0, 0) and
- its dimension (w, h) as its width and height.
x, y, w, h are integer numbers counting a distance in pixels.
A region knows nothing about its visual content (windows, pictures, graphics, text, ...). It only knows the position on the screen and its dimension.
New regions can be created, based on an existing region: you can extend a region in all directions or get the adjacent rectangle up to the bounds of the screen horizontally or vertically.
The visual content of a region is evaluated by using methods like
Region.find()
, which looks for a given rectangular pixel pattern
(defined by a Pattern
object) within the region. The matching
content in the region has a similarity between 0 (not found) and 1 (found and
it is per pixel exactly matches to the pattern). The find can be advised, to
search with a minimum similarity, so that some minor variations in shape and
color can be ignored. If nothing else is specified, Sikuli searches with a
minimum similarity of 0.7, which does what is expected in general cases.
(This default value can be changed in Settings.MinSimilarity
.)
Find operations return a
Match
object, which has all the attributes and methods of a region
and can be used in exactly the same way (e.g. to find or click another
target within it). A Match
has the dimension of the pattern used
for searching and also knows the position where it was found and its similarity
score. A region preserves the best match of the last successful find operation
and all matches of the last successful Region.findAll()
(accessible
with Region.getLastMatches()
. You can wait for patterns to show up
using Region.wait()
, to vanish using Region.waitVanish()
or
just check whether a pattern exists
without the need to handle
exceptions.
Sikuli supports visual event driven programming. You can tell a region to observe that something appears, vanishes or changes. It is possible to wait for the completion of an observation or let it run in the background while your following script continues running. When one of the visual events happens, a handler in your script is called. Each region has one observer and each observer can handle multiple visual events. It’s your responsibility to stop an observation.
Creating a Region, Setting and Getting Attributes¶
In this chapter, you can find information on how to create a new region object. The attributes of a region object can be accessed directly or via method calls, e.g. to change their position and/or dimension. Here you will find the HowTo’s.
-
class
Region
-
Region
(x, y, w, h)¶ -
Region
(region) -
Region
(Rectangle) Create a region object
Parameters: - x – x position of a rectangle.
- y – y position of a rectangle.
- w – height of a rectangle.
- h – width of a rectangle.
- region – an existing region object.
- rectangle – an existing object of Java class Rectangle
Returns: a new region object.
In addition to creating a region by using the tool provided by the IDE, a region can be created by specifying a rectangle. This is how the visual representation in the IDE of such a region is internally set up in the script. A region can also be created by users in run-time using
Region.selectRegion()
.You can create a region based on another region. This just duplicates the region (new object). This can be useful, if you need the same region with different attributes, such as another observation loop or use
Region.setThrowException()
to control whether throwing an exception when finding fails. Another way to create a region is to specify a rectangle object or to extend an existing region.Note: The position and dimension attributes are named x, y (values representing the top left corner) and w, h (width and height). You might use these or the get/set methods.
topLeft = Location(reg.x, reg.y) # equivalent to topLeft = reg.getTopLeft() theWidth = reg.w # equivalent to theWidth = reg.getW() reg.h = theWidth # equivalent to reg.setH(theWidth)
Note: Additionally you might use
selectRegion()
to interactively create a new region.
-
setX
(number)¶ -
setY
(number)¶ -
setW
(number)¶ -
setH
(number)¶ Set the respective attribute of the region to the new value. This effectively moves the region around and/or changes its dimension.
Parameters: number – the new value
New in version X1.0-rc2.
-
moveTo
(location)¶ Set the position of this region regarding it’s top left corner to the given location (the x and y values are modified).
Parameters: location – location object becomes the new top left corner Returns: the modified region object reg.moveTo(anotherLocation) # equivalent to reg.setX(anotherLocation.x); reg.setY(anotherLocation.y)
-
setROI
(x, y, w, h)¶ -
setROI
(rectangle) -
setRect
(x, y, w, h)¶ -
setRect
(rectangle) Both methods are doing exactly the same: setting position and dimension to new values. The motivation for two names is to make scripts more readable:
setROI()
is intended to restrict the search to a smaller area to speed up processing searches (region of interest), whereassetRect()
should be used to redefine a region (which could be enlarging it).Parameters: - x, y, w, h (all) – the attributes of a rectangle
- rectangle – a rectangle object
Returns: None
New in version X1.0-rc2.
-
morphTo
(region)¶ Set the position and dimension of this region to the corresponding values of the region given as parameter. (see:
setRect()
)Parameters: region – a region object Returns: the modified region object reg.morphTo(anotherRegion) # equivalent to r = anotherRegion; reg.setX(r.x); reg.setY(r.y); reg.setW(r.w); reg.setH(r.h)
New in version X1.0-rc2.
-
getTopLeft
()¶ -
getTopRight
()¶ -
getBottomLeft
()¶ -
getBottomRight
()¶ Get the location of the region’s respective corner
Returns: Location object
-
getScreen
()¶ Returns the screen object that contains this region.
Returns: a new Screen
objectThis method only makes sense in Multi Monitor Environments, since it always returns the default screen in a single monitor environment.
-
getLastMatch
()¶ -
getLastMatches
()¶ Returns: the best match as a Match
object or one or more match objects as anIterator
object respectivelyAll successful find operations (explicit like
find()
or implicit likeclick()
), store the best match in the lastMatch attribute of the region that was searched.findAll()
stores all found matches into lastMatches atribute of the region that was searched as an iterator.To access these attributes use
region.getLastMatch()
orregion.getLastMatches()
respectively.How to use the iterator object returned by getLastMatches() is documented here.
-
setAutoWaitTimeout
(seconds)¶ Set the maximum waiting time for all subsequent find operations.
Parameters: seconds – a number, which can have a fraction. The internal granularity is milliseconds. This method enables all find operations to wait for the given pattern to appear until the specified amount of time has elapsed. The default is 3.0 seconds. This method is intended for users to override this default setting. As such it lets
Region.find()
work likeRegion.wait()
, without being able to set an individual timeout value for a specific find operation.
-
getAutoWaitTimeout
()¶ Get the current value of the maximum waiting time for find operations.
Returns: timeout in seconds
-
Extending a Region¶
These methods (exception inside()
) return a new region object, that is
based on the specified region (sometimes called spatial operators).
The range parameter, if given as positive integer number, restricts the
dimension of the new region (width and/or height respectively) to that
value. If range is not specified, the new region extends to the respective
boundary of the screen the given region belongs to. An exception is nearby()
,
which uses 50 as its default range.
Note: In all cases the current region remains unchanged.
Note: In all cases the new region does not extend beyond any boundary of the screen that contains the given region.
-
class
Region
-
offset
(location)¶ Returns a new Region object, whose upper left corner is relocated adding the location’s x and y value to the respective values of the given region. Width and height are the same. So this clones a region at a different place.
Parameters: location – a Location
objectReturns: a new Region
objectnew_reg = reg.offset(Location(xoff, yoff)) # same as new_reg = Region(reg.x + xoff, reg.y + yoff, reg.w, reg.h)
-
inside
()¶ Returns the same object. Retained for upward compatibility.
Returns: the Region
objectThis method can be used to make scripts more readable.
region.inside().find()
is totally equivalent toregion.find()
.
-
nearby
([range])¶ Returns a new
Region
that includes the nearby neighbourhood of the the current region. The new region is defined by extending the current region’s dimensions in all directions by range number of pixels. The center of the new region remains the same.Parameters: range – a positive integer indicating the number of pixels (default = 50). Returns: a Region
object
-
above
([range])¶ Returns a new
Region
that is defined above the current region’s top border with a height of range number of pixels. So it does not include the current region. If range is omitted, it reaches to the top of the screen. The new region has the same width and x-position as the current region.Parameters: range – a positive integer defining the new height Returns: a Region
object
-
below
([range])¶ Returns a new
Region
that is defined below the current region’s bottom border with a height of range number of pixels. So it does not include the current region. If range is omitted, it reaches to the bottom of the screen. The new region has the same width and x-position as the current region.Parameters: range – a positive integer defining the new height Returns: a Region
object
-
left
([range])¶ Returns a new
Region
that is defined left of the current region’s left border with a width of range number of pixels. So it does not include the current region. If range is omitted, it reaches to the left border of the screen. The new region has the same height and y-position as the current region.Parameters: range – a positive integer defining the new width Returns: a Region
object
-
right
([range])¶ Returns a new
Region
that is defined right of the current region’s right border with a width of range number of pixels. So it does not include the current region. If range is omitted, it reaches to the right border of the screen. The new region has the same height and y-position as the current region.Parameters: range – a positive integer defining the new width Returns: a Region
object
-
Finding inside a Region and Waiting for a Visual Event¶
Besides acting on visual objects, finding them is one of the core functions of Sikuli.
PS: means, that either a Pattern
or a string (path to an image
file or just plain text) can be used as parameter. A find operation is
successful, if the given image is found with the given minimum similarity or
the given text is found exactly. Similarity is a value between 0 and 1 to
specify how likely the given image looks like the target. By default, the
similarity is 0.7 if an image rather than a pattern object with a specific
similarity is given to Region.find()
.
Normally all these region methods are used as reg.find(PS)
, where reg is a
region object. If written as find(PS)
it acts on the default screen, which is
an implicit region in this case (see: SCREEN as Default Region).
But in most cases it is a good idea to use
region.find()
to restrict the search to a smaller region in order to speed up
processing.
If a find operation is successful, the returned match is additionally stored
internally with the region that was used for the search. So instead of using a
variable to store the match ( m = reg.find()
), you can use reg.getLastMatch()
to access
it afterwards. Unsuccessful find operations will leave these values unchanged.
By default, if the visual object (image or text) cannot be found, Sikuli
will stop the script by raising an Exception FindFailed.
This follows the standards of the Python language, so that you could handle such exceptions using
try: ... except: ...
.
If you are not used to programming using the Python language or because of other reasons, you might just want to bypass the exception handling, which means just ignoring it (None is returned in that case). Or you might interactively react on a FindFailed situation (e.g. optionally repeat the find). Read more about concepts and options at: Exception FindFailed.
If you have multiple monitors, please read Multi Monitor Environments.
Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the file system bundle-path). Behind the curtain, the images itself are specified simply by using a string containing the file name (path to an image file).
-
class
Region
-
find
(PS)¶ Parameters: PS – a Pattern
object or a string (path to an image file or just plain text)Returns: a Match
object that contains the best match or fails if not foundFind a particular GUI element, which is seen as the given image or just plain text. The given file name of an image specifies the element’s appearance. It searches within the region and returns the best match, which shows a similarity greater than the minimum similarity given by the pattern. If no similarity was set for the pattern by
Pattern.similar()
before, a default minimum similarity of 0.7 is set automatically.If autoWaitTimeout is set to a non-zero value, find() just acts as a wait().
Side Effect lastMatch: the best match can be accessed using
Region.getLastMatch()
afterwards.
-
findAll
(PS)¶ Parameters: PS – a Pattern
object or a string (path to an image file or just plain text)Returns: one or more Match
objects as an iterator object or fails if not foundHow to iterate through is documented here.
Repeatedly find ALL instances of a pattern, until no match can be found anymore, that meets the requirements for a single
Region.find()
with the specified pattern.By default, the returned matches are sorted by the similiarty. If you need them ordered by their positions, say the Y coordinates, you have to use Python’s sorted function. Here is a example of sorting the matches from top to bottom.
1def by_y(match): 2 return match.y 3 4icons = findAll(
) 5# sort the icons by their y coordinates and put them into a new variable sorted_icons 6sorted_icons = sorted(icons, key=by_y) 7# another shorter version is using lambda. 8sorted_icons = sorted(icons, key=lambda m:m.y) 9for icon in sorted_icons: 10 pass # do whatever you want to do with the sorted icons
Side Effect lastMatches: a reference to the returned iterator object containing the found matches is stored with the region that was searched. It can be accessed using getLastMatches() afterwards. How to iterate through an iterator of matches is documented here.
-
wait
([PS][, seconds])¶ Parameters: - PS – a
Pattern
object or a string (path to an image file or just plain text) - seconds – a number, which can have a fraction, as maximum waiting
time in seconds. The internal granularity is milliseconds. If not
specified, the auto wait timeout value set by
Region.setAutoWaitTimeout()
is used. Use the constant FOREVER to wait for an infinite time.
Returns: a
Match
object that contains the best match or fails if not foundIf PS is not specified, the script just pauses for the specified amount of time. It is still possible to use
sleep(seconds)
instead, but this is deprecated.If PS is specified, it keeps searching the given pattern in the region until the image appears ( would have been found with
Region.find()
) or the specified amount of time has elapsed. At least one find operation is performed, even if 0 seconds is specified.)Side Effect lastMatch: the best match can be accessed using
Region.getLastMatch()
afterwards.Note: You may adjust the scan rate (how often a search during the wait takes place) by setting
Settings.WaitScanRate
appropriately.- PS – a
-
waitVanish
(PS[, seconds])¶ Wait until the give pattern PS in the region vanishes.
Parameters: - PS – a
Pattern
object or a string (path to an image file or just plain text) - seconds – a number, which can have a fraction, as maximum waiting
time in seconds. The internal granularity is milliseconds. If not
specified, the auto wait timeout value set by
Region.setAutoWaitTimeout()
is used. Use the constant FOREVER to wait for an infinite time.
Returns: True if the pattern vanishes within the specified waiting time, or False if the pattern stays visible after the waiting time has elapsed.
This method keeps searching the given pattern in the region until the image vanishes (can not be found with
Region.find()
any longer) or the specified amount of time has elapsed. At least one find operation is performed, even if 0 seconds is specified.Note: You may adjust the scan rate (how often a search during the wait takes place) by setting
Settings.WaitScanRate
appropriately.- PS – a
-
exists
(PS[, seconds])¶ Check whether the give pattern is visible on the screen.
Parameters: - PS – a
Pattern
object or a string (path to an image file or just plain text) - seconds – a number, which can have a fraction, as maximum waiting
time in seconds. The internal granularity is milliseconds. If not
specified, the auto wait timeout value set by
Region.setAutoWaitTimeout()
is used. Use the constant FOREVER to wait for an infinite time.
Returns: a
Match
object that contains the best match. None is returned, if nothing is found within the specified waiting timeDoes exactly the same as
Region.wait()
, but no exception is raised in case of FindFailed. So it can be used to symplify scripting in case that you only want to know wether something is there or not to decide how to proceed in your workflow. So it is typically used with an if statement. At least one find operation is performed, even if 0 seconds is specified. So specifying 0 seconds saves some time, in case there is no need to wait, since its your intention to get the information “not found” directly.Side Effect lastMatch: the best match can be accessed using
Region.getLastMatch()
afterwards.Note: You may adjust the scan rate (how often a search during the wait takes place) by setting
Settings.WaitScanRate
appropriately.- PS – a
-
Observing Visual Events in a Region¶
You can tell a region to observe that something appears or vanishes,
or something changes in that region. Using the methods
Region.onAppear()
, Region.onVanish()
and Region.onChange()
,
you register an event observer that starts its observation when you
call Region.observe()
. Each region object can have exactly one observation active and
running. For each observation, you can register as many event observers as
needed. So you can think of it as grouping some wait()
and waitVanish()
together and have them processed simultanouesly, while you are waiting for one
of these events to happen.
It is possible to let the script wait for the completion of an observation or
let it run in the background, while your script is continuing. With a timing
parameter you can tell Region.observe()
to stop observation anyway after the given time.
When one of the visual events happens, an event handler written by you is
called. An event handler is a function contained in your script and expects a
SikuliEvent
object as a parameter.
During the processing in your handler, the
observation is paused until your handler has ended. Information between the
main script and your handlers can be given forward and backward using global
variables.
It’s your responsibility to stop an observation. This can either be done by
calling Region.stopObserver()
or by starting the observation with a timing parameter.
Since you can have as many region objects as needed and each region can have one observation active and running. So theoretically it’s possible to have as many visual events being observered at the same time as needed. But in reality, the number of observation is still limited by the system resources available to Sikuli at that time.
Be aware, that every observation is a number of different find operations that
are processed repeatedly. So to speed up processing and keep your script
acting, you may want to define a region for observation as small as possible.
You may adjust the scan rate (how often a search during the observation takes
place) by setting
Settings.ObserveScanRate
appropriately.
PS: means, that either a Pattern or a String (path to an image file or just plain text) can be used as parameter.
handler: as a parameter in the following methods, you have to specify the name of a function, which will be called by the observer, in case the observed event happens. The function itself has to be defined in your script before using the method that references the function. The existance of the function will be checked before starting the script. So to get your script running, you have to have at least the following statements in your script:
def myHandler(event): # you can choose any valid function name
# event: can be any variable name, it references the SikuliEvent object
pass # add your statements here
onAppear("path-to-an-image-file", myHandler) # or any other onEvent()
observe(10) # observes for 10 seconds
Read SikuliEvent
to know what is contained in a SikuliEvent object
Note on performance: Normally all the region methods are used as reg.onAppear(PS)
, where reg
is a
region object. If written as onAppear(PS)
it operates on the default screen being the implicit
region in this case. Using region.onEvent()
instead will restrict the search to the
region’s rectangle and speed up processing, if region is significantly smaller
than the whole screen.
Note: In case of having more than one Monitor active, read Multi Monitor Environments before.
Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the file system bundle-path). Behind the curtain the images itself are specified by using a string containing the file name (path to an image file).
-
class
Region
-
onAppear
(PS, handler)¶ Parameters: - PS – a
Pattern
object or a string (path to an image file or just plain text. - handler – the name of a handler function in the script
With the given region you register an observer, that should wait for the pattern to be there or to appaear and is activated with the next call of
observe()
. In the moment the internal find operation on the given pattern is successful during observation, your handler is called and the observation is paused until you return from your handler.- PS – a
-
onVanish
(PS, handler)¶ Parameters: - PS – a
Pattern
object or a string (path to an image file or just plain text. - handler – the name of a handler function in the script
With the given region you register an observer, that should wait for the pattern to be not there or to vanish and is activated with the next call of
observe()
. In the moment the internal find operation on the given pattern fails during observation, your handler is called and the observation is paused until you return from your handler.- PS – a
-
onChange
([minChangedSize, ]handler)¶ Parameters: - minChangedSize – the minimum size in pixels of a change to trigger a change event.
If omitted: 50 is used (see
Settings.ObserveMinChangedPixels
). - handler – the name of a handler function in the script
New in version X1.0-rc2: minChangedSize
With the given region you register an observer, that should wait for the visual content of the given region to change and is activated with the next call of
observe()
. In the moment the visual content changes during observation, your handler is called and the observation is paused until you return from your handler.Here is a example that highlights all changes in an observed region.
def changed(event): print "something changed in ", event.region for ch in event.changes: ch.highlight() # highlight all changes sleep(1) for ch in event.changes: ch.highlight() # turn off the highlights with selectRegion("select a region to observe") as r: # any change in r larger than 50 pixels would trigger the changed function onChange(50, changed) observe(background=True) wait(30) # another way to observe for 30 seconds r.stopObserver()
- minChangedSize – the minimum size in pixels of a change to trigger a change event.
If omitted: 50 is used (see
-
observe
([seconds][, background = False | True])¶ Begin observation within the region.
Parameters: - seconds – a number, which can have a fraction, as maximum
observation time in seconds. Omit it or use the constant FOREVER to
tell the observation to run for an infinite time (or until stopped
by a call of
stopObserve()
). - background – a flag indicating whether observation is run in the background. when set to True, the observation will be run in the background and processing of your script is continued immediately. Otherwise the script is paused until the completion of the observation.
For each region object, only one observation can be running at a given time.
Note: You may adjust the scan rate (how often a search during the observation takes place) by setting
Settings.ObserveScanRate
appropriately.- seconds – a number, which can have a fraction, as maximum
observation time in seconds. Omit it or use the constant FOREVER to
tell the observation to run for an infinite time (or until stopped
by a call of
-
stopObserver
()¶ Stop observation within the region.
This must be called on a valid region object. The source region of an observed visual event is available as one of the attributes of the event parameter that is passed to the handler function when the function is invoked.
For example, to stop observation within a handler function, simply call
event.region.stopObserver()
inside the handler function.:def myHandler(event): event.region.stopObserver() # stops the observation onAppear("path-to-an-image-file", myHandler) observe(FOREVER) # observes until stopped in handler
-
New in version X1.0-rc2.
-
class
SikuliEvent
¶ When processing an observation in a region, a handler function is called, when one of your registered events
Region.onAppear()
,Region.onVanish()
orRegion.onChange()
happen.The one parameter, you have access to in your handler function is an instance of Class SikuliEvent. You have access to the following attributes of the event, that might help to identify the cause of the event and act on the resulting matches.
- Usage:
event.attribute
- where
event
is the parameter from the definition of your handler function:def myHandler(event):
-
type
¶
The
type
of this event can beSikuliEvent.Type.APPEAR
,SikuliEvent.Type.VANISH
, orSikuliEvent.Type.CHANGE
.-
pattern
¶
The source pattern that triggered this event. This is only valid in APPEAR and VANISH events.
-
region
¶
The source region of this event.
-
match
¶
In an APPEAR event, match saves the top
Match
object that appeared in the observed region.In a VANISH event, match saves the last
Match
object that was in the observed region but vanished.This attribute is not valid in a CHANGE event.
-
changes
¶
This attribute is valid only in a CHANGE event.
This changes attribute is a list of
Match
objects that changed and their sizes are at leastminChangedSize
pixels.- Usage:
Acting on a Region¶
Besides finding visual objects on the screen, acting on these elements is one of the kernel operations of Sikuli. Mouse actions can be simulated as well as pressing keys on a keyboard.
The place on the screen, that should be acted on (in the end just one specific
pixel, the click point), can be given either as a pattern
like with the find
operations or by directly referencing a pixel location
or as center of a region
object (match
or
screen
also) or the target offset location connected with a match
. Since all
these choices can be used with all action methods as needed, they are abbreviated
and called like this:
PSMRL: which means, that either a Pattern
object or a string (path to an image file or just
plain text) or a Match
or a Region
or a Location
can be used as parameter, in
detail:
- P: pattern: a
Pattern
object. An implicit find operation is processed first. If successful, the center of the resulting matches rectangle is the click point. If the pattern object has a target offset specified, this is used as click point instead. - S: string: a path to an image file or just plain text. An implicit find operation with the default minimum similarity 0.7 is processed first. If successful, the center of the resulting match object is the click point.
- M: match: a
match
object from a previous find operation. If the match has a target specified it is used as the click point, otherwise the center of the matches rectangle. - R: region: a
region
object whose center is used as click point. - L: location: a
location
object which by definition represents a point on the screen that is used as click point.
It is possible to simulate pressing the so called key modifiers together with the mouse operation or when simulating keyboard typing. The respective parameter is given by one or more predefined constants. If more than one modifier is necessary, they are combined by using “+” or “|”.
Normally all these region methods are used as reg.click(PS)
, where reg is a
region object. If written as click(PS)
the implicit find is done on the default
screen being the implicit region in this case (see: SCREEN as Default Region).
But using reg.click(PS)
will
restrict the search to the region’s rectangle and speed up processing, if region is
significantly smaller than the whole screen.
Generally all aspects of find operations and of Region.find()
apply.
If the find operation was successful, the match that was acted on, can be
recalled using Region.getLastMatch()
.
As a default, if the visual object (image or text) cannot be found, Sikuli will stop the script by raising an Exception FindFailed (details see: not found).
Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the file system bundle-path). Behind the curtain the images itself are specified by using a string containing the file name (path to an image file).
Note: If you need to implement more sophisticated mouse and keyboard actions look at Low Level Mouse and Keyboard Actions.
Note: In case of having more than one Monitor active, refer to Multi Monitor Environments for more details.
Note on Mac: it might be necessary, to use switchApp()
before, to prepare the
application for accepting the action.
-
class
Region
-
click
(PSMRL[, modifiers])¶ Perform a mouse click on the click point using the left button.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- modifiers – one or more key modifiers
Returns: the number of performed clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found).
Side Effect if PS was used, the match can be accessed using
Region.getLastMatch()
afterwards.Example:
1# Windows XP 2click(
) 3 4# Windows Vista 5click(
) 6 7# Windows 7 8click(
) 9 10# Mac 11click(
)
-
doubleClick
(PSMRL[, modifiers])¶ Perform a mouse double-click on the click point using the left button.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- modifiers – one or more key modifiers
Returns: the number of performed double-clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found).
Side Effect if PS was used, the match can be accessed using
Region.getLastMatch()
afterwards.
-
rightClick
(PSMRL[, modifiers])¶ Perform a mouse click on the click point using the right button.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- modifiers – one or more key modifiers
Returns: the number of performed right cilicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found).
Side Effect if PS was used, the match can be accessed using
Region.getLastMatch()
afterwards.
-
highlight
([seconds])¶ Highlight the region for some period of time.
Parameters: seconds – a decimal number taken as duration in seconds The region is highlighted showing a red colored frame around it. If the parameter seconds is given, the script is suspended for the specified time. If no time is given, the highlighting is started and the script continues. When later on the same highlight call without a parameter is made, the highlighting is stopped (behaves like a toggling switch).
Example:
m = find(some_image) # the red frame will blink for about 7 - 8 seconds for i in range(5): m.highlight(1) wait(0.5) # a second red frame will blink as an overlay to the first one m.highlight() for i in range(5): m.highlight(1) wait(0.5) m.highlight() # the red frame will grow 5 times for i in range(5): m.highlight(1) m = m.nearby(20)
Note: The red frame is just an overlay in front of all other screen content and stays in its place, independently from the behavior of this other content, which means it is not “connected” to the content of the defining region. But it will be adjusted automatically, if you change position and/or dimension of this region in your script, while it is highlighted.
-
hover
(PSMRL)¶ Move the mouse cursor to hover above a click point.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- modifiers – one or more key modifiers
Returns: the number 1 if the mousepointer could be moved to the click point. A 0 (integer null) returned means that because of some reason, no move could be performed (in case of PS may be not Found).
Side Effect if PS was used, the match can be accessed using
Region.getLastMatch()
afterwards.
-
dragDrop
(PSMRL, PSMRL[, modifiers])¶ Perform a drag-and-drop operation from a starting click point to the target click point indicated by the two PSMRLs respectively.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- modifiers – one or more key modifiers
If one of the parameters is PS, the operation might fail due to not Found.
Sideeffect: when using PS, the match of the target can be accessed using
Region.getLastMatch()
afterwards. If only the first parameter is given as PS, this match is returned byRegion.getLastMatch()
.When the operation does not perform as expected (usually caused by timing problems due to delayed reactions of applications), you may adjust the internal timing parameters
Settings.DelayAfterDrag
andSettings.DelayBeforeDrop
eventually combined with the internal timing parameterSettings.MoveMouseDelay
.Another solution might be, to use a combination of
Region.drag()
andRegion.dropAt()
combined with your ownwait()
usages. If the mouse movement from source to target is the problem, you might break up the move path into short steps usingRegion.mouseMove()
.Note: If you need to implement more sophisticated mouse and keyboard actions look at Low Level Mouse and Keyboard Actions.
-
drag
(PSMRL)¶ Start a drag-and-drop operation by dragging at the given click point.
Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. Returns: the number 1 if the operation could be performed. A 0 (integer null) returned means that because of some reason, no move could be performed (in case of PS may be not Found). The mousepointer is moved to the click point and the left mouse button is pressed and held, until another mouse action is performed (e.g. a
Region.dropAt()
afterwards). This is nomally used to start a drag-and-drop operation.Side Effect if PS was used, the match can be accessed using
Region.getLastMatch()
afterwards.
-
dropAt
(PSMRL[, delay])¶ Complete a drag-and-drop operation by dropping a previously dragged item at the given target click point.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- delay – time to wait after in seconds as decimal value
Returns: the number 1 if the operation could be performed. A 0 (integer null) returned means that because of some reason, no move could be performed (in case of PS may be not Found).
The mousepointer is moved to the click point. After waiting for delay seconds the left mouse button is released. This is normally used to finalize a drag-and-drop operation. If it is necessary to visit one ore more click points after dragging and before dropping, you can use
Region.mouseMove()
inbetween.Side Effect if PS was used, the match can be accessed using
Region.getLastMatch()
afterwards.
-
type
([PSMRL, ]text[, modifiers])¶ Type the text at the current focused input field or at a click point specified by PSMRL.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- modifiers – one or more key modifiers
Returns: the number 1 if the operation could be performed, otherwise 0 (integer null), which means, that because of some reason, it was not possible or the click could be performed (in case of PS may be not Found).
This method simulates keyboard typing interpreting the characters of text based on the layout/keymap of the standard US keyboard (QWERTY). Special keys (ENTER, TAB, BACKSPACE, ...) can be incorporated into text by using the constants defined in Class Key using the standard string concatenation (+).
If PSMRL is given, a click on the clickpoint is performed before typing, to gain the focus. (Mac: it might be necessary, to use
switchApp()
to give focus to a target application before, to accept typed/pasted characters.)If PSMRL is omitted, it performs the typing on the current focused visual component (normally an input field or an menu entry that can be selected by typing something).
Side Effect if PS was used, the match can be accessed using
Region.getLastMatch()
afterwards.Note: If you need to type international characters or you are using layouts/keymaps other than US-QWERTY, you should use
Region.paste()
instead. Since type() is rather slow because it simulates each key press, for longer text it is preferrable to useRegion.paste()
.Best Practice: As a general guideline, the best choice is to use
paste()
for readable text andtype()
for action keys like TAB, ENTER, ESC, .... Use onetype()
for each key or key combination and be aware, that in some cases a shortwait()
after atype()
might be necessary to give the target application some time to react and be prepared for the next Sikuli action.
-
paste
([PSMRL, ]text)¶ Paste the text at a click point.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- modifiers – one or more key modifiers
Returns: the number 1 if the operation could be performed, otherwise 0 (integer null), which means, that because of some reason, it was not possible or the click could be performed (in case of PS may be not Found).
Pastes text using the clipboard (OS-level shortcut (Ctrl-V or Cmd-V)). So afterwards your clipboard contains text.
paste()
is a temporary solution for typing international characters or typing on keyboard layouts other than US-QWERTY.If PSMRL is given, a click on the clickpoint is performed before typing, to gain the focus. (Mac: it might be necessary, to use
switchApp()
to give focus to a target application before, to accept typed/pasted characters.)If PSMRL is omitted, it performs the paste on the current focused component (normally an input field).
Side Effect if PS was used, the match can be accessed using
Region.getLastMatch()
afterwards.Note: Special keys (ENTER, TAB, BACKSPACE, ...) cannot be used with
paste()
. If needed, you have to split your complete text into two or morepaste()
and usetype()
for typing the special keys inbetween. Characters like \n (enter/new line) and \t (tab) should work as expected withpaste()
, but be aware of timing problems, when using e.g. intervening \t to jump to the next input field of a form.
-
Extracting Text from a Region¶
-
class
Region
-
text
()¶ Extract the text contained in the region using OCR.
Returns: the text as a string. Multiple lines of text are separated by intervening ‘n’. Note: Since this feature is still in an experimental state, be aware, that in some cases it might not work as expected. If you face any problems look at the Questions & Answers / FAQ’s and the Bugs.
-
Low-level Mouse and Keyboard Actions¶
-
class
Region
-
mouseDown
(button)¶ Press the mouse button down.
Parameters: button – one or a combination of the button constants Button.LEFT
,Button.MIDDLE
,Button.RIGHT
.Returns: the number 1 if the operation is performed successfully, and zero if otherwise. The mouse button or buttons specified by button are pressed until another mouse action is performed.
-
mouseUp
([button])¶ Release the mouse button previously pressed.
Parameters: button – one or a combination of the button constants Button.LEFT
,Button.MIDDLE
,Button.RIGHT
.Returns: the number 1 if the operation is performed successfully, and zero if otherwise. The button specified by button is released. If button is omitted, all currently pressed buttons are released.
-
mouseMove
(PSRML)¶ Move the mouse pointer to a location indicated by PSRML.
Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. Returns: the number 1 if the operation could be performed. If using PS (which invokes an implicity find operation), find fails and you have switched off FindFailed exception, a 0 (integer null) is returned. Otherwise, the script is stopped with a FindFailed exception. Sideeffects: when using PS, the match can be accessed using
Region.getLastMatch()
afterwards
-
wheel
(PSRML, WHEEL_DOWN | WHEEL_UP, steps)¶ Move the mouse pointer to a location indicated by PSRML and turn the mouse wheel in the specified direction by the specified number of steps.
Parameters: - PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
- WHEEL_DOWN|WHEEL_UP – one of the two constants denoting the wheeling direction.
- steps – an integer indicating the amoung of wheeling.
Sideeffects: when using PS, the match can be accessed using
Region.getLastMatch()
afterwards
-
keyDown
(key | list-of-keys)¶ Press and hold the specified key(s) until released by a later call to
Region.keyUp()
.Parameters: key|list-of-keys – one or more keys (use the consts of class Key). A list of keys is a concatenation of several key constants using “+”. Returns: the number 1 if the operation could be performed and 0 if otherwise.
-
keyUp
([key | list-of-keys])¶ Release given keys. If no key is given, all currently pressed keys are released.
Parameters: key|list-of-keys – one or more keys (use the consts of class Key). A list of keys is a concatenation of several key constants using “+”. Returns: the number 1 if the operation could be performed and 0 if otherwise.
-
Exception FindFailed¶
As a default, find operations (explicit
and implicit) when not successful
raise an Exception FindFailed
, that will
stop the script immediately. This is of great help, when developing a script, to
step by step adjust timing and similarity. When the script runs perfectly, then an
exception FindFailed signals, that something is not as it should be.
To implement some checkpoints, where you want to asure your workflow, use
Region.exists()
, that reports a not found situation without raising
FindFailed (returns False instead).
To run all or only parts of your script without FindFailed exceptions to be raised,
use Region.setThrowException()
or Region.setFindFailedResponse()
to switch it on and off as needed.
For more sophisticated concepts, you can implement your own exception handling using
the standard Python construct try: ... except: ...
.
New in version X1.0-rc2.
- These are the possibilities to handle “not found” situations:
- generally abort a script, if not handled with
try: ... except: ...
(the default setting or using
setThrowException(True)
orsetFindFailedResponse(ABORT)
)
- generally abort a script, if not handled with
- generally ignore all “not found” situations
(using
setThrowException(False)
orsetFindFailedResponse(SKIP)
),
- want to be prompted in such a case
(using
setFindFailedResponse(PROMPT)
)
- advise Sikuli to wait forever (be careful with that!)
(using
setFindFailedResponse(RETRY)
)
Comment on using PROMPT:
This feature is helpful in following situations:
- you are developing something, that needs an application with it’s windows to be in place, but this workflow you want to script later. If it comes to that point, you get the prompt, arrange the app and click Retry. Your workflow should continue.
- you have a workflow, where the user might do some corrective actions, if you get a FindFailed
- guess you find more ;-)
In case of a FindFailed, you get the following prompt:

Clicking Retry would again try to find the image. Skip would continue the script and Abort would end the script.
Examples: 4 solutions for a case, where you want to decide how to proceed in a workflow based on the fact that a specific image can be found. (pass is the python statement, that does nothing, but maintains indentation to form the blocks):
# --- nice and easy
if exists("path-to-image"): # no exception, returns None when not found
pass # it is there
else:
pass # we miss it
# --- using exception handling
# every not found in the try block will switch to the except block
try:
find("path-to-image")
pass # it is there
except FindFailed:
pass # we miss it
# --- using setFindFailedResponse
setFindFailedResponse(SKIP) # no exception raised, not found returns None
if find("path-to-image"):
setFindFailedResponse(ABORT) # reset to default
pass # it is there
else:
setFindFailedResponse(ABORT) # reset to default
pass # we miss it
# --- using setThrowException
setThrowException(False) # no exception raised, not found returns None
if find("path-to-image"):
setThrowException(True) # reset to default
pass # it is there
else:
setThrowException(True) # reset to default
pass # we miss it
-
class
Region
Reminder If used without specifying a region, the default/primary screen (default region SCREEN) is used.
New in version X1.0-rc2.
-
setFindFailedResponse
(ABORT | SKIP | PROMPT | RETRY)¶ For the specified region set the option, how Sikuli should handle “not found” situations. The option stays in effect until changed by another
setFindFailedResponse()
.Parameters: - ABORT – all subsequent find failed operations (explicit or implicit) will raise exception FindFailed (which is the default when a script is started).
- SKIP – all subsequent find operations will not raise
exception FindFailed. Instead, explicit find operations such as
Region.find()
will return None. Implicit find operations (action functions) such asRegion.click()
will do nothing and return 0. - PROMPT – all subsequent find operations will not raise exception FindFailed. Instead you will be prompted for the way to handle the situation (see using PROMPT). Only the current find operation is affected by your response to the prompt.
- RETRY – all subsequent find operations will not raise
exception FindFailed. Instead, Sikuli will try to find the target
until it gets visible. This is equivalent to using
wait( ... , FOREVER)
instead offind()
or using setAutoWaitTimeout(FOREVER).
-
getFindFailedResponse
()¶ Get the current setting in this region.
Returns: ABORT or SKIP or PROMPT or RETRY Usage:
val = getFindFailedResponse() if val == ABORT: print "not found will stop script with Exception FindFailed" elif val == SKIP: print "not found will be ignored" elif val == PROMPT: print "when not found you will be prompted" elif val == RETRY: print "we will always wait forever" else: print val, ": this is a bug :-("
Note: It is recommended to use
set/getFindFailedResponse()
instead ofset/getThrowException()
since the latter ones might be deprecated in the future.-
setThrowException
(False | True)¶ By using this method you control, how Sikuli should handle not found situations in this region.
Parameters: - True – all subsequent find operations (explicit or implicit) will raise exception FindFailed (which is the default when a script is started) in case of not found.
- False – all subsequent find operations will not raise
exception FindFailed. Instead, explicit find operations such as
Region.find()
will return None. Implicit find operations (action functions) such asRegion.click()
will do nothing and return 0.
-
getThrowException
()¶ Returns: True
orFalse
Get the current setting as True or False (after start of script, this is True by default) in this region.
-
Grouping Method Calls ( with Region: )¶
Instead of:
# reg is a region object
if not reg.exists(image1):
reg.click(image2)
reg.wait(image3, 10)
reg.doubleClick(image4)
you can group methods applied to the same region using Python’s with
syntax:
# reg is a region object
with reg:
if not exists(image1):
click(image2)
wait(image3, 10)
doubleClick(image4)
All methods inside the with
block that have the region omitted are redirected to the
region object specified at the with
statement.
Note for Sikuli X versions up to rc3
This should work for Regions and Region-like objects (Region, Screen, App.window()).
Match objects have to be made a Region explicitly using Region(some_match)
.
This is a general workaround when facing any problems using with some_region:
.
Special Methods for Developers¶
Here we document Region methods, that might be helpful, when developing complex scripts with its own API, as it is the fact, when developing an Extension.
Be aware, that these might change without notice (should not be, but might be ;-).
These two methods might help to implement more generic functions than just find() and click().
-
class
Region
New in version X1.0-rc2.
-
getRegionFromPSRM
(PSRM)¶ Returns a new Region object derived from the given parameter. In case of PS, internally a find() is done inside this region. If found, the match is returned. In case RM, just a copy of the given region is returned.
Params PSRM: a Pattern, String, Region or Match object Returns: a new Region object
New in version X1.0-rc2.
-
getLocationFromPSRML
(PSRML)¶ Returns a new Location object derived from the given parameter. In case of PS, internally a find() is done inside this region. If found, the match’s target offset position is returned. In case RM, just a copy of the given region’s respective location (center or target offset) is returned.
Params PSRML: a Pattern, String, Region, Match or Location object Returns: a new Location object
-