=================================================================================

Novel approachs to GRP can leverages graphical user interface (GUI) automation tools and techniques. Such modern GUI automation tools use computer vision and machine learning algorithms to process visual elements in GUIs in near real time and to emulate user interactions in reaction to those events. They can use GUI automation to couple third-party programming tools with proprietary (graphical) robot programming environments by enabling program generation in those target environments while bypassing vendor-provided application programming interfaces (APIs).

The user can connect a computer running a GUI automation tool, such as SikuliX [1] or PyAutoGUI [2], to a target robot programming environment (e.g., an HMI hosted on a teach pendant) to generate a program in the application environment. This saves time and allows for maintaining a unified code base, from which programs for proprietary target environments can easily be generated.

Keyboard and mouse can be controlled by GUI (graphical user interface) automation (automation of keyboard typing and mouse clicking).

[1] SikuliX. Available online: http://sikulix.com/ (accessed on 25 November 2020).
[2] PyAutoGUI. Available online: https://pyautogui.readthedocs.io/ (accessed on 25 November 2020).