Hello everyone,
My name is Thierry Gourdin and I'm a RPA addict 🙂
I'm product adoption Manager in SAP Intelligent RPA and I would like to share some best practices how to handle limitations with screen locking, which can be a very confusing topic.
Let's dive into the case.
It may happen that after having finished an automation implementation and testing on his desktop, a user notices that the automation doesn't execute properly when the factory sends the job to a machine in particular configuration:
If one of these 2 conditions triggers an error with your automation whereas it was working before (when running in attended mode for example), then this article may help...
This kind of issue mostly comes from "low-level" mouse or keyboard simulations:
Such issue can also occur when using UIAutomation connector even with standard action such as click, set, get. Here is why: The UIAutomation connector helps automate any kind of desktop application that implements the UIAutomation interface. But if an application does not respond correctly to the UIAutomation connector request, the connector sends mouse clicks or keystrokes as backup plan to execute the action anyway. But it's not working with a locked screen or disconnected VDI.
To identify if an action sent through UIAutomation connector triggers a low level "backup plan" action, you can try this:
Surface automation relies on mouse clicks and keystrokes that require a desktop with an unlocked screen.
If you find the following keystroke action:
you can replace it with the following:
If you find the following click action:
you can replace it with the following:
This behavior is related to:
The effect will be the one below:
This may be an issue if the developer needs to retrieve those messages.
So this previous solution using low level action (ctx.keystroke) is valid but only if you launch these automations along non screenlocked desktop or VDI. Here is the correct way to handle a robust automation of this kind of rich web application:
For example, it is possible to simulate a specific event that is required on the page to proceed, here it is a mouse event:
• Button
• Edit
It is important to track down those limitations early in your POC, design, or implementation. In order to do so, you can:
Here we are. If you get stuck in lock screen or VDI disconnexion issue that is often linked to the use of unattended mode execution, I hope this guide will help you. All best practices presented above are linked to a specific context you need to prior assess in order to decide which solution to try.
Thanks for your time and have a happy automation journey!
My name is Thierry Gourdin and I'm a RPA addict 🙂
I'm product adoption Manager in SAP Intelligent RPA and I would like to share some best practices how to handle limitations with screen locking, which can be a very confusing topic.
Presentation
SAP Intelligent RPA solution sometimes requires the user to handle applications with low-level actions such as mouse clicks or direct keystrokes simulations. But in certain circumstances, these low-level actions can cause limitations.
Even it's mostly encountered during unattended execution, this behavior isn't limited to this execution mode, as the user can be expected to launch attended scenarios on his machine and let it work while the screen is locked.
Let's dive into the case.
It may happen that after having finished an automation implementation and testing on his desktop, a user notices that the automation doesn't execute properly when the factory sends the job to a machine in particular configuration:
The job executes on a user’s machine with its screen locked.
The job executes on a VDI with no active connexion.
If one of these 2 conditions triggers an error with your automation whereas it was working before (when running in attended mode for example), then this article may help...
Reason
This kind of issue mostly comes from "low-level" mouse or keyboard simulations:
- application.page.item.click(true);
- application.page.item.clickMouse();
- ctx.mouse.<any>
- application.page.item.keystroke("text");
- ctx.keyStroke("not working");
These functions literally ask the agent to simulate a mouse click or press a key. It won't work as well when the session is locked or the VDI is disconnected.
Such issue can also occur when using UIAutomation connector even with standard action such as click, set, get. Here is why: The UIAutomation connector helps automate any kind of desktop application that implements the UIAutomation interface. But if an application does not respond correctly to the UIAutomation connector request, the connector sends mouse clicks or keystrokes as backup plan to execute the action anyway. But it's not working with a locked screen or disconnected VDI.
To identify if an action sent through UIAutomation connector triggers a low level "backup plan" action, you can try this:
Check if the cursor moves to the item position when calling the click action.
- Combine the action with a delay during which you can manually lock the screen or disconnect the session:
ctx.wait(function(ev){
application.page.item.click();
},10000);Surface automation relies on mouse clicks and keystrokes that require a desktop with an unlocked screen.
Best practices
Try to avoid Mouse Click and Keystroke Actions
To resolve issues with mouse click and keystroke actions, you need to track down and identify those actions. These actions may have been implemented in the project that way, to avoid complicated code or overcome improper item handling by the tool.
If you find the following keystroke action:
ctx.keystroke / application.page.item.keystroke
you can replace it with the following:
application.page.item.set()
If you find the following click action:
ctx.mouse.click / application.page.item.clickMouse / click(true)
you can replace it with the following:
application.page.item.click()
Web Application Automation
Setting a value with the standard set method may not work on every website, especially on a website with rich framework (e.g. Bootstrap, Angular …).
This behavior is related to:
- Javascript events attached to items on the web page
- Surface controls
- A complex page structure
Let's describe this on a concrete example. Below website relies on a rich JS framework that makes the following actions whenever an user sets a value in the "Component Search" search bar (highlighted in red):
- does a surface control of the input value
- displays a message under 'Messages' label when the input is accepted.
IRPAComponent.pMain.oSearchBox.set("IRPA");The effect will be the one below:
The input 'IRPA' has been set but no message has been displayed under the 'Messages' label. This indicates that:
- the web page has not been correctly acknowledged that an entry has been entered
- the surface control action has not been executed.
This may be an issue if the developer needs to retrieve those messages.
The developer may use the following workaround:
IRPAComponent.pMain.activate();
IRPAComponent.pMain.oSearchBox.setFocus();
ctx.keyStroke("IRPA");As a result, the input value is set and a message is displayed. 
So this previous solution using low level action (ctx.keystroke) is valid but only if you launch these automations along non screenlocked desktop or VDI. Here is the correct way to handle a robust automation of this kind of rich web application:
IRPAComponent.pMain.oSearchBox.set("IRPA");
IRPAComponent.pMain.oSearchBox.trigger("input");
As a result, the tooltip and the corresponding message are displayed and this solution can be applied to desktops with a locked screen.
A more complex web page may require the user to work directly in the browser by analyzing the internal events with web developer mode.
For example, it is possible to simulate a specific event that is required on the page to proceed, here it is a mouse event:
application.page.item.scriptItem( "dispatchEvent( new MouseEvent( 'mouseover', {'bubbles': true, 'cancelable': true}))");Win32 Automations
For some Win32 applications, if standard methods oblige you to use low-level click or key stroke actions, you can try by capturing the application with UIAutomation and enable Win32 automations option (only available with version 1.0.8+ of the Desktop Studio).
This option is only available on items with the following class names:
• Button
application.page.item.click()• Edit
application.page.item.set()In the Captured Data section, each of these items must also have the capture property hWnd, different from 0x0.
Keep Screen Unlocked
A common best practice is to configure your environment to prevent the screen from locking when it's possible.
This will allow mouse clicks and keystrokes to be used without getting stuck and sent to the screen lock instead of the targeted application.
Keep a Connection to the Virtual Desktop Infrastructure
Same idea here, if you are using a Virtual Desktop Infrastructure and mouse clicks and keystrokes are necessary, you can:
Configure the Virtual Desktop Infrastructure to prevent the screen from locking.
Directly connect to a Virtual Desktop Infrastructure from a local machine.
Eventually you may have a project that launches this connection from a local machine.
Identify Another Functional Resolution
If all the resolutions previously suggested do not work, try to find a functional workaround. Another sequence of screen, another button to click on, if any other way to validate your form may also work without needing a mouse click or a keystroke.
If none of the workarounds apply, it may become a blocker for your project.
It is important to track down those limitations early in your POC, design, or implementation. In order to do so, you can:
Use action + delay + manual screen lock
Perform regular tests on a disconnected Virtual Desktop Infrastructure or on a machine with a locked screen.
Your turn!
Here we are. If you get stuck in lock screen or VDI disconnexion issue that is often linked to the use of unattended mode execution, I hope this guide will help you. All best practices presented above are linked to a specific context you need to prior assess in order to decide which solution to try.
Thanks for your time and have a happy automation journey!
- SAP Managed Tags:
12 Comments
