<html><head></head><body>Automation of device testing (covering both hardware and software elements of said devices) is a trend that has been steadily growing.&nbsp;Touchscreen device testing is the leading use case, with a large number existing solutions available.&nbsp;However, replicating human interaction on devices with physical buttons and switches is more complex, and fewer solutions are available.&nbsp;Developing a tool that evaluates button and switch devices from both hardware (through physical input) and software perspectives is the next logical step in the evolution of device testing automation.<h2 id="isPasted">The use case</h2>The case of MATT, a physical input device testing&nbsp;robot, performing test scenarios on a gaming console originated not from the need of testing the console, its controller, or a video game, but through one client’s need of interacting with those elements to test the network security and additional related parameters. The use case can be summed up to executing simple commands in a video game to generate movement in order to send network packets via WIFI.&nbsp;To develop the use case and to easily integrate with the device, multiple addons needed to be implemented for MATT to accurately interact with not only the controller, but the whole console suite.&nbsp;&nbsp;During the solution’s creation process two technical challenges were met, namely interacting with the controller’s side buttons (in addition to the top ones), and connecting with the monitor displaying the console’s output, as well as creating a system to monitor and analyze the collected video stream and determine if what is shown on the screen is the correct response to the executed action.&nbsp;<h2 id="isPasted">Planning&nbsp;&nbsp;</h2>From the beginning of the use case’s development process the goal was set to make MATT be able to use the gaming console as a human would and simulate an online player in a video game. This meant that the testing robot needed to navigate the console’s menu, adjust the desired settings, choose a character and enter a game. Additionally, to prevent the servers from determining MATT was not an active player (as it was not taking part in the game through actually playing the story line, the missions, following specific instructions) a series of game specific actions were executed.&nbsp;Having the goal set to making MATT replicate as closely as possible as an actual human QA manual tester and replicate how they behave during testing processes with the same requirement, meant that the robot had to behave with the dexterity, observation, and visual abilities of a person. For this reason, the plan was to create a system that would give the device testing robot the ability to recognize console-specific elements and interact with them with the precision of a real player, starting from menu navigation, game selection, online game launch to acting like an active player on a platform specific game (ex: Mario Cart, Call of Duty, Elder Scrolls, Pokémon).&nbsp;<h2 id="isPasted">Challenges and Solutions&nbsp;</h2><h3>Building the controller and console set-up</h3>MATT’s platform&nbsp;was&nbsp;built&nbsp;to physically support&nbsp;mobile devices:&nbsp;phones, tablets,&nbsp;or smartwatches- mounted using the same system. Dealing with device features different from everything tested until then (shape, size, utility, way of interaction), special&nbsp;cradles and mechanisms were developed for the controllers to be set into place and operated on.&nbsp;Even more so, the mechanical side of the interaction with the consoles needed to accommodate&nbsp;multiple&nbsp;console&nbsp;models&nbsp;(Play Station, Nintendo, Xbox). Dealing with diversity in shape and button placement, for each console type, 3D printed cradles were&nbsp;created&nbsp;for&nbsp;their specific controllers&nbsp;to be placed&nbsp;on MATT’s&nbsp;platform.&nbsp;For each one of them, the position of their side buttons&nbsp;(L and R buttons)&nbsp;was&nbsp;analyzed,&nbsp;and actuators designed and set into place for optimal interaction.&nbsp;Real life operability of the consoles was assured by actuators acting upon&nbsp;lateral&nbsp;buttons and MATT’s fingers pressing on the controller’s top&nbsp;buttons&nbsp;(joystick, directional pad, menu button,&nbsp;menu&nbsp;and option selection buttons).&nbsp;All button pressing&nbsp;commands&nbsp;on controllers were&nbsp;realized&nbsp;through and controlled by MATT’s software.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjYxMjU2NzM2NzcyLWluIGFydGljbGUtIGNvbnNvbGUgdGVzdGluZyB3ZXZvbHZlci5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 757px;" class="fr-fic fr-dib">The console controller mounted on MATT's platform using a specially designed cradle, together with two actuators interacting with the L and R lateral buttons.Using the specially designed mechanisms and cradles the controllers and actuators were placed on MATT’s platform and a TV monitor set up for the console’s output. To accurately interact with the controllers, their button placement was mapped out using the calibration process included in MATT’s API. Considering that gaming console’s actions can also be composed by complex button combinations including interaction between both top and lateral buttons (for example L and left arrow button to ‘throw’ in GTA5 on a PS4), high level functions were implemented, automatically translating to movements in the respective actuator and MATT’s fingers. This method opened the possibility of writing testcases replicating human interaction with the console. &nbsp;&nbsp;&nbsp;<h3>Capturing and analyzing the console's output&nbsp;</h3>For&nbsp;touchscreen testing use cases,&nbsp;MATT’s&nbsp;computer vision&nbsp;API&nbsp;and&nbsp;a&nbsp;variety of other algorithms&nbsp;were used&nbsp;to take&nbsp;pictures of the device’s screen&nbsp;and verify if&nbsp;the icon choice or response to gestures were executed correctly&nbsp;(with the response given to the performed action shown&nbsp;on the device’s screen).&nbsp;The&nbsp;intricate situation for&nbsp;the&nbsp;gaming&nbsp;console case was the&nbsp;response&nbsp;output. It&nbsp;did not appear on the device mounted on&nbsp;MATT’s&nbsp;platform&nbsp;(as in the touchscreen device’s instance)&nbsp;but was displayed on a separate TV screen. Hence, the challenge&nbsp;was&nbsp;approached by&nbsp;firstly&nbsp;finding a method to&nbsp;integrate with&nbsp;a&nbsp;TV monitor&nbsp;and&nbsp;secondly, capturing the video feed directly sent to the TV and performing the computer vision analysis&nbsp;on the console video stream.&nbsp;The first step was to establish the navigation through the console’s menu. Taking into consideration that gaming consoles output HDMI signals that are transmitted directly to a monitor, integrating an HDMI capture card to the computers used to perform the test case turned out to be the ideal solution. This method permitted real time image recognition to be performed on the captured video stream. Additionally, using computer vision selected menu or in-game elements were easily recognized followed by commands sent to MATT to further navigate through the consoles’ or game’s interface. For example, MATT pressing the center of the directional pad to confirm a setting choice, or an actuator pressing R2 side button and accessing the right menu tab.&nbsp;&nbsp;The aim of the developed mechanism was to create a close to perfection synergy between what was actually seen on the TV screen and computer vision, which allowed for accurate commands to be passed on to MATT for execution, as well as effectively constructing and monitoring feedback loops. Whenever a button was not pressed, wrongly pressed, or a button pressing was not registered, the mistake was easily noticed, and the command automatically repeated. As a result, the menu navigation and the process of joining a real game was achieved effortlessly, as well as performing preliminary actions (charge ammo, choose character attributes, move left-right, jump, crouch) to simulate a player starting the game and prevent the servers from expelling ‘MATT player’ from a human player’s playthrough. In the end the system was able to accomplish what was aimed at from the beginning: behaving and reacting as a human tester would.&nbsp;<h2 id="isPasted">Conclusion&nbsp;&nbsp;</h2>The client’s final use case was to generate enough WIFI traffic by creating movements inside of a video game, with the purpose of capturing and analyzing WIFI network packets that are sent between the gaming console and the access point (the WIFI access point). At its core, the use case was based mostly on security aspects and WIFI security protocols, but its design made it possible for multi situational application (covering testing protocols for game designers, joystick manufacturers, software developers) that requires human input for navigating through a menu or console controls.&nbsp;&nbsp;In the end, using MATT’s computer vision API and the specifically developed mechanisms for console and controller interaction, actions and in-console&nbsp;or&nbsp;in-game element recognition are performed&nbsp;with precision of a human, simulating real life player interaction.&nbsp;&nbsp;</body></html>

MATT interacting with both top and lateral buttons of the controller 

Developing a tool that evaluates button and switch devices from both hardware (through physical input) and software perspectives is the next logical step in the evolution of device testing automation.

The evolution of hardware and software console testing automation

robotics

This article taps into discussing problems met by retailers and ways in which they can be solved with robotic automation.  

Robotic Automation for Brick-and-Mortar Retailers

Human & retail robots together solving key issues in the retail industry

In a highly competitive environment, retailers are leveraging cutting-edge technologies to boost efficiency and run their businesses more effectively. AI-powered robots are transforming the retail sector by optimizing pricing strategies, inventory management, and more.

How AI-powered robots are transforming the retail industry

Measured in milliseconds, the response time to a tap on a smartphone’s screen might not be a “show stopping” issue, but a delay between an action and its reaction in the mobile gaming industry can make an important difference.

Measuring touch screen response time- A gaming use case

Automotive infotainment system testing with MATT, the device testing robot

MATT overtakes challenges met during the process of testing in-vehicle-infotainment systems using its characteristic adaptability and capabilities to support an effective operation throughout the testing process.

More by Adapta Robotics

Robotic Automation for Brick-and-Mortar Retailers

How AI-powered robots are transforming the retail industry

Measuring touch screen response time- A gaming use case