How adding Python control to MATT Framework improved the functionality of Our Robots

MATT Framework is the core software application that was designed to allow users to calibrate, prototype, develop testing scenarios, and manually operate the MATT robot, making it as easy to interact with as possible. It functions like an application server with a dedicated user interface for seamless robot control and operation.

MATT Robot product line-up

MATT Framework, as a software product, has expanded over time to offer multiple APIs that accommodate various programming languages, reducing user-side integration efforts.

In an earlier version of the Framework, which was already experiencing effortless adoption among MATT Robot users, the Python API was introduced. Looking towards further improving the functionality of the software, the next objective was adding a Python Read-Eval-Print Loop (REPL). Up until recently, carrying out such a task proved to be technologically challenging, but the difficulty diluted with the advancement of available and used methods and knowledge.

Python runs in the browser

To see the set intention being implemented into MATT Framework, multiple methods have been explored in order to find the most appropriate and more efficient one for the current situation. In the end, adding Python to the software was carried out through Pyodide.

Pyodide is a Python distribution designed specifically for the browser environment, offering seamless integration of JavaScript and Python through a robust foreign function interface (FFI). It leverages the power of WebAssembly, enabling the execution of Python code with full support for the standard library and the ability to load additional packages from PyPI, the Python Package Index.

PyScript is a popular project using Pyodide — pyscript.net

When it comes to browser-based Python implementations, Pyodide stands out as a comprehensive and feature-rich solution. With the arrival of WebAssembly, Pyodide takes advantage of this technology to deliver a highly capable Python environment while ensuring compatibility across different browsers. Unlike other alternatives such as Transcrypt, which require a pre-compilation step to transpile the Python code into JavaScript, Pyodide offers a more direct and streamlined approach.

Moreover, Pyodide excels in its ability to directly load pure Python packages from PyPI. This expands its functionality and allows developers to leverage a wide range of existing Python libraries, enhancing the capabilities and versatility of their browser-based applications. In comparison, alternatives like Brython may not provide the same level of support or seamless integration with PyPI.

Enhancing MATT Console

To make the utilization of MATT robots during testing as smooth as possible, the integrated MATT Framework software was designed to contain a console section that allows the robot operator to easily interact with their test cases. Aiming to make automation as accessible as possible, the necessary code for performing test operations is generated by the console. Considering that MATT is a testing robot built to interact with devices having touchscreens and buttons, the most common operations consist of taps, swipes, identifying icons, and saving pictures.

MATT’s console was initially built to support JSON API. As the number of teams utilizing the device testing robot grew, the necessity of updating the console by integrating Python API was met, as it was getting increasingly difficult for Python users to operate MATT.

Laptop control of testing robot — Adapta Robotics on Unsplash

The initial intention of the update (named ServicePack 2 software update) was only to incorporate Python code generation support in order to facilitate robot operation. Soon, the objective evolved into making the best of both worlds by implementing bidirectional code translation that supports both the Python and JSON APIs. This meant that anyone who is using the JSON API can now take advantage of this feature to convert the test cases to Python code and vice-versa.

Translating API calls

To achieve the above mentioned, during the technical development of the update it has been made use of the already implementation of the Python API as a wrap around the existent JSON command protocol and the API’s significant surface overlap.

For example, a Python step that tells the system to tap a specific location on a device looks like this:

While its corresponding JSON command is:

By patching some of the methods in the Python API, specifically those for sending commands over a socket connection, the generation the JSON commands were made possible simply by calling the Python method inside the browser using Pyodide. The one-way conversion was not just possible, but easier to implement than initially expected.

Documentation portal for the MATT, the device testing robot – docs.mattrobot.ai

JSON to Python 

Next, the focus was on completing the development of the reverse transform, enabling the conversion of JSON commands into Python calls. To accomplish this, a converter class, specifically designed to accept JSON commands as input and generate corresponding string representations of the Python calls, was created.

Limitations

In mathematics, a transform is deemed reversible when the sequence A => B => A1 results in identical values for both A and A1. In the present case, however, the code translation process is only partially reversible, because when handling iterative loops, they become flattened into individual operations.

Limitations when translating API calls

Refactoring to React

An important step that was achieved during the implementation of the developed MATT Framework software update was rewriting the console feature using React. This came as a natural choice since the component was separable from the main project while also being very heavy in the business logic department.

By utilizing PrismJS and its capabilities, enhancing the component with syntax highlighting was not only easy but also necessary (contributing to the overall scope of delivering the robots’ operator an accessible, easy-to-use interaction). The end of the process presented itself with numerous files inside the console editor project. For this reason, it has been compiled into a separate module using WebPack with the result of the compilation being imported into the main project.

Conclusion

Comprising all added updates and new functionalities, the picture featured below shows the improved user interface emphasizing the append mode, syntax highlighting, and instant API to API translation.

Ultimately, the ability to execute Python code in the browser through Pyodide has proved to be a game-changer for the utilized web-based user interface. This development decision not only reduced the workload significantly, but it allowed many more features to be shipped through the release updates that ultimately closely follow the MATT Robot product line roadmap.

The above-described walkthrough serves as a great example of how library choices can impact the product’s roadmap and why thorough technology research is crucial.

The improved console of the MATT Framework software