CZ2023384A3 - Method of generating instructions for a stationary robot and system for implementing this method - Google Patents

Abstract

The subject of the invention is a method of generating instructions for a stationary robot (5) comprising the steps of: - performing a work operation for surface treatment of a part (7) using a hand-held work tool (6) with a mounted sensor unit (1), - monitoring the position and orientation of the hand-held work tool (6) by the sensor unit (1) during the step of performing the work operation, - monitoring the working state of the hand-held work tool (6) by the working state sensor (10) during the step of performing the work operation, - providing calibration data to the data processing module (3), wherein the calibration data provided to the data processing module (3) includes information about the mutual spatial relationship between the reference coordinate systems of the sensor unit (1), the hand-held work tool (6), the stationary robot (5) and the part (7), and - processing data by the data processing module (3), wherein the data processing step includes the step of generating instructions for the stationary robot (5) in the programming language of at least one manufacturer of stationary robots (5). The subject of the invention is also a system for performing this method.

Description

Method of generating instructions for a stationary robot and system for implementing this method

Technical area

The present invention relates to a method and system for generating instructions for a stationary robot based on a demonstration of a work operation for surface treatment of a part. More specifically, the invention relates to generating instructions, e.g., for a painting robot.

State of the art

Currently, there are solutions for automating a work operation, e.g. industrial painting, using industrial robots. For example, various demonstrators are used for automation, with which the user demonstrates the movement performed for the relevant operation and, based on recording this demonstrated movement, the robot is programmed to copy this movement during the subsequent automated operation. However, this demonstrator is typically not a real work tool and thus does not allow capturing all the essential characteristics of the work operation. Available solutions are also often unable to generate robotic instructions for various industrial robots, or for industrial robots from different manufacturers, because different manufacturers use different programming languages for programming robots. Such solutions are then applicable only for one given robot, but not for robots from other manufacturers, which is disadvantageous.

It would therefore be desirable to come up with a solution that would allow the actual performance of a work operation to be recorded in such a way that all essential characteristics of this work operation are captured, and that would allow the generation of a robotic program executable by a specific industrial robot. The generated program should include not only movement instructions, i.e. information about the trajectory along which the robotic work tool should move, but also other instructions for controlling the robotic work tool, e.g. when the robotic work tool should perform a work operation and when not, i.e. e.g. when the robot paint gun should be triggered. The required solution should also allow the performed work operation to be recorded sufficiently accurately and reliably so that the automated work operation qualitatively corresponds to the manually performed work operation. At the same time, the recording of the performed work operation should not affect the course of this work operation or in any way limit the user who performs this work operation.

The essence of the invention

The above shortcomings are to some extent eliminated by the method of generating instructions for a stationary robot based on a demonstration of a work operation for surface treatment of a part, the essence of which is that it includes the steps:

- performing a work operation for surface treatment of a part using a hand-held working tool with a mounted sensor unit,

- monitoring the position and orientation of the hand-held work tool by a sensor unit during the step of performing a work operation for surface treatment of the part, wherein the step of monitoring the position and orientation of the hand-held work tool includes a step of recording acceleration and angular velocity over time by an inertial sensor and a step of recording visual images of the surrounding environment at different points in time by at least one camera,

- monitoring the working state of the hand-held working tool by a working state sensor during the step of performing a working operation for surface treatment of the part, wherein the working state sensor provides information about whether the hand-held working tool is started,

- 1 CZ 2023 - 384 A3

- providing calibration data to the data processing module, the calibration data provided to the data processing module including information about the mutual spatial relationship between the reference coordinate systems of the sensor unit, the hand-held work tool, the stationary robot and the part, and

- processing data from the inertial sensor, at least one camera and the working condition sensor as well as calibration data by a data processing module, wherein the data processing step includes a step of generating instructions for the stationary robot in a programming language of at least one stationary robot manufacturer.

The above method allows generating an executable robot program (instructions) for a stationary robot of various manufacturers in such a way that this program contains all the essential characteristics of the work operation for the surface treatment of the part. The work operation is monitored with high accuracy and reliability, while recording the work operation using a sensor unit attached to the hand-held working tool does not affect the course of the work operation performed in this way or limit the person who manipulates the hand-held working tool in any way. The work operation for the surface treatment of the part can be, for example, industrial painting (wet or powder), but also grinding, polishing, chamfering, blasting, etc. The work operation for the surface treatment of the part can also be quality control performed in connection with the surface treatment, i.e., for example, quality control of the painted part.

The term stationary robot is well-established in this field of technology and refers to a robot that cannot move from place to place by its own means. A stationary robot can be primarily an industrial robot, e.g. a painting robot, or a collaborative robot. Alternatively, a stationary robot can be attached to an external mechanism adding an additional degree of freedom, e.g. a rail conveyor. The generated instructions can then also contain instructions for controlling this additional mechanism. The generated instructions can also be used not only for a real stationary robot, but also for a virtual stationary robot, i.e. for its so-called digital twin. This can be used for simulation purposes.

A hand-held work tool is an actual work tool that enables the performance of the required work operation for the surface treatment of a part. For example, if the work operation for the surface treatment of a part is industrial painting, this hand-held work tool is a hand-held paint spray gun.

The monitoring of the position and orientation (i.e. rotation) of the hand-held working tool by the sensor unit is realized by the combined use of an inertial sensor and at least one camera, and to increase the accuracy, multiple cameras can also be used. The camera together with the inertial sensor can be collectively referred to as a tracking camera, and it is worth mentioning that its output is data with six degrees of freedom, specifically the position of the tracking camera in time in three spatial axes and the orientation of the tracking camera in time in three spatial axes. I.e. position and orientation data in its reference coordinate system. Given that the sensor unit is attached to the hand-held working tool and given that this recorded data is then transformed using calibration data, it can be said that the position and orientation of the hand-held working tool is monitored by the sensor unit, as mentioned above. To calculate position and orientation from data recorded by sensors (from acceleration in three spatial axes in time and angular velocity in three spatial axes in time), the tracking camera may include its own computing unit.

The computing unit of the tracking camera advantageously includes a detection module for detecting visual elements in the camera images. These visual elements are, for example, various visually significant points of the surrounding environment, e.g. corners or edges visible in the images. The detection of visual elements is advantageous primarily because it reduces the number of points that need to be taken into account by an order of magnitude, typically from thousands of pixels (depending on the camera resolution) to a few hundred visual elements. Each visual element is labeled, spatially mapped and arranged in a vector of visual elements.

- 2 CZ 2023 - 384 A3 elements. After taking a new image, the detection module compares the newly marked elements arranged in the newly constructed vector of visual elements and calculates the change in position based on this comparison.

Alternatively, the position and orientation from the recorded data can be calculated elsewhere than in the tracking camera, e.g. by a microcomputer in the control unit or by a data processing module that may be part of a remote server.

Alternatively, the output of the tracking camera may be only acceleration over time, angular velocity over time, and visual images, while the actual calculation of position and orientation may be performed subsequently, e.g. using a microcomputer in the control unit or using a data processing module.

In addition to a three-axis accelerometer for recording acceleration and a three-axis gyroscope for recording angular velocity, the sensor unit may advantageously also include a magnetometer, thanks to which it is possible to more accurately calculate position and orientation.

The step of monitoring the position and orientation of the hand-held work tool is performed concurrently with the step of monitoring the working state of the hand-held work tool, which is performed using a working state sensor. The working state sensor provides information about whether the hand-held work tool is started. The working state sensor can thus, for example, provide a signal, the analysis of which can be used to obtain the desired working state of the hand-held work tool. This analysis can be a comparison of the values of the measured signal with predetermined threshold values. The analysis of this signal to obtain the working state can be performed, for example, by a data processing module. Alternatively, this analysis can be performed, for example, by a microcomputer in the control unit or by the working state sensor's own computing unit, or by another computing unit. In addition to determining the working state, other information can also be determined from the working state sensor signal, for example, about what specific activity of the given work operation was performed at a certain time, etc.

After the completion of the work operation step, the recorded data from the inertial sensor, camera and working condition sensor are sent to the data processing module. Alternatively, the recorded data can also be sent to the data processing module continuously. Calibration data that needs to be provided to this module is also sent to the data processing module.

The calibration data provided preferably includes at least information about the mutual spatial relationship between the reference coordinate system G of the hand-held work tool and the reference coordinate system C of the sensor unit, information about the mutual spatial relationship between the reference coordinate system C of the sensor unit and the reference coordinate system P of the part, and information about the mutual spatial relationship between the reference coordinate system R of the stationary robot and the reference coordinate system P of the part. The information about the mutual spatial relationship (in other words, transformation relationships) between the respective reference coordinate systems serves for the transformation between these coordinate systems and can be preferably written in the form of transformation matrices. The calibration data provided therefore preferably includes at least the transformation matrices Tgc, Tcp and Trp. Alternatively, however, this information can also be written in another mathematical form.

To obtain calibration data, which is provided to the data processing module, the method preferably includes the step of performing a calibration measurement. Preferably, the calibration data is measured, but in general it can be determined in another way and provided to the data processing module. The calibration measurement can be performed at various stages of the method, with at least part of the calibration being performed at the beginning, during, or at the end of the position and orientation tracking step, when the recording of position and orientation data is still active and the data recording has not been restarted.

For the calibration measurement step, unique visual markers are advantageously used, with information about the spatial relationship between the reference coordinate systems G and C,

- 3 CZ 2023 - 384 A3 information about the mutual spatial relationship between the reference coordinate systems C and P, information about the mutual spatial relationship between the reference coordinate systems R and P, and in addition also information about the mutual spatial relationship between the reference coordinate systems C and R, are obtained based on the detection of visual marks by at least one camera and on the basis of determining the relative orientation of the sensory unit to the visual marks. The uniqueness of visual marks lies in the fact that each visual mark contains its identifier to distinguish it from other visual marks. Therefore, when the camera detects a visual mark, it can identify which visual mark it is and combine this information with information about its location. This visual mark can be, for example, a QR code.

Preferably, the calibration measurement step includes four calibration parts that, in addition to the visual marks, also use calibration fixtures. Specifically, at least one first calibration fixture with at least one calibration hole and a set of unique first visual marks and also a second calibration fixture including a unique second visual mark is provided for the calibration measurement, wherein the relative position of the second calibration fixture and the part is fixed. The second calibration fixture may be located on, for example, a frame on which the part is suspended. The relative positions of the first visual marks relative to each calibration hole are known.

At least one calibration aperture is shaped to receive a hand-held tool in only one particular position and orientation, and at least one calibration aperture is shaped to receive a robotic tool in only one particular position and orientation. In other words, the calibration aperture together with the hand-held tool and the robotic tool, respectively, function on a lock and key basis. The first calibration fixture may include multiple calibration apertures, e.g., two calibration apertures, one of which is shaped to receive a hand-held tool and the other is shaped to receive a robotic tool. Alternatively, however, the first calibration element may include only one calibration aperture, and this is adapted to receive both a hand-held and a robotic tool sequentially. Alternatively, two first calibration fixtures may be provided, one of which is adapted to receive a hand-held tool and the other to receive a robotic tool.

The step of performing a calibration measurement using visual markers includes the following four parts, with the first part including the steps:

- inserting a hand-held working tool with a mounted sensor unit into the calibration hole of the first calibration fixture,

- recording information about position and orientation using a sensory unit,

- detecting individual first visual marks using a camera,

- determining the relative position and orientation of the sensory unit with respect to the individual first visual markers.

- obtaining information about the spatial relationship between the reference coordinate system G of the hand-held working tool and the reference coordinate system C of the sensory unit.

The second part of the calibration includes the steps:

- guiding the robotic working tool with the attached sensor unit into the calibration hole of the first calibration fixture,

- recording information about position and orientation using a sensory unit,

- detecting individual first visual marks using a camera,

- 4 CZ 2023 - 384 A3

- determining the relative position and orientation of the sensory unit with respect to individual first visual markers and

- obtaining information about the spatial relationship between the reference coordinate system C of the sensor unit and the reference coordinate system R of the stationary robot.

The third part of the calibration includes the steps:

- approaching the hand-held working tool with the mounted sensor unit to the second calibration fixture with the second visual mark,

- recording information about position and orientation using a sensory unit,

- detecting a second visual mark using a camera,

- determining the relative position and orientation of the sensory unit with respect to the second visual mark and

- obtaining information about the spatial relationship between the reference coordinate system C of the sensory unit and the reference coordinate system P of the part using information about the spatial relationship between the reference coordinate system G of the hand-held working tool and the reference coordinate system C of the sensory unit.

The fourth part of the calibration includes the steps:

- approaching the robotic working tool with the mounted sensor unit to the second calibration fixture with the second visual mark,

- recording information about position and orientation using a sensory unit,

- detecting a second visual mark using a camera,

- determining the relative position and orientation of the sensory unit with respect to the second visual mark and

- obtaining information about the spatial relationship between the reference coordinate system R of the stationary robot and the reference coordinate system P of the part using information about the spatial relationship between the reference coordinate system C of the sensor unit and the reference coordinate system R of the stationary robot.

The first and third parts of the calibration (i.e. calibration using a manual working tool) are performed before, during, or at the end of the position and orientation tracking step, when the recording of position and orientation data is still active and the data recording has not been restarted. The second and fourth parts of the calibration (i.e. calibration using a robotic working tool) can be performed at any time. The order of the individual parts of the calibration does not have to be fixed and, for example, the last steps of the sub-parts of the calibration (obtaining information about the spatial relationship, i.e. specifically, e.g. the construction of the relevant transformation matrices) can be performed later. Similarly, the steps of determining the relative position and orientation of the sensory unit with respect to the visual marks can also be performed (calculated) subsequently and the physical manipulation required for the given part of the calibration ends with the detection of the relevant visual mark and the recording of the position and rotation information using the sensory unit.

An additional sensor unit may be used for the second and fourth parts of the calibration. Alternatively, one sensor unit may be used for the entire calibration and may be transferred between the manual work tool and the robotic work tool to perform individual parts of the calibration. To record position and orientation information using the sensor unit, the user may signal the control unit via the user interface of the sensor unit, alternatively via the user interface of the control unit.

- 5 CZ 2023 - 384 A3

Alternatively, the calibration measurement can be performed on a purely mechanical principle, but the calibration process using visual marks is easier, faster, and thanks to the contactless (optical) detection of visual marks, also more accurate, as there is no risk of the part shifting from its rest position when pushed by a manual working tool or a robotic working tool into the calibration fixture, which is attached, for example, to the frame on which the part is suspended. At the same time, the risk of incorrect insertion of the working tool into the calibration hole caused by human error is eliminated.

This alternative calibration method is presented below, and this mechanical calibration involves only three parts, as it is not necessary to perform a part analogous to the second part of the optical calibration described above.

The first part of the mechanical calibration:

The calibration of the sensor unit with respect to the hand tool is carried out using a calibration fixture in the shape of a cube with defined holes located on the individual walls of the cube, which, by the principle of a mechanical lock and key, allow the user to insert the tip of the hand tool into them in exactly one defined way, i.e. with a specific position and orientation. The mutual positions of these holes are defined and known. During this part of the calibration, the user gradually inserts the tip of the hand tool with the attached sensor unit into the individual holes in a defined order and at the moment when the hand tool is correctly positioned, the user gives, e.g. using the user interface of the sensor unit, a signal to the control unit to store information about the position and orientation provided by the sensor unit. From the known definition of the geometry of this calibration fixture and the information about the position and orientation from the sensor unit stored during this part of the calibration, information about the spatial relationship between the reference coordinate system G of the hand tool and the reference coordinate system C of the sensor unit is subsequently obtained. Specifically, the appropriate transformation matrix can be constructed.

The second part of mechanical calibration:

Next, the user calibrates the reference coordinate system C of the sensor unit with the reference coordinate system P of the part. This part of the calibration is performed using a calibration fixture that contains a hole that, using the principle of a mechanical lock and key, allows the user to insert the tip of the hand-held working tool into it in exactly one defined way, i.e. with a specific position and orientation. The mutual position of this calibration fixture and the part is fixed, but does not have to be known. However, it must remain unchanged during the recording of the work operation and during the third part of the mechanical calibration (stationary robot - part). The user calibrates by inserting the hand-held working tool into the hole of the calibration element and giving a signal to the control unit to store the data on the position and orientation that the sensor unit provides at that moment. From this data, together with the data obtained from the calibration of the sensor unit with respect to the hand-held working tool (the first part of the mechanical calibration), information is subsequently obtained about the spatial relationship between the reference coordinate system G of the hand-held working tool and the reference coordinate system P of the part. Specifically, the appropriate transformation matrix can be constructed. This part of the calibration is performed at the beginning of each demonstration of the work operation for which the user wants to generate a robot program.

The third part of mechanical calibration:

Calibration of the reference coordinate system R of the stationary robot with respect to the reference coordinate system P of the part is performed using a calibration fixture that contains a hole that, by the principle of a mechanical lock and key, allows the tip of the robotic working tool to be inserted into it in exactly one defined way, i.e. with a specific position and orientation. The mutual position of this calibration fixture and the part is fixed, but does not have to be known. However, it must remain unchanged during the recording of the work operation and during calibration.

- 6 CZ 2023 - 384 A3 of the stationary robot relative to the part. The user performs the calibration by using the stationary robot controller to guide the robotic working tool into the opening of this calibration element and using the robot controller to note the position and orientation of the robotic working tool in the reference coordinate system R of the stationary robot. A transformation matrix defining the spatial relationship between the reference coordinate system R of the stationary robot and the reference coordinate system P of the part is then constructed from this data, i.e. information is obtained about the mutual spatial relationship of the reference coordinate systems R and P. This part of the calibration needs to be performed only once, provided that the part for which the robotic instructions are generated will subsequently always be located in the same place relative to the stationary robot, which will execute the generated robotic program on it.

The step of generating instructions for a stationary robot is preferably performed by an instruction generator containing information on the syntax, semantics, required structure and format of instructions for a stationary robot of at least one stationary robot manufacturer, wherein the instructions are written by the instruction generator in the required structure and format based on the information on the syntax and semantics and contain keywords and argument values that are the result of the data processing step by the data processing module. These argument values are the resulting position and orientation coordinates in the reference coordinate system R of the stationary robot, which are the result of the data processing steps that precede the step of generating instructions. Alternatively, the resulting position and orientation coordinates can be expressed in another user coordinate system, which however has a fixed and known transformation to the reference coordinate system R of the stationary robot. In such a case, the user must provide this transformation as an additional input to the data processing module, and therefore to the instruction generator. Alternatively, these argument values can be the resulting position and orientation coordinates, which can be modified. For example, so that the generated instructions lead to the execution of an automated operation, for example, faster or slower than was performed by a manual working tool when tracking position and orientation.

The data processing step by the data processing module further prior to the step of generating instructions for the stationary robot preferably includes a data filtering step, a data time synchronization step, a dynamic subsampling step and a step of applying a spatial transformation based on the provided calibration data. However, the data processing step may alternatively include all of these steps - the steps of time synchronization and applying a spatial transformation based on the provided calibration data are nevertheless essential for proper operation. Specifically, applying a spatial transformation consists in multiplying the calibration matrices to obtain a resulting transformation relationship. This resulting transformation relationship allows the measured data to be transformed into the reference coordinate system of the stationary robot so that instructions can subsequently be generated which, according to the convention of the programming language of the stationary robot, are expected to be expressed in the coordinate system of the robot.

Time synchronization is used to synchronize the measured data, i.e. data from the inertial sensor, camera and working condition sensor. This ensures that a certain image from the camera corresponds to the same moment at which the signal was recorded by the working condition sensor and the inertial sensor.

Dynamic subsampling allows you to generate a robot program with a sufficiently low number of robot instructions to be easy for the user to read, understand, and edit, while capturing all the significant elements of the recorded work operation so that it can be replicated with sufficient quality by a stationary robot.

The step of monitoring the working state is preferably performed by an audio sensor recording an audio signal during the step of performing a work operation for surface treatment of the part. The audio signal can be analyzed, for example, by a working state computing unit, but can also be analyzed, for example, by a microcomputer in the control unit or by a data processing module. In addition to the information itself about the working state, it is also possible to obtain, for example, from the analysis of the audio signal

- 7 CZ 2023 - 384 A3 information about which sub-activity of a given work operation was performed at a given time, if these sub-activities differ in the sound they produce.

The above-mentioned shortcomings are also to some extent eliminated by the system for performing the method according to the present invention, the essence of which consists in that it includes a sensor unit and a control unit communicatively connected to the sensor unit and adapted to control the sensor unit, the sensor unit comprising:

- fastening element for attachment to a hand-held working tool,

- inertial sensor for recording acceleration and angular velocity over time,

- at least one camera for recording visual images of the surrounding environment at different points in time, and

- a working state sensor for monitoring the working state of the hand-held work tool, the working state sensor providing information about whether the hand-held work tool is started, the system further comprising a data processing module adapted to receive data from the inertial sensor, at least one camera and the working state sensor as well as calibration data containing information about the mutual spatial relationship between the reference coordinate systems of the sensor unit, the hand-held work tool, the stationary robot and the part, the data processing module comprising an instruction generator comprising an interface for generating instructions for the stationary robot in a programming language of at least one stationary robot manufacturer.

A communication connection is here understood as a connection that enables data transmission, whether it is realized electrically by means of a cable or wirelessly, e.g. by means of Wi-Fi technology. The control unit and the sensor unit may each contain their own housing, but alternatively they may also be housed in one housing and may thus be part of one additional unit, which is attached to the hand-held working tool by means of a fastening element. As already mentioned above, multiple cameras can be used to achieve better accuracy.

The system of the present invention further preferably comprises at least one first calibration fixture with at least one calibration aperture and a set of unique first visual marks, wherein the relative positions of the first visual marks with respect to each calibration aperture are known, and wherein at least one calibration aperture is shaped to receive a manual work tool in only one particular position and orientation, and at least one calibration aperture is shaped to receive a robotic work tool in only one particular position and orientation, and the system further comprises a second calibration fixture comprising a unique second visual mark, wherein the relative position of the second calibration fixture and the part is fixed. These calibration fixtures and visual marks enable advantageous calibration using visual marks that is easier, more accurate, and faster than purely mechanical calibration.

The working condition sensor is preferably an acoustic sensor, which allows reliable and sufficiently sensitive detection of the working condition of the hand-held working tool. Alternatively, less preferably, the working condition sensor may also be another sensor operating on a different principle, e.g. a Hall sensor, a flow meter, which e.g. in the painting process can measure the flow of paint through a supply hose, or an ammeter, which can measure the current flowing in the case of powder coating, etc.

The control unit preferably includes a microcomputer for controlling the sensor unit and a memory for storing data from the sensor unit and a communication module for communication with the data processing module. This basic arrangement allows for efficient and reliable control of data collection and provision to the data processing module.

- 8 CZ 2023 - 384 A3

The system further preferably comprises a remote server communicatively coupled to the control unit, wherein the data processing module is preferably part of the remote server. Alternatively, the data processing module may not be part of the remote server and may, for example, be part of a microcomputer of the control unit, which would mean that this microcomputer would perform the relevant data processing steps.

The system further preferably includes a user interface of the data processing module for configuring the instruction generator and also an access device for accessing the user interface of the data processing module, wherein the user interface of the data processing module is communicatively connected to the data processing module. This user interface of the data processing module allows the user to configure the details of how the instructions for the stationary robot are to be generated, and also allows the generated instructions to be subsequently downloaded. Using this interface, for example, it is possible to select data to be processed, configure data processing parameters (e.g. dynamic subsampling, etc.) or select the type (manufacturer) of the stationary robot. This user interface of the data processing module may preferably be a web interface that is part of a remote server. The access device may be, for example, a computer, a mobile phone or other device with access to the web. Alternatively, the user interface of the data processing module may not be designed as a web interface, but may be any other suitable interface, e.g. a device with appropriate controls.

Clarification of drawings

The essence of the invention is further explained by examples of its implementation, which are described using the attached drawings, where:

Fig. 1 schematically shows the system according to the present invention in connection with a stationary robot and with emphasis on the arrangement of the sensor unit and the control unit, Fig. 2 schematically shows the system according to the present invention in connection with a stationary robot, with an indication of the calibration and with emphasis on the arrangement of the remote server and Fig. 3 schematically shows a diagram of the method according to the first exemplary embodiment of the present invention.

Examples of implementation of the invention

The invention will be further explained by way of example with reference to the accompanying drawings. First, an exemplary embodiment of a system by which the method of generating instructions according to the present invention can be implemented will be described in detail, and then the individual steps of this method will be described.

The system for carrying out the method according to the present invention in the first exemplary embodiment comprises a sensor unit 1, a control unit 2 communicatively connected to the sensor unit 1 and adapted to control the sensor unit 1, and also a remote server 21, which includes a data processing module 3. The remote server 21 containing the data processing module 3 is communicatively connected to the control unit 2, this communication connection being provided by a communication module 15 in the control unit 2, as will be described in more detail below when describing the arrangement of the control unit 2 in the first exemplary embodiment of the system.

As schematically shown in Fig. 1, the sensor unit 1 in the first exemplary embodiment includes an inertial sensor 8, a camera 9, a working state sensor 10, a fastening element 11 for fastening the sensor unit 1 to a hand-held working tool 6 and an attachment 12. For example, the sensor unit 1 is formed by a housing for storing individual components, while the attachment 12 is connected to this housing and includes a fastening element 11 for fastening to a hand-held working tool.

- 9 CZ 2023 - 384 A3 tool 6. For example, the fastening element 11 is shaped for attachment to the hand-held working tool 6, i.e. in this embodiment it is possible to fasten the sensor unit 1 to the hand-held working tool 6 even without using screws.

The sensor unit 1, after being attached to the hand-held working tool 6, enables monitoring of the behavior of the hand-held working tool 6, specifically monitoring of the position and orientation (i.e. rotation) of the hand-held working tool 6 and also monitoring of the working state of the hand-held working tool 6.

The tracking of the position and orientation of the hand-held working tool 6 is realized by the combined use of the inertial sensor 8 and the camera 9, in other words, using the data recorded by the inertial sensor 8 and the visual images from the camera 9. In the described first exemplary embodiment of the system, the inertial sensor 8 is an inertial measurement unit (IMU), which includes a three-axis accelerometer and a three-axis gyroscope. The three-axis accelerometer records acceleration values over time in three spatial axes, and the three-axis gyroscope records angular velocity values over time in three spatial axes. The camera 9 is a stereo camera in the described first exemplary embodiment, and this camera 9 records visual images of the surrounding environment at different points in time, i.e. it provides images with a time stamp.

The camera 9 together with the inertial sensor 8 can be collectively referred to as a so-called tracking (or monitoring) camera, and in the first exemplary embodiment the tracking camera includes its own computing unit (not shown in the figures) for calculating the position and orientation of its reference coordinate system (i.e. the reference coordinate system C of the sensor unit 1) in time. This basically tracks the position and orientation of the hand-held working tool 6 (to which the sensor unit 1 is attached), although a transformation using calibration data must still be applied to convert the position and orientation of the tracking camera to the position and orientation of the hand-held working tool 6, as will be discussed below.

The position and orientation of the tracking camera is calculated from the data of the inertial sensor 8 and the camera 9, i.e. from the recorded values of acceleration over time in three spatial axes, angular velocity over time in three spatial axes and based on visual elements detected in the visual images from the camera 9. To detect visual elements, the computing unit of the tracking camera includes a detection module, which means that an algorithm detecting visual elements is applied to the images from the camera 9. These visual elements are, for example, various visually significant points of the surrounding environment, e.g. corners or edges visible in the images. The detection of visual elements is advantageous primarily because the number of points that need to be taken into account is reduced by an order of magnitude, typically from thousands of pixels (depending on the resolution of the camera 9) to a few hundred visual elements. Each visual element is marked, spatially mapped and arranged in a vector of visual elements. After taking a new image, the detection module compares the newly marked features arranged in the newly constructed vector of visual features and calculates the position change based on this comparison.

The above-described procedure is known as V-SLAM (Visual Simultaneous Localization and Mapping) technology, and the output of the computing unit of the tracking camera performing V-SLAM is data with six degrees of freedom, namely the position of the tracking camera in time in three spatial axes and the orientation of the tracking camera in time in three spatial axes. In the first embodiment described, the data from the motion sensor 8 and the camera 9 are fed to the data processing module 3 already in this combined and recalculated form, i.e. as position and orientation in time in three spatial axes. In other words, V-SLAM is used to calculate the position and orientation of the tracking camera in time, and indirectly, therefore, to track the position and orientation of the hand-held working tool 6 in the environment that is mapped using the camera 9. Commercially available tracking cameras can be used for these purposes.

As mentioned above, in addition to the inertial sensor 8, the sensor unit 1 also includes a working state sensor 10, which in the first exemplary embodiment described is a sound sensor (or microphone). The sound sensor serves to record an audio signal during a step

- 10 CZ 2023 - 384 A3 performing a work operation for surface treatment of the part 7, wherein this step of performing a work operation for surface treatment of the part 7 is performed using a hand-held work tool 6 with an attached sensor unit 1, as will also be described below when describing the method according to the present invention.

In the first exemplary embodiment, as indicated in Fig. 2, the hand-held working tool 6 is a hand-held paint gun and the working operation for the surface treatment of the part 7 is industrial painting. This means that the working state sensor 10 provides information about whether the hand-held paint gun is triggered, in other words, about whether the hand-held paint gun is currently painting. This information is essential for generating robotic instructions, since it is important for the stationary robot 5 to reproduce not only the movement (position and orientation) of the hand-held working tool 6 during the subsequent automated working operation, but also when the hand-held working tool 6 was triggered, and therefore when the robotic working tool 25 should be triggered.

Specifically, the working state of the hand-held working tool 6 (started/not started) is determined by analyzing this sound signal, which can be performed, for example, by the own computing unit of this working state sensor 10 (not shown in the figures). For example, the computing unit of the working state sensor 10 is adapted to compare the values of the sound signal at different points in time with predetermined threshold values and is also adapted to subsequently determine the working state of the hand-held working tool 6. For example, if the sound signal is higher than a first threshold value, the working state is indicated as started, and if the sound signal is lower than a second threshold value, the working state is indicated as not started. Information about the working state of the hand-held working tool 6 is supplied to the data processing module 3, as will be described further below. In addition to determining the working state, the computing unit of the working state sensor 10 can also calculate other information from the sound signal, e.g. about which sub-activity of a given work operation was performed at a given time, if these sub-activities differ in the sound they produce.

In the first exemplary embodiment, the sound sensor is attached to the inner side of the housing of the sensor unit 1, specifically to that inner wall of the housing that is closest to the hand-held working tool 6 to which the sensor unit 1 is attached. The sensor unit 1 further includes a user interface 19 of the sensor unit 1, e.g. control buttons and signaling diodes, which serve to control the sensor unit 1 and/or the control unit 2 and to interact with the control unit 2 during data recording by the sensor unit 1. The function of the user interface 19 of the sensor unit 1 will be discussed in more detail below in the description of the method.

The control unit 2 in the first exemplary embodiment of the system, as schematically shown in Fig. 1, includes a microcomputer 13, a memory 14, a communication module 15 for communication connection with a remote server 21, a battery 16 for powering the microcomputer 13, a charging connector 17 for charging the battery 16 and also a connector for electrical connection with the sensor unit 1 via a cable. The individual components are stored in the housing of the control unit 2, which is indicated by a dashed line in Fig. 1. The housing of the control unit 2 is further provided with at least one strap 18, which allows the user to wear the control unit 2 on the body, for example over the shoulder or on the back, and move freely with it. The control unit 2 also further includes a user interface 20 of the control unit, e.g. control buttons and signaling diodes, which serve to control the control unit 2 and/or the sensor unit 1 and to interact with the sensor unit 1 during data recording by the sensor unit 1. The function of the user interface 20 of the control unit 1 will be discussed in more detail below when describing the method.

The microcomputer 13 in the control unit 2 is powered by a battery 16 and further powers and controls other components of the control unit 2 (user interface 20 of the control unit 2, memory 14, communication module 15), but also components of the sensor unit 1. The microcomputer 13 therefore specifically also controls the inertial sensor 8, the camera 9 and the working state sensor 10 and controls the collection (recording, recording) of data and implements their storage in the memory 14 with which it works. The microcomputer 13 also controls the communication module 15, while in the described first exemplary embodiment the communication module 15 is

- 11 CZ 2023 - 384 A3

The Wi-Fi module and the communication connection of the control unit 2 and the remote server 21 is implemented as a wireless connection using Wi-Fi technology. This is indicated in Fig. 1 or Fig. 2 by a dashed arrow and also by a graphic signal symbol, since the communication connection of the control unit 2 and the remote server 21 serves to transfer data recorded by the sensor unit 1 from the control unit 2 to the remote server 21, the so-called cloud. Thanks to the mutual connection of the control unit 2 and the sensor unit 1, the remote server 21 is therefore indirectly connected to the sensor unit 1 via the control unit 2. In other words, it can also be said that the data recorded by the sensor unit 1 is uploaded to the remote server 21 via the control unit 2.

The remote server 21, as schematically shown in Fig. 2, includes in the first exemplary embodiment a storage 22 of uploaded data, a data processing module 3 and a storage 24 of generated instructions. The recorded data is thus transferred to the recorded data storage 22 and is further processed by the data processing module 3, the output of which is instructions for the stationary robot 5. More specifically, the instructions for the stationary robot 5 (or so-called robotic instructions) are an executable program for automated control of the stationary robot 5. Such programs are subsequently stored in the generated instructions storage 24, where they are ready for loading into the stationary robot 5. If the program is loaded into the controller of the stationary robot 5 and run on the stationary robot 5, it implements automated control of the stationary robot 5 in such a way that the robotic working tool 25 on the stationary robot 5 reproduces the required work operation for surface treatment of the part 7, as it was performed by the manual working tool 6 with the attached sensor unit 1.

As also schematically shown in Fig. 2, the data processing module 3 comprises an instruction generator 23 comprising an interface for generating instructions in the programming language of at least one manufacturer of stationary robots 5. In addition to the generation of instructions, however, the data processing module 3 is also adapted for further data processing, which is performed using appropriate algorithms before generating instructions. Specifically, the data processing module 3 in the first exemplary embodiment comprises a data filtering module, a data time synchronization module, a dynamic subsampling module and a spatial transformation application module. The significance of these individual further processing steps is discussed in the description of the method below. The data processing module 3 comprises sufficient computing capacity for generating instructions and for performing this further data processing.

The input of the data processing module 3, and thus also of the instruction generator 23, is data from the inertial sensor 8 and the camera 9, which in this particular embodiment are already combined data, i.e. the position and orientation in time in three spatial axes. Another input is also data from the working state sensor 10 including information about whether the hand-held working tool 6 is started. Yet another input is calibration data, which includes transformation matrices defining the mutual spatial relationship between the reference coordinate systems of the sensor unit 1, the hand-held working tool 6, the stationary robot 5 and the part 7, as will be described in more detail below. Based on these input data, the instruction generator 23 generates instructions for the stationary robot 5, which will also be described in more detail below in the description of the method.

In the first exemplary embodiment, the system includes a user interface of the data processing module and also an access device 4 for accessing this user interface of the data processing module 3. In the first exemplary embodiment, this user interface of the data processing module 3 is a web interface, which is part of the remote server 21 and which allows the user to select data to be processed, configure data processing parameters, e.g. the type (manufacturer) of the stationary robot 5 for which instructions are to be generated, and subsequently download the generated instructions from the remote server 21, specifically from the storage of generated instructions. The access device 4 is therefore communicatively connected to the remote server 21, specifically at least with the generator 23 of instructions and with the storage 24 of generated instructions. The access device 4 is, e.g., a computer, a mobile phone or other device with access to the web.

- 12 CZ 2023 - 384 A3

The following section will describe in more detail the method of generating instructions, which in a first exemplary embodiment includes the steps:

- performing calibration measurements to obtain calibration data,

- performing a work operation for surface treatment of the part 7 using a hand-held work tool 6 with a mounted sensor unit 1,

- monitoring the position and orientation of the hand-held working tool 6 by the sensor unit 1 during the step of performing the work operation for surface treatment of the part 7, wherein the step of monitoring the position and orientation of the hand-held working tool 6 includes the step of recording the acceleration and angular velocity over time by the inertial sensor 8 and the step of recording visual images of the surrounding environment at different points in time by the camera 9,

- monitoring the working state of the hand-held working tool 6 by the working state sensor 10 during the step of performing the working operation for the surface treatment of the part 7, wherein the working state sensor 10 provides information about whether the hand-held working tool 6 is started,

- providing calibration data to the data processing module 3, wherein the calibration data provided to the data processing module 3 includes transformation matrices defining the mutual spatial relationship between the reference coordinate systems of the sensor unit 1, the manual work tool 6, the stationary robot 5 and the part 7, and

- processing data from the inertial sensor 8, the camera 9 and the working condition sensor 10 as well as calibration data by the data processing module 3, wherein the data processing step includes the step of generating instructions for the stationary robot 5 in the programming language of at least one manufacturer of stationary robots 5.

First, the calibration measurement will be described, which in the first exemplary embodiment serves to obtain calibration data, specifically to obtain four transformation matrices defining the mutual spatial relationship between the reference coordinate systems of the sensor unit 1, the manual working tool 6, the stationary robot 5 and the part 7.

First, the transformation matrix Tgc is obtained, defining the mutual spatial relationship between the reference coordinate system G of the hand-held work tool 6 (denoted by the letter G after the English "gun" for a pistol) and the reference coordinate system C of the sensor unit 1 (denoted by the letter C after the English "camera" for a camera that is part of the sensor unit 1).

This first part of the calibration is implemented using a first calibration fixture 26, which includes a shape-defined calibration hole. This calibration hole is adapted for inserting the tip of the hand tool 6 in just one defined way, i.e. with a specific position and a specific orientation of the hand tool 6. In other words, the calibration hole together with the hand tool 6 function on the principle of lock and key. The first calibration fixture 26 is schematically shown in Fig. 2, wherein the insertion of the hand tool 6 is indicated by an arrow leading to this calibration hole. The first calibration fixture 26 further includes a set of first visual marks, which are visible by a camera 9, which is part of the sensor unit 1 mounted on the hand tool 6, after inserting the hand tool 6 into the calibration hole. The mutual positions of these first visual marks and the calibration hole are defined and known. The visual marks are unique and each contains its own identifier for recognition from other visual marks.

After inserting the tip of the hand tool 6 with the attached sensor unit 1 into the calibration hole and at the moment when the hand tool 6 is correctly positioned, the user gives a signal using the user interface 19 of the sensor unit 1, i.e. for example by pressing a button

- 13 CZ 2023 - 384 A3 to the control unit 2 to store the information about the position and orientation, which is provided by the sensor unit 1. The sensor unit 1, specifically the camera 9, simultaneously detects the first visual marks and determines its relative position and orientation to these individual first visual marks. From the known definition of the geometry of the first calibration fixture 26 and the information about the position and orientation from the sensor unit 1 stored during this part of the calibration, a transformation matrix TGc is subsequently constructed defining the mutual spatial relationship between the reference coordinate system G of the hand-held work tool 6 and the reference coordinate system C of the sensor unit 1. This first part of the calibration is performed each time the user attaches the sensor unit 1 to the hand-held work tool 6 in order to demonstrate a work operation for subsequent automation.

Furthermore, in an analogous manner, the transformation matrix Tcr is obtained defining the mutual spatial relationship between the reference coordinate system C of the sensor unit 1 and the reference coordinate system R of the stationary robot 5 (denoted by the letter R after the English "robot"), specifically the robotic working tool 25.

This second part of the calibration is also carried out using the first calibration fixture 26, however, instead of the manual working tool 6, a robotic working tool 25 is inserted into the calibration hole (guided by the controller of the stationary robot 5), to which the sensor unit 1 is attached for the purposes of this part of the calibration. After the user gives a signal to the control unit 2, the camera 9 detects the first visual marks and measures its relative position and orientation with respect to these first visual marks. From the known definition of the geometry of the first calibration fixture 26 and the information about the position and orientation from the sensor unit 1 stored during this part of the calibration, a transformation matrix TCR is subsequently constructed defining the mutual spatial relationship between the reference coordinate system C of the sensor unit 1 and the reference coordinate system R of the stationary robot 5, specifically the robotic working tool 25.

Furthermore, a transformation matrix Tcp is obtained defining the mutual spatial relationship between the reference coordinate system C of the sensor unit 1 and the reference coordinate system P of part 7 (denoted by the letter P according to the English "product" or "part" for part 7).

This third part of the calibration is carried out by means of a second calibration fixture 27, which includes a second visual mark which is located at a fixed mutual distance from the part 7. This fixed mutual distance does not have to be known, but must remain unchanged during the calibration and during the step of performing the work operation for the surface treatment of the part 7 using the hand-held work tool 6, i.e. during the demonstration. The second calibration fixture 27 is schematically shown in Fig. 2 with the part 7 which is suspended from a support frame for the work operation, e.g. for painting.

The user performs this part of the calibration by bringing the hand tool 6 closer to the second calibration fixture 27 and aiming the camera 9 in the sensor unit 1 at the second visual mark. The sensor unit 1 recognizes the second visual mark, since the second visual mark is, like all visual marks used in the calibration, unique, and measures its relative position and orientation to it. Furthermore, the user gives the user interface 19 of the sensor unit 1, i.e. for example, by pressing a button, a signal to the control unit 2 to store data on the position and orientation relative to the reference coordinate system C of the sensor unit 1, which is provided by the sensor unit 1 at a given moment. From this data, together with the data obtained from the first part of the calibration (between the reference coordinate system G of the hand-held working tool 6 and the reference coordinate system C of the sensor unit 1), a transformation matrix Tcp is subsequently constructed defining the spatial relationship between the reference coordinate system C of the sensor unit 1 and the reference coordinate system P of the part 7. This calibration is performed at the beginning of each demonstration of a work operation for which the user wants to generate a robot program.

Furthermore, the transformation matrix TRP defining the mutual spatial relationship between the reference coordinate system R of the stationary robot 5 and the reference coordinate system P of the part 7 is obtained in an analogous manner.

- 14 CZ 2023 - 384 A3

This fourth part of the calibration is also carried out using the second calibration fixture 27, however, instead of the manual working tool 6, a robotic working tool 25 is approached to the second calibration fixture 27 with the second visual mark (guided using the controller of the stationary robot 5), to which the sensor unit 1 is attached for the purposes of this part of the calibration. The mutual position of the second calibration fixture 27 and the part 7 is fixed, but does not have to be known. However, it must remain unchanged during the calibration and during the step of performing the work operation for the surface treatment of the part 7 using the manual working tool 6, i.e. during the demonstration.

The user therefore performs this part of the calibration by approaching the second visual mark and aiming the camera 9 at the second visual mark. The camera 9 detects this second visual mark and determines its relative position and orientation with respect to it. The user further notes the position and orientation of the robotic working tool 6 in the reference coordinate system R of the stationary robot 5 using the controller of the stationary robot 5. From this data, together with the data obtained from the second part of the calibration (between the reference coordinate system C of the sensor unit 1 and the reference coordinate system R of the stationary robot 5), a matrix Trp is subsequently constructed defining the mutual spatial relationship between the reference coordinate system R of the stationary robot 5 and the reference coordinate system P of the part 7. This calibration needs to be performed only once, provided that the part 7, for which robotic instructions are generated, will subsequently always be located in the same place relative to the stationary robot 5, which will perform an automated work operation on it according to the generated robotic program.

At this point it can be mentioned that the second part of the calibration is necessary because the camera 9 is used for the fourth part of the calibration (stationary robot 5-part 7). It is also worth mentioning that the sensor unit 1 is mounted on the robotic working tool 25 only for the purpose of calibration and is no longer mounted on the robotic working tool 25 during the execution of the automated working operation.

In the first exemplary embodiment of the method, the step of performing the calibration measurement is followed by the step of performing the work operation for the surface treatment of the part 7 using the hand-held working tool 6 with the attached sensor unit 1. During the performance of the work operation for the surface treatment of the part 7, i.e. for example during the demonstrative painting of the part 7 with a hand-held paint gun, the position and orientation and also the working state of the hand-held working tool 6 are monitored using the sensor unit 1, as has also been described above. This monitoring (i.e. recording/recording of data) is started by the user, e.g. using the user interface 19 of the sensor unit 1 or the user interface 20 of the control unit 2, signaling the start of recording to the control unit 2. After performing the work operation for the surface treatment of the part 7 using the hand-held working tool 6, the user signals the end of recording in a similar manner. During recording, the data from the sensor unit 1 are stored in the memory 14.

After recording is complete, the stored data is transferred to a remote server 21 via wireless (Wi-Fi) communication, where it is processed by the data processing module 3 in order to generate instructions for the stationary robot 5. Calibration data is also transferred to the input of the data processing module 3.

In the described first exemplary embodiment of the method, as schematically shown in Fig. 3, data processing includes several sub-steps, namely a data filtering step, a data time synchronization step, a dynamic subsampling step, a step of applying a spatial transformation based on the provided calibration data and subsequently a step of generating instructions for the stationary robot 5. By applying a spatial transformation based on the calibration data, the recorded data are transformed into the reference coordinate system R of the stationary robot 5, which allows generating instructions that are expressed in this reference coordinate system R of the stationary robot 5.

The instruction generation step is performed by an instruction generator 23, which in the first embodiment described is part of the remote server 21 and which includes an interface for generating

- 15 CZ 2023 - 384 A3 instructions for a stationary robot 5 in the programming language of at least one manufacturer of stationary robots 5. The instruction generator 23 contains information about the syntax, semantics, required structure and format of the instructions for the stationary robot 5, including information about the keywords used to write individual instructions for the stationary robot 5 in the programming language of the given manufacturer of the stationary robot 5. The executable robot program is created by the instruction generator 23 writing individual instructions according to the rules of syntax and semantics and using the values of the arguments that are the result of previous data processing.

For example, for an ABB painting robot, the 23-instruction generator creates a file with the extension .mod, which contains the corresponding MODULE and PROCEDURE sections, and in the PROCEDURE section, individual instructions are listed on individual lines. These instructions begin with the keyword PaintL and are followed by the values of the instruction arguments x, y, z, qi, q2, q3, q ₄ (and others), after which the 23-instruction generator substitutes specific numbers that determine where the robotic work tool 25 should move, how fast, etc.

The method and system of the present invention may also be implemented in other exemplary embodiments than the first exemplary embodiment described in detail above. The individual alternatives by which the other exemplary embodiments may differ are explained in the Summary of the Invention section.

Industrial applicability

The above-described method and system can be used to generate instructions usable in various surface treatment operations, such as painting, grinding, polishing, chamfering, blasting, etc. In addition, the generated instructions (robotic programs) can be uploaded not only to the controller of a real stationary robot, but also to the controller of a virtual stationary robot, i.e. to its digital twin for simulation purposes.

Claims (13)

1. A method of generating instructions for a stationary robot (5) based on a demonstration of a work operation for surface treatment of a part (7), characterized in that it includes the steps of: - performing a work operation for surface treatment of the part (7) using a hand-held work tool (6) with a mounted sensor unit (1), - monitoring the position and orientation of the hand-held working tool (6) by the sensor unit (1) during the step of performing the work operation for surface treatment of the part (7), wherein the step of monitoring the position and orientation of the hand-held working tool (6) includes the step of recording acceleration and angular velocity over time by an inertial sensor (8) and the step of recording visual images of the surrounding environment at different points in time by at least one camera (9), - monitoring the working state of the hand-held working tool (6) by a working state sensor (10) during the step of performing a working operation for surface treatment of the part (7), wherein the working state sensor (10) provides information on whether the hand-held working tool (6) is started, - providing calibration data to the data processing module (3), the calibration data provided to the data processing module (3) including information on the mutual spatial relationship between the reference coordinate systems of the sensor unit (1), the hand-held work tool (6), the stationary robot (5) and the part (7), and - processing data from the inertial sensor (8), at least one camera (9) and the working condition sensor (10) as well as calibration data by the data processing module (3), wherein the data processing step includes the step of generating instructions for the stationary robot (5) in the programming language of at least one manufacturer of stationary robots (5). 2. The method according to claim 1, characterized in that the provided calibration data includes at least information about the mutual spatial relationship between the reference coordinate system G of the hand-held working tool (6) and the reference coordinate system C of the sensor unit (1), information about the mutual spatial relationship between the reference coordinate system C of the sensor unit (1) and the reference coordinate system P of the part (7) and information about the mutual spatial relationship between the reference coordinate system R of the stationary robot (5) and the reference coordinate system P of the part (7). 3. A method according to any one of the preceding claims, characterized in that to obtain calibration data that is provided to the data processing module (3), the method comprises the step of performing a calibration measurement. 4. The method according to claims 2 and 3, characterized in that unique visual markers are used for the step of performing the calibration measurement, wherein information on the mutual spatial relationship between the reference coordinate systems G and C, information on the mutual spatial relationship between the reference coordinate systems C and P, information on the mutual spatial relationship between the reference coordinate systems R and P, and in addition also information on the mutual spatial relationship between the reference coordinate systems C and R, are obtained based on detecting the visual markers by at least one camera (9) and on determining the relative orientation of the sensor unit (1) with respect to the visual markers. 5. The method according to any one of the preceding claims, characterized in that the step of generating instructions for the stationary robot (5) is performed by an instruction generator (23) containing information on the syntax, semantics, required structure and format of instructions for the stationary robot (5) of at least one manufacturer of stationary robots (5), wherein the instructions are written by the instruction generator (23) in the required structure and format based on the information on the syntax and semantics and contain keywords and argument values that are the result of the data processing step by the data processing module (3). 6. The method according to any one of the preceding claims, characterized in that the step of processing data by the data processing module (3) further before the step of generating instructions for the stationary robot (5) comprises a step of filtering data, a step of time synchronization of data, a step of dynamic subsampling and a step of applying a spatial transformation based on the provided calibration data. - 17 CZ 2023 - 384 A3 7. The method according to any one of the preceding claims, characterized in that the step of monitoring the working state is performed by an audio sensor recording an audio signal during the step of performing the working operation for the surface treatment of the part (7). 8. A system for performing the method according to any one of the preceding claims 1 to 7, characterized in that it comprises a sensor unit (1) and a control unit (2) communicatively connected to the sensor unit (1) and adapted to control the sensor unit (1), wherein the sensor unit (1) comprises: - a fastening element (11) for fastening to a hand-held working tool (6), - inertial sensor (8) for recording acceleration and angular velocity over time, - at least one camera (9) for recording visual images of the surrounding environment at different points in time and - a working state sensor (10) for monitoring the working state of the hand-held work tool (6), wherein the working state sensor (10) provides information about whether the hand-held work tool (6) is started, wherein the system further comprises a data processing module (3) adapted to receive data from the inertial sensor (8), at least one camera (9) and the working state sensor (10) as well as calibration data containing information about the mutual spatial relationship between the reference coordinate systems of the sensor unit (1), the hand-held work tool (6), the stationary robot (5) and the part (7), wherein the data processing module (3) comprises an instruction generator (23) comprising an interface for generating instructions for the stationary robot (5) in a programming language of at least one manufacturer of stationary robots (5). 9. The system of claim 8, further comprising at least one first calibration fixture (26) having at least one calibration aperture and a set of unique first visual indicia, wherein the relative positions of the first visual indicia with respect to each calibration aperture are known, and wherein the at least one calibration aperture is shaped to receive the manual work tool (6) in only one particular position and orientation, and the at least one calibration aperture is shaped to receive the robotic work tool (25) in only one particular position and orientation, the system further comprising a second calibration fixture (27) including a unique second visual indicia, wherein the relative position of the second calibration fixture (27) and the part (7) is fixed. 10. The system according to any one of the preceding claims 8 to 9, characterized in that the operating state sensor (10) is an acoustic sensor. 11. The system according to any one of the preceding claims 8 to 10, characterized in that the control unit (2) comprises a microcomputer (13) for controlling the sensor unit (1), a memory (14) for storing data from the sensor unit (1) and a communication module (15) for communication connection with the data processing module (3). 12. The system according to any one of the preceding claims 8 to 11, characterized in that it further comprises a remote server (21) communicatively connected to the control unit (2), wherein the data processing module (3) is part of the remote server (21). 13. The system according to any one of the preceding claims 8 to 12, characterized in that it further comprises a user interface of the data processing module (3) for configuring the instruction generator (23) and also an access device (4) for accessing the user interface of the data processing module (3), wherein the user interface of the data processing module (3) is communicatively connected to the data processing module (3).