CN117784637A - Universal semi-physical simulation platform and method for planning and controlling multiple robots - Google Patents

Abstract

This application relates to a universal semi-physical simulation platform and method for multi-robot planning and control. The universal semi-physical simulation platform adopts the physical design of a small omnidirectional mobile robot combined with the virtual presentation of test scenarios. It can support users to customize initial parameters such as task environment, task type, team size, and multi-robot physical characteristics, and provide friendly human interface. The computer interactive interface allows users to intervene in the test process at any time; it can output detailed process data, improve the display of test results, reduce the loss of physical robots, and accelerate the update and iteration process of the algorithm.

Description

Universal semi-physical simulation platform and method for multi-robot planning and control

Technical field

This application relates to the field of robotics technology, and in particular to a universal semi-physical simulation platform and method for multi-robot planning and control.

Background technique

In recent decades, multi-robot systems, especially distributed multi-robot systems, have attracted much attention due to their inherent advantages over single-robot systems. Thanks to the absence of a centralized control mechanism, each robot operates independently based on local sensing information and controllers, and the corresponding working environment can be broader and more dynamic. Precisely due to the complexity and diversity of the system itself and its operating environment, the current mainstream multi-robot planning and control algorithm testing platforms and methods can be divided into two types, namely simulation verification systems and physical verification systems. The above two platforms can support some Simple multi-robot algorithm experiment, but still has the following shortcomings:

1) The algorithm testing effect of the simulation verification system is not good. Most of the existing algorithm testing systems are numerical simulations or two-dimensional plane simulations of simplified robot models (such as particle models), that is, they do not build the kinematics model or dynamics model of the robot. At the same time, unfavorable factors inherent in the actual environment such as the heterogeneous characteristics of the robot, the collision volume in the environment, perceived noise and communication delays are not taken into account, and the implementation effect of the algorithm is tested in an extremely ideal simulation environment. As a result, there is a large difference between the simulation test setup and the real environment, and the algorithm test results cannot be reproduced in the physical test, thus losing the significance of algorithm testing.

2) The physical verification system still has certain limitations. First, due to the large size and high cost of each robot, the number of robots available for a test is generally small, and there are high requirements for the test site, which is not conducive to expansion. At the same time, the hardware and software design of the physical robot is complex. In order to ensure the normal progress of the test, it is generally necessary to carry out sensor calibration, motor adjustment and other preparatory work in advance, which makes each test time-consuming and not conducive to repeated experiments. In addition, the experimental effect display is low, and the test environment needs to be pre-set according to the application background of different algorithms, which usually requires temporary modification of the laboratory, and the task environment is difficult to build.

Summary of the invention

Based on this, it is necessary to provide a stable, fast and easy-to-operate universal physical-in-the-loop simulation platform and method for multi-robot planning and control in response to the above technical problems.

A universal semi-physical simulation platform for multi-robot planning and control. The platform includes: several robots with Apriltag tags pasted on the bottom, a test scene simulation system and a workstation.

The test scene simulation system includes a test scene display module that supports unlimited multi-point simultaneous touch. It is used to display the test scene image designed by the workstation and simulate the test scene. During the simulation process, virtual obstacles or physical obstacles are added according to the task needs; it is also used It is used to obtain the test environment information and robot motion information, and transmit the environment information and robot information to the workstation; the test environment information includes the type, location, and dimensions of obstacles; the robot motion information includes the robot's position, orientation, speed, and acceleration.

The workstation is used to design test scene images based on Unreal Engine according to preset tasks, and send the test scene images to the test scene display module; it is also used to receive test environment information, robot motion information, and robot status information, and based on the test environment information, Robot motion information, robot status information and preset control algorithms determine the motion control instructions of each robot and send each motion control instruction to the corresponding robot.

The robot is a small omnidirectional mobile robot, used to move on the upper surface of the test scene display module according to the received motion control instructions, and transmit its own robot status information back to the workstation.

In one embodiment, the robot adopts a double-layer structure design. The upper layer includes a battery, a sensor module, and a communication motherboard, and the lower layer includes a gear train, a motor module, and a motor drive board.

The sensor module is connected to the communication motherboard. WiFi is used for data transmission between the communication motherboard and the workstation. The motor module is connected to the motor drive board. The motor drive board and the communication motherboard transmit data through the I2C bus. The motor module is installed in conjunction with the wheel train. .

The communication mainboard is used to receive motion control instructions sent by the workstation and send the motion control instructions to the motor drive board; it is also used to transmit the robot status information obtained by the sensor module back to the workstation.

The motor driver board is used to receive motion control instructions sent by the communication motherboard, and perform closed-loop control of the motor module according to the motion control instructions.

The battery is used to provide power to the sensor module, communication motherboard, motor module and driver board.

In one embodiment, the communication mainboard includes a power supply module, a main chip module, a USB conversion module and a WiFi antenna.

The power supply module includes two voltage-reduction units. The input terminal of the first voltage-reduction unit is connected to the output terminal of the battery. The first output terminal of the first voltage-reduction unit is connected to the input terminal of the second voltage-reduction unit. The output terminal of the buck unit is connected to the main chip module, and the second output terminal of the first buck unit is connected to the motor drive board.

The main control chip module transmits data with the motor driver board through the I2C bus. The main control chip module is connected to the USB conversion module through the UART interface; the WiFi antenna is connected to the main control chip module.

In one embodiment, the motor driving board includes a motor driving board main chip and two motor driving chips.

The main chip of the motor driving board and the communication main board transmit data through the I2C bus. The main chip of the motor driving board is connected to the motor driving chip, and the motor driving chip is connected to the motor module.

The main chip of the motor driver board is used to receive the robot control instructions sent by the workstation transmitted by the communication mainboard, and convert the robot control instructions into PWM signals after processing and transmit them to the motor driver chip; and receive and process the information from the motor Hall encoder, and convert the processed information into I2C signals and transmit them to the communication mainboard.

In one embodiment, the wheel system adopts a double-layer orthogonal omnidirectional wheel design with a 6-roller structure, and the relationship between the robot's body speed and the wheel speeds of the three wheels is:

Among them, Va, Vb, and Vc are the wheel speeds of the three wheels respectively, v _x and v _y are the body speeds of the robot in the x direction and y direction respectively, and ω is the rotation speed of the robot.

In one embodiment, the test scene display module includes: several touch screens with a borderless design, the touch screens are spliced into a touch screen array; the touch screen is a full high-definition liquid crystal multi-touch screen;

The virtual obstacle is an obstacle directly drawn on the touch screen through human-computer interaction; the virtual obstacle has an actual position and size; the touch screen converts the category, position and size of the virtual obstacle into TUIO format information, and transmits the obtained TUIO format information to the workstation.

Physical obstacles are miniature obstacle models that are placed at the bottom of the touch screen and pasted with Apriltag marking the type of obstacle.

The touch screen captures the physical QR code, touch point information and object shape above it through the built-in computer system, then converts the captured information into TUIO format information, and transmits the obtained TUIO format information to the workstation.

In one embodiment, the simulation software running in the workstation is designed based on the robot's open source operating system. The workstation and the robot communicate through the MQTT protocol, and the workstation and the test scene display module communicate through the TUIO protocol.

The simulation software includes data acquisition module, control module and signal transmission module.

The data acquisition module is used to receive environmental information and robot motion information in TUIO format from the touch screen, convert the received TUIO format information into ROS messages, and transmit them to the control module.

The control module is used to process the ROS message using a preset control algorithm, determine the motion control instructions of each robot, and transmit each motion control instruction to the signal transmission module.

The signal transmission module is used to convert the motion control instructions in ROS format issued by the control module into MQTT message format, and send them to the corresponding small omnidirectional mobile robot; it is also used to receive the robot status information in MQTT format uploaded by the robot, and Convert the robot status information in MQTT format to ROS format and then transmit it to the control module.

In one of the embodiments, the control module is also used to monitor and control the semi-physical simulation process through a simulation control terminal designed based on Qt.

The simulation control terminal includes the simulation interface, physical H-MRS selection area, initialization area, dynamic parameter adjustment area, and environment setting area.

The simulation interface is used to display the constructed semi-physical simulation environment and display the movement method and trajectory of the physical robot in real time; it is also used to correspond to the display of the Unreal Engine and display animations on the touch screen.

The dynamic parameter adjustment area is used to adjust the algorithm parameters of the robot according to the test algorithm design.

The initialization area is used to initialize the number of robots, initialize the robot positions, and initialize the robot formation connections; the environment setting area is used for users to construct virtual scenes by adding or deleting different types of obstacles and targets according to preset task requirements.

When the simulation mode does not select the physical H-MRS mode, a pure simulation experiment is performed through the simulation control terminal.

When the physical H-MRS mode is selected as the simulation mode, the simulation interface corresponds to the position of the physical robot on the touch screen. Instructions are issued to the robot through the simulation control terminal and dynamic parameter adjustment is realized; the virtual robot in the simulation interface is Directly affects the physical robot when dragging and guiding.

In one of the embodiments, the platform is also used to simulate random dynamic obstacles, and the simulation of random dynamic obstacles is achieved by sliding a handheld object on the touch screen.

The platform is also used to simulate external interference during the simulation process. External interference is achieved by dragging or blocking the physical robot.

The platform is also used to simulate the process of adding or removing any robot from the formation during the simulation.

A universal hardware-in-the-loop simulation method for multi-robot planning and control. The method is suitable for conducting simulation experiments using any of the above-mentioned universal hardware-in-the-loop simulation platforms for multi-robot planning and control; the method includes:

Design experimental scene images based on Unreal Engine according to preset tasks in the workstation.

Display the test scene image in the test scene display module to simulate the test scene, and add virtual obstacles or physical obstacles according to task needs during the simulation process.

Place several robots with Apriltag tags pasted on the bottom on the display screen of the test scene display module; the test scene display module supports unlimited simultaneous touches.

The test scene display module obtains test environment information and robot motion information; test environment information includes the type, location, and dimensions of obstacles; robot motion information includes the robot's position, orientation, speed, and acceleration.

In the workstation, the motion control instructions of each robot are determined based on the test environment information, robot motion information, robot status information and preset control algorithms.

The robot moves on the upper surface of the test scene display module according to the corresponding motion control instructions, and transmits its own robot status information back to the workstation.

The above-mentioned universal semi-physical simulation platform and method for multi-robot planning and control. The universal semi-physical simulation platform adopts the physical design of small omnidirectional mobile robots combined with the virtual presentation of test scenarios, and can support users to customize task environments, task types, and teams. Initial parameters such as scale and multi-robot physical characteristics are provided, and a friendly human-computer interaction interface is provided to facilitate users to intervene in the test process at any time; detailed process data can be output to improve the display of test results, reduce physical robot losses, and accelerate algorithms. update iterative process.

Description of drawings

Figure 1 is a block diagram of a general semi-physical simulation platform for multi-robot planning and control in one embodiment;

Figure 2 is a schematic diagram of the composition of a general semi-physical simulation platform for multi-robot planning and control in one embodiment;

Figure 3 is a structural diagram and a multi-view view of a small omnidirectional mobile robot in another embodiment, in which (a) is a structural diagram of a small omnidirectional mobile robot, (b) is a main view, (c) is a top view, and (d) is Left view, (e) is the rear view;

Figure 4 shows the speed decomposition of a small omnidirectional mobile robot in another embodiment;

Figure 5 is a schematic block diagram of the electrical system of the robot in another embodiment;

Figure 6 shows the identification of the positions and directions of multiple robots in another embodiment;

Figure 7 shows the 2×2 multi-screen splicing effect in another embodiment;

Figure 8 shows an indoor fire environment in another embodiment;

Figure 9 shows the software framework and node composition in another embodiment;

FIG10 is a simulation control terminal implemented based on Qt in another embodiment;

Figure 11 shows a general physical-in-the-loop simulation method for multi-robot planning and control in another embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

In one embodiment, as shown in Figures 1 and 2, a general semi-physical simulation platform for multi-robot planning and control is provided. The platform includes: several robots with Apriltag tags pasted on the bottom 10. A test scene simulation system 20 and workstation 30.

The test scene simulation system 20 includes a test scene display module that supports unlimited multi-point simultaneous touch, and is used to display the test scene image designed by the workstation 30, simulate the test scene, and add virtual obstacles or physical obstacles according to task needs during the simulation process; It is also used to obtain test environment information and robot motion information, and transmit the environment information and robot information to the workstation; the test environment information includes the type, location, and dimensions of obstacles; the robot motion information includes the position, orientation, speed, and acceleration.

Specifically, as a semi-physical simulation platform, the test scene used is a simulation environment. The test scene simulation system 20 adopts a comprehensive computer system that has its own computer system and can capture the QR code/object shape/touch point information above it. HD LCD touch screen.

The virtual presentation of test scenarios can reduce the cost of setting up different task environments, improve test efficiency, accelerate algorithm iteration, and enhance the versatility of the system. At the same time, combined with the use of physical robots and miniature models, the displayability of the test process is greatly improved. It supports a variety of human-computer interaction methods. Users can directly intervene in the test process by touching, dragging, etc., fully embodying the human factor in multi-robot planning and control algorithms. The workstation 30 is used to design test scene images based on Unreal Engine according to preset tasks, and send the test scene images to the test scene display module; it is also used to receive test environment information, robot motion information, and robot status information, and according to the test environment information , robot motion information, robot status information and preset control algorithms, determine the motion control instructions of each robot 20, and send each motion control instruction to the corresponding mobile robot 20.

The robot 20 is a small omnidirectional mobile robot, used to move on the upper surface of the test scene display module according to the received motion control instructions, and transmit its own robot status information back to the workstation 30 .

Specifically, as a semi-physical simulation platform, the robot used adopts physical design, retains the physical characteristics of the robot itself, and makes the operation effect closer to the real scene. Under the premise of limited test environment, in order to support large-scale multi-robot system to carry out tests at the same time (number>30), the robots used should meet the requirements of small size, low cost, easy maintenance/replacement, and individual capabilities such as communication, control, planning and movement.

The most basic physical characteristics of robot experiments are retained, and the impact of adverse factors such as collision volume, perception noise, and communication delay on the test results in real environments can be reflected, making the algorithm test results more reliable. At the same time, the miniaturized design also reduces the manufacturing cost and cycle of a single robot, and can support larger-scale multi-robot experiments in a limited range of scenarios.

The above-mentioned universal semi-physical simulation platform for multi-robot planning and control. The universal semi-physical simulation platform adopts the physical design of small omnidirectional mobile robots combined with the virtual presentation of test scenarios, and can support users to customize the task environment, task type, team size and Initial parameters such as the physical characteristics of multiple robots are provided, and a friendly human-computer interaction interface is provided to facilitate users to intervene in the test process at any time; detailed process data can be output to improve the display of test results, reduce the loss of physical robots, and accelerate algorithm updates. Iteration process.

In one embodiment, the robot adopts a double-layer structure design, the upper layer includes a battery, a sensor module and a communication main board, and the lower layer includes a wheel system, a motor module and a motor drive board.

The sensor module is connected to the communication motherboard. WiFi is used for data transmission between the communication motherboard and the workstation. The motor module is connected to the motor drive board. The motor drive board and the communication motherboard transmit data through the I2C bus. The motor module is installed in conjunction with the wheel train. .

The communication main board is used to receive motion control instructions sent by the workstation and send the motion control instructions to the motor drive board; it is also used to transmit the robot status information obtained by the sensor module back to the workstation.

The motor driver board is used to receive motion control instructions sent by the communication motherboard, and perform closed-loop control of the motor module according to the motion control instructions.

The battery is used to provide power to the sensor module, communication motherboard, motor module and driver board.

In one embodiment, the communication mainboard includes a power supply module, a main chip module, a USB conversion module and a WiFi antenna.

The power supply module includes two voltage-reduction units. The input terminal of the first voltage-reduction unit is connected to the output terminal of the battery. The first output terminal of the first voltage-reduction unit is connected to the input terminal of the second voltage-reduction unit. The output terminal of the buck unit is connected to the main chip module, and the second output terminal of the first buck unit is connected to the motor drive board.

The main control chip module transmits data with the motor driver board through the I2C bus. The main control chip module is connected to the USB conversion module through the UART interface; the WiFi antenna is connected to the main control chip module.

In one embodiment, the motor driving board includes a motor driving board main chip and two motor driving chips.

The main chip of the motor driving board and the communication main board transmit data through the I2C bus. The main chip of the motor driving board is connected to the motor driving chip, and the motor driving chip is connected to the motor module.

The main chip of the motor driver board is used to receive the robot control instructions sent by the workstation transmitted by the communication main board, process the robot control instructions and convert them into PWM signals and transmit them to the motor driver chip; and receive and process the information from the motor Hall encoder. , convert the processed information into I2C signals and transmit them to the communication main board.

In one embodiment, the wheel system adopts a double-layer orthogonal omnidirectional wheel design with a 6-roller structure, and the relationship between the robot's body speed and the wheel speeds of the three wheels is:

Among them, Va, Vb, and Vc are the wheel speeds of the three wheels respectively, _vx and _vy are the body speeds of the robot in the x and y directions respectively, and ω is the rotation speed of the robot.

In one embodiment, the small omnidirectional mobile robot adopts a double-layer structure design. The upper layer is a battery and sensor module with a control and communication motherboard, and the lower layer is a gear train and motor module with a drive board. The overall size is controlled within a radius of 50mm and a height of 55mm. The structure is simple and is conducive to mass production. The structure of the small omnidirectional mobile robot is shown in (a) in Figure 3. The front view, top view, left view and rear view are shown in (b), (c), (d) and (e) in Figure (3) respectively. .

The specific design plan of the small omnidirectional mobile robot is as follows:

① Power supply method: The main power consumption of the omnidirectional mobile robot includes the drive motor and WIFI chip. The theoretical maximum power consumption of the motor is 2.4W under the rated voltage and current, and the WIFI chip is 1.2W when WIFI is turned on. Assuming that the total power consumption of the robot is 9W when both the 3 motors and the WIFI chip are running at maximum power consumption, the 3S model aircraft battery (550mAh) can be used for 40 minutes without considering over-discharge.

②Data connection method: In the wireless connection method, Bluetooth transmission speed is slow, so this solution uses 2.4GHz WIFI for data transmission. In addition, a USB module is added to the mobile robot for program burning and serial port monitoring during debugging.

In order to simplify the robot's motion control difficulty and improve its flexibility, a three-wheel omnidirectional motion mechanism design is adopted. The following is the wheel arrangement and drive motor selection scheme:

① Wheel train arrangement: In order to realize the omnidirectional movement of the robot, a double-layer orthogonal omnidirectional wheel design with a 6-roller structure is used to improve the flexibility of the robot's movement. The relationship between the body speed v _x and v _y and the wheel speed v ₁ , v ₂ and v ₃ is shown in Figure 4. The calculation formula is as follows:

v ₁ =v _x +ωL

v ₂ =-v _x cosθ ₁ -v _y sinθ ₁ +ωL

v ₃ =-v _x sinθ ₂ +v _y cosθ ₂ +ωL

Its matrix form is:

② Drive motor selection: Select a 12V motor based on the selected 3S battery, and reasonably select the reduction ratio to obtain greater torque and ensure that the omnidirectional mobile robot can move at a faster speed. The motor torque is analyzed from the perspective of power. Its minimum torque needs to support the movement of the omnidirectional mobile robot. Assume that only one wheel is rotating. At this time, there are:

Power=torque×speed

Also:

Rotation speed = speed/(π×wheel diameter)

To make the robot move, that is, the friction force does work. At this time, there is:

Power = Friction × Speed

In this embodiment, the robot weighs about 200g, the wheel diameter is 21mm, and the rolling friction coefficient between the rubber tire and the glass is usually between 0.1 and 0.3. Assuming the maximum value of 0.3, when a single wheel drives the robot to move, the torque must satisfy:

Torque=0.3×200×10 ^-3 ×9.8×π×21≈38.8mN*m

Based on mature and reliable, it is compatible with the latest design scheme, reduces manufacturing costs, improves reliability and manufacturability, and meets the test technology requirements. The electrical circuit of the system is divided into two parts, and the data line is used to integrate the I2C signal line and the power supply line for connection, as shown in Figure 5. The functions of the two circuit boards are the upper WIFI board (ESP32_A) and the lower motor driver board (ESP32_B).

The circuit module scheme of the WIFI board (ESP32_A) is mainly divided into power supply module, main chip module and USB conversion module, which are described as follows:

① Power supply module: The main function is to step down the 12V voltage from the 3S aircraft model battery, convert it into 5V, and then convert 5V into 3.3V. Because the voltage step-down range from 12V to 5V is large, using LDO for voltage step-down will cause severe heating, so the 5V step-down module uses the DCDC step-down method, and the 3.3V step-down module uses the LDO step-down method. In addition, the power supply module reserves an external power switch interface (the switch is embedded in the robot shell) and a power display module interface;

②Main chip module: The main chip of the WIFI board adopts ESP32-MINI-1U. Its main functions are to receive WIFI signals from the host, transmit I2C signals to the motor driver board (ESP32_B), transmit UART signals to the USB conversion module, and control LED lights ( LED is a reserved interface and is embedded in the robot shell);

③USB conversion module: The UART signal from the ESP32-MINI-1U main chip is converted into a USB signal through the conversion chip CP2102, which is used for code burning and serial port viewing.

The circuit module design of the motor driver board (ESP32_B) is mainly divided into the main chip module and the motor driver module, which are described by module:

① Main chip module: The main chip of the motor driver board adopts ATMEGA328, whose main function is to receive the I2C signal from the WIFI circuit board ESP32_A, convert it into PWM signal after processing and transmit it to the motor driver chip, and receive the information from the motor Hall encoder, convert it into I2C signal after processing and transmit it back to the WIFI board ESP32_A;

② Motor drive module: Two L293DDs are used. The power supply comes from the data line between the two circuit boards. 12V is used to drive the motor and 5V is used to power the motor drive chip.

Table 1 Selection of key components

In one embodiment, the test scene display module includes: several touch screens with a frameless design, the touch screens are spliced into a touch screen array; the touch screen is a full high-definition liquid crystal multi-touch screen.

Virtual obstacles are obstacles drawn directly on the touch screen through human-computer interaction; virtual obstacles have actual positions and dimensions; the touch screen converts the categories, positions and dimensions of virtual obstacles into TUIO format information, and transmit the obtained TUIO format information to the workstation.

Physical obstacles are miniature obstacle models that are placed at the bottom of the touch screen and pasted with Apriltag marking the type of obstacle.

The touch screen captures the physical QR code, touch point information and object shape above it through the built-in computer system, then converts the captured information into TUIO format information, and transmits the obtained TUIO format information to the workstation.

TUIO (Table-Top User Interfaces Objects) is a simple and universal protocol used for the human-computer interaction interface of touch screen devices.

The number of touch screens may be, but is not limited to, 4, and may also be 6, 8, 9, 10, etc. The touch screen array may be, but is not limited to, a square touch screen array or a rectangular touch screen array.

In a specific embodiment, four MultiTraction MT557D full-HD LCD multi-touch screens are spliced together as the basic test scene display means. The device is based on the Clearsight technology touch engine and can support unlimited multiple points of simultaneous touch. The touch delay is about 10 milliseconds and reaches sub-pixel accuracy. The Apriltag mark attached to the bottom of the object can quickly identify the position and orientation information of the physical object placed on it, and calculate the speed and acceleration of the object. After numbering, it can be tracked and output in multiple formats. Taking the above-mentioned small omnidirectional mobile robot as an example, the recognition effect As shown in Figure 6.

As a general-purpose semi-physical simulation platform, this system uses MultiTraction's excellent display effects to quickly build a variety of test environments. As shown in Figure 7, this embodiment uses four full-HD LCD multi-touch screens to be spliced in a 2×2 manner. The screen adopts a borderless design, allowing the robot to move across the screen. Combined with the smaller physical robot size, it can support multi-robot systems with a number of >30 units to carry out algorithm experiments.

According to the task type, the screen display image can be designed based on Unreal Engine (UE) to simulate the test environment, and virtual obstacles or physical obstacles can be added as needed. Take the task of a multi-robot system to extinguish a fire and search for survivors in an indoor environment as an example, as shown in Figure 8. Bar obstacles (walls, cabinets, etc.) and circular obstacles (seats, etc.) can be set on the screen. These obstacles are displayed as virtual images, but have actual position and size information, and are fed back to the nearby robots by the screen, and then "detected" by the robots. The required test environment can also be arranged by placing physical objects. By pasting Apriltag (different from the robot and cannot be reused) at the bottom of the miniature model, the screen can detect the position and direction of the object. Relying on virtual and physical obstacles, the test scene can be arranged flexibly, and the position of the obstacles can be adjusted at any time without reprogramming, further highlighting the semi-physical simulation characteristics of this system.

Thanks to the design of the above semi-physical simulation platform, it can also realize rich human-computer interaction processes and increase human intervention in multi-robot planning and control algorithms. Users can directly draw virtual obstacles, objects of interest and other environmental information by touching the screen, and can also hold real objects and slide them on the screen to simulate random dynamic obstacles. You can also directly control and intervene the physical robot by dragging or blocking it, and you can even join or remove any robot in the formation at any time. The above interactions can simulate complex factors in a variety of planning or control algorithms, providing more diverse algorithm testing requirements.

In one embodiment, the simulation software running in the workstation is designed based on the robot open source operating system, the workstation communicates with the robot through the MQTT protocol, and the workstation communicates with the test scene display module through the TUIO protocol.

The simulation software includes data acquisition module, control module and signal transmission module.

The data acquisition module is used to receive the TUIO format environmental information and robot motion information from the touch screen, convert the received TUIO format information into ROS messages, and transmit them to the control module.

The control module is used to process the ROS message using a preset control algorithm, determine the motion control instructions of each robot, and transmit each motion control instruction to the signal transmission module.

The signal transmission module is used to convert the motion control instructions in ROS format issued by the control module into MQTT message format, and send them to the corresponding small omnidirectional mobile robot; it is also used to receive the robot status information in MQTT format uploaded by the robot, and Convert the robot status information in MQTT format to ROS format and then transmit it to the control module.

MQTT (Message Queuing Telemetry Transport, Message Queuing Telemetry Transport Protocol) is a "lightweight" communication protocol based on the publish/subscribe (publish/subscribe) mode.

In one of the embodiments, the control module is also used to monitor and control the hardware-in-the-loop simulation process through a simulation control terminal designed based on Qt.

The simulation control terminal includes the simulation interface, physical H-MRS selection area, initialization area, dynamic parameter adjustment area, and environment setting area.

The simulation interface is used to display the constructed semi-physical simulation environment and to display the movement method and trajectory of the physical robot in real time. It is also used to correspond to the display of the Unreal Engine and to perform animation display on the touch screen.

The dynamic parameter adjustment area is used to adjust the robot's algorithm parameters (such as the robot's intention field, hybrid strategy, and guidance vector field) according to the test algorithm design.

The initialization area is used to initialize the number of robots, initialize robot positions, and initialize robot formation connections; the environment setting area is used for users to construct virtual scenes by adding or deleting different types of obstacles and targets according to preset task requirements.

When the physical H-MRS mode is not selected in the simulation mode, pure simulation experiments are performed through the simulation control terminal.

When the physical H-MRS mode is selected as the simulation mode, the simulation interface corresponds to the position of the physical robot on the touch screen. Instructions are issued to the robot through the simulation control terminal and dynamic parameter adjustment is realized; the virtual robot in the simulation interface is Directly affects the physical robot when dragging and guiding.

In one embodiment, the robot open source operating system (ROS, Robot Operating System) is used to complete the construction of the simulation software framework, which is mainly divided into three parts, including tuio.launch to obtain screen messages, the control algorithm node shared_control_node, and the mqtt_bridge.launch that sends robot messages, as shown in Figure 9. The squares in Figure 9 represent ROS topics, including /tuio and /vs. The communication of ROS messages between nodes relies on topics to implement publishing and subscription; the ovals represent ROS nodes, among which the nodes /tuio1, /tuio2, /tuio3 and / tuio4 represents the message format conversion node (used to convert the TUIO message format obtained by 4 screens into ROS message format), the node shared_control_node represents the control algorithm node (this node can only process ROS message format), and the node /mqtt_bridge represents the message format. The conversion node (used to realize the mutual conversion between ROS message format and MQTT message format).

① Among them, tuio.launch (tuio.launch is the launcher, because the test scene display module is spliced by four screens, which means four message conversion nodes are started at the same time) is used to convert TUIO messages into ROS message format, thereby obtaining the information on the screen. Basic information and sent to the workstation for processing. The built-in computer system of MultiTraction MT557D can capture the QR code/object shape/touch point information above it and convert it into TUIO information format. TUIO is an open framework that defines common protocols and application programming interfaces for tangible multi-touch surfaces. The TUIO protocol allows the transmission of abstract descriptions of interactive surfaces, including touch events and tangible object states. TUIO mainly includes Object, Cursor, and Blob information. Object is mainly used to identify QR code information on the screen. Cursor is mainly used to identify touch point information on the screen. Blob is mainly used to identify the shape information of objects placed on the screen. The three message formats are:

(1)Object message format:

object->getsymbolID();

object->getx(); object->getY();

object->getAngle();

object->getMotionSpeed();

object->getRotationSpeed();

object->getMotionAccel();

object->getRotationAccel();

(2)Cursor message format:

cursor->getCursorID();

cursor->getX(); cursor->getY();

cursor->getMotionSpeed();

cursor->getMotionAccel();

(3)Blob message format:

blob->getBlobID() blob->getx();

blob->getY()blob->getAngle();

blob->getwidth()blob->getHeight();

blob->getArea().

Correspondingly, the ROS message formats object.msg, cursor.msg and blob.msg are defined to receive TUIO messages obtained by the screen.

②The shared_control_node performs algorithm processing after receiving the corresponding ROS message. It can use C++ language or python language to define the corresponding interface and write code, and supports multiple algorithms such as deep learning. At the same time, in order to monitor and control the semi-physical simulation process, the present invention develops a Qt-based simulation control terminal.

As shown in Figure 10, when the physical H-MRS is not selected, pure simulation experiments are first performed based on the simulation control terminal to facilitate initial debugging of the algorithm and avoid damage to the physical robot by malignant bugs. When physical H-MRS is selected, the simulation interface corresponds to the position of the physical robot on the screen. Through the control terminal, commands such as start/stop can be issued to the robot and dynamic parameter adjustments can be realized. Actions such as dragging and guiding the robot in the terminal interface can directly affect the physical robot.

At the same time, the control terminal supports users to construct various scenarios for the semi-physical simulation environment. They can add or delete robots, establish robot formation connections, and provide robot path guidance. It also supports the addition of multiple elements, such as virtual obstacles or task targets of various shapes, allowing real-time display of the movement direction and trajectory of the physical robot, which is more conducive to experimental verification of the algorithm. In addition, the simulation control terminal can also correspond to the display of the UE and can directly perform animation display on MultiTraction to enhance the display effect of the test.

③Mqtt.launch is used to start the mqtt_bridge node and convert the ROS message sent by the control node into the MQTT message format. MQTT is a "lightweight" communication protocol based on the publish/subscribe model. Due to its low protocol overhead and low power consumption, it can meet the message transmission needs between small omnidirectional mobile robots. The pseudo code for mqtt_bridge parameter configuration is:

1mqtt:

2client:

3protocol:4#MQTTv311

4connections:

5host:localhost

6 port:1883

7 keepalive:60

8private_path:device/001

9#serializer:msgpack:dumps

10#deserializer:msgpack:loads

11bridge:

12#tuio mgtt

13-factory:mqtt_bridge.bridge:RosToMqttBridge

14msg_type:vs.msg

15topic from:/vs

16topic to:mgtt vs

17-factory:mqtt bridge.bridge:MgttToRosBridge

18msg_type:robot.msg

19topic_from:mgtt_robot

20topic to:/robot

This protocol completes the mutual conversion of ROS and MQTT messages based on the bridge file, and configures the corresponding parameters and message format. The robot receives the movement instructions sent by the shared_control_node through mqtt_bridge, and can additionally obtain the sensor information carried on the robot and send it back to the shared_control_node.

In one of the embodiments, the platform is also used to simulate random dynamic obstacles, which is achieved by sliding a physical object on a touch screen; the platform is also used to simulate external interference during the simulation process, which is achieved by dragging or blocking a physical robot; the platform is also used to simulate the process of adding or withdrawing any robot in the formation during the simulation.

In one embodiment, as shown in Figure 11, a universal hardware-in-the-loop simulation method for multi-robot planning and control is provided. The method is suitable for conducting simulation experiments using any of the above-mentioned universal hardware-in-the-loop simulation platforms for multi-robot planning and control; The method includes the following steps::

Step 400: Design a test scene image based on Unreal Engine according to the preset task on the workstation.

Step 402: Display the test scene image in the test scene display module to simulate the test scene, and add virtual obstacles or physical obstacles according to task requirements during the simulation process.

Step 404: Place several robots with Apriltag tags pasted on their bottoms on the display screen of the test scene display module; the test scene display module supports unlimited multi-point simultaneous touch.

Step 406: The test scene display module obtains test environment information and robot motion information; the test environment information includes the type, position, and size of obstacles; the robot motion information includes the position, orientation, speed, and acceleration of the robot.

Step 408: Determine the motion control instructions of each robot in the workstation based on the test environment information, robot motion information, robot status information and preset control algorithm.

Step 410: The robot moves on the upper surface of the test scene display module according to the corresponding motion control instructions, and transmits its own robot status information back to the workstation.

It should be understood that, although the various steps in the flowchart of Figure 11 are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in Figure 11 may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims (10)

1. A universal semi-physical simulation platform for multi-robot planning and control, characterized in that the platform includes: a number of robots with Apriltag tags pasted on the bottom, a test scene simulation system and a workstation; The test scene simulation system includes a test scene display module that supports unlimited multi-point simultaneous touch, which is used to display the test scene image designed by the workstation, simulate the test scene, and add virtual obstacles or physical obstacles according to task needs during the simulation process. object; also used to obtain test environment information and robot motion information, and transmit the environment information and robot information to the workstation; the test environment information includes the type, location, and dimensions of obstacles; the robot Movement information includes the robot's position, orientation, speed and acceleration; The workstation is used to design test scene images based on Unreal Engine according to preset tasks, and send the test scene images to the test scene display module; and is also used to receive the test environment information, the robot motion information, and Robot status information, and based on the test environment information, the robot motion information, the robot status information and the preset control algorithm, determine the motion control instructions of each robot, and send each of the motion control instructions to the corresponding robot; The robot is a small omnidirectional mobile robot, used to move on the upper surface of the test scene display module according to the received motion control instructions, and transmit its own robot status information back to the workstation. 2. The platform according to claim 1, characterized in that the robot adopts a double-layer structure design, the upper layer includes a battery, a sensor module and a communication mainboard, and the lower layer includes a gear train, a motor module and a motor drive board; The sensor module is connected to the communication mainboard, the communication mainboard and the workstation use WiFi for data transmission, the motor module is connected to the motor drive board, the motor drive board and the communication mainboard use I2C bus for data transmission; the motor module is installed in conjunction with the gear train; The communication mainboard is used to receive the motion control instructions sent by the workstation and send the motion control instructions to the motor drive board; and is also used to transmit the robot status information obtained by the sensor module back to said workstation; The motor drive board is used to receive the motion control instructions sent by the communication main board, and perform closed-loop control on the motor module according to the motion control instructions; The battery is used to provide power to the sensor module, communication main board, motor module and drive board. 3. The platform according to claim 2, characterized in that the communication mainboard includes a power supply module, a main chip module, a USB conversion module and a WiFi antenna; The power supply module includes two voltage-reduction units. The input terminal of the first voltage-reduction unit is connected to the output terminal of the battery. The first output terminal of the first voltage-reduction unit is connected to the input terminal of the second voltage-reduction unit. Connection, the output end of the second voltage reduction unit is connected to the main chip module, and the second output end of the first voltage reduction unit is connected to the motor drive board; The main control chip module performs data transmission with the motor drive board through the I2C bus, the main control chip module is connected with the USB conversion module through a UART interface, and the WiFi antenna is connected with the main control chip module. 4. The platform according to claim 2, characterized in that the motor drive board includes a motor drive board main chip and two motor drive chips; The main chip of the motor driving board and the communication main board perform data transmission through the I2C bus, the main chip of the motor driving board is connected to the motor driving chip, and the motor driving chip is connected to the motor module; The motor drive board main chip is used to receive the robot control instructions sent by the workstation transmitted by the communication main board, process the robot control instructions and convert them into PWM signals and transmit them to the motor drive chip; and receive from The information from the motor Hall encoder is processed, and the processed information is converted into an I2C signal and transmitted to the communication main board. 5. The platform according to claim 2, characterized in that the wheel train adopts a double-layer orthogonal omnidirectional wheel design with a 6-roller structure, and the relationship between the body speed of the robot and the wheel speed of the three wheels is: Among them, Va, Vb, and Vc are the wheel speeds of the three wheels respectively, _vx and _vy are the body speeds of the robot in the x and y directions respectively, and ω is the rotation speed of the robot. 6. The platform according to claim 1, characterized in that the test scene display module comprises: a plurality of touch screens with a borderless design, the touch screens are spliced into a touch screen array; the touch screen is a full high-definition liquid crystal multi-touch screen; The virtual obstacles are obstacles drawn directly on the touch screen through human-computer interaction; the virtual obstacles have actual positions and dimensions; the touch screen displays the categories, Convert the position and overall size into TUIO format information, and transmit the obtained TUIO format information to the workstation; The physical obstacle is a miniature model of the obstacle with an Apriltag mark that identifies the obstacle type and is placed on the bottom of the touch screen by physical placement; The touch screen captures the physical QR code, touch point information and object shape above it through the built-in computer system, then converts the captured information into TUIO format information, and transmits the obtained TUIO format information to the workstation. 7. The platform according to claim 6, characterized in that the simulation software running in the workstation is designed based on the robot open source operating system, the workstation communicates with the robot through the MQTT protocol, and the workstation communicates with the robot through the MQTT protocol. The above test scene display module communicates through the TUIO protocol; The simulation software includes a data acquisition module, a control module and a signal transmission module; The data acquisition module is used to receive the environmental information and robot motion information of the touch screen in TUIO format, convert the received TUIO format information into ROS messages, and transmit them to the control module; The control module is configured to use a preset control algorithm to process the ROS message after receiving it, determine the motion control instructions of each robot, and transmit each of the motion control instructions to the signal transmission module; The signal transmission module is used to convert the motion control instructions in ROS format issued by the control module into MQTT message format, and send them to the corresponding small omnidirectional mobile robot; it is also used to receive the MQTT format uploaded by the robot. robot status information, and convert the robot status information in MQTT format into ROS format and then transmit it to the control module. 8. The platform according to claim 7, characterized in that the control module is also used to monitor and control the hardware-in-the-loop simulation process through a simulation control terminal designed based on Qt; The simulation control terminal includes a simulation interface, a physical H-MRS selection area, an initialization area, a dynamic parameter adjustment area, and an environment setting area; The simulation interface is used to display the constructed semi-physical simulation environment and display the movement method and movement trajectory of the physical robot in real time; it is also used to correspond to the display of the Unreal Engine and perform animation display on the touch screen; The dynamic parameter adjustment area is used to adjust the algorithm parameters of the robot according to the test algorithm design; The initialization area is used to initialize the number of robots, initialize robot positions, and initialize robot formation connections; the environment setting area is used for users to construct virtual scenes by adding or deleting different types of obstacles and targets according to preset task requirements; When the physical H-MRS mode is not selected in the simulation mode, a pure simulation experiment is performed through the simulation control terminal; When the simulation mode selects the physical H-MRS mode, the simulation interface corresponds to the position of the physical robot on the touch screen, and instructions are issued to the robot through the simulation control terminal and dynamic parameter adjustment is realized; The virtual robot in the simulation interface directly affects the physical robot when dragged and guided. 9. The platform according to any one of claims 1 to 8, characterized in that the platform is also used to simulate random dynamic obstacles, and the simulation of random dynamic obstacles is achieved by sliding a handheld object on the touch screen. of; The platform is also used to simulate the application of external interference during the simulation process. The application of external interference is achieved by dragging or blocking the physical robot; The platform is also used to simulate the process of adding or removing any robot in the formation during the simulation process. 10. A universal hardware-in-the-loop simulation method for multi-robot planning and control, characterized in that the method is suitable for simulation using the universal hardware-in-the-loop simulation platform for multi-robot planning and control described in any one of claims 1-9. Experiment; the methods include: Design test scene images based on Unreal Engine based on preset tasks on the workstation; Display the test scene image in the test scene display module to simulate the test scene, and add virtual obstacles or physical obstacles according to task needs during the simulation process; Place several robots with Apriltag tags pasted on the bottom on the display screen of the test scene display module; the test scene display module supports unlimited multi-point simultaneous touch; The test scene display module obtains test environment information and robot motion information; the test environment information includes the category, location, and dimensions of obstacles; the robot motion information includes the robot's position, orientation, speed, and acceleration; Determine the motion control instructions of each robot in the workstation based on the test environment information, the robot motion information, the robot status information and the preset control algorithm; The robot moves on the upper surface of the test scene display module according to the corresponding motion control instructions, and transmits its own robot status information back to the workstation.