WO2025033210A1 - Information processing method, information processing device, terminal device, and system - Google Patents

Abstract

An information processing method, wherein an information processing device executes processing for: providing one or more terminal devices with other partial map information generated by another terminal device different from the terminal device itself; acquiring, from each terminal device, partial map information generated by the device itself and information relating to a matching result obtained as a result of the terminal device matching a relative positional relationship between trajectory position information and the other partial map information on the basis of the other partial map information and the trajectory position information, which indicates a position on a trajectory along which the device itself moves through a region included in the partial map information; and performing optimization processing for correcting an error in overall trajectory position information constructed by combining the trajectory position information of the terminal device on the basis of the information relating to the matching result.

Description

Information processing method, information processing device, terminal device, and system

This disclosure relates to an information processing method, an information processing device, a terminal device, and a system.

A technology called SLAM (Simultaneous Localization and Mapping) is known that simultaneously estimates the vehicle's position and creates a map of the environment.

For example, in Patent Document 1, in a robot system including a robot and a computer (server device), the computer generates and updates map data based on sensor data collected by each robot, and each robot estimates its own position based on the map data obtained from the computer.

JP 2012-256344 A

However, the above conventional technology does not necessarily make it possible to appropriately reduce the processing load on the server device for SLAM calculations targeting multiple robots in a SLAM system in which multiple robots operate simultaneously.

For example, the above-mentioned conventional technology reduces the computational load on a server device for movement control calculations for multiple robots in a SLAM system in which multiple robots operate simultaneously, but does not take into consideration reducing the computational load on a server device for SLAM calculations for multiple robots.

This disclosure therefore proposes an information processing method, information processing device, terminal device, and system that can appropriately reduce the processing load on a server device for SLAM calculations targeting multiple robots in a SLAM system in which multiple robots operate simultaneously.

According to the present disclosure, an information processing method is provided in which an information processing device provides one or more terminal devices with other partial map information generated by a terminal device different from the terminal device, and the terminal device acquires from the terminal device information and other partial map information, matching result information obtained by matching the relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device and the other partial map information, and performs an optimization process for correcting an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices based on the matching result information.

The present disclosure also provides an information processing method in which a terminal device acquires other partial map information generated by another terminal device different from the terminal device itself from an information processing device on the cloud side, and performs processing to match the relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating the position on the trajectory along which the terminal device moves through an area included in the partial map information generated by the terminal device itself, point cloud information indicating the position of an object within the partial map information, and the other partial map information, and uploads information on the matching result of the relative positional relationship between the trajectory position information and the other partial map information, and the partial map information to the information processing device that generates an environmental map corresponding to the partial map.

As described above, according to the present disclosure, in a SLAM system in which multiple robots operate simultaneously, it is possible to appropriately reduce the processing load on a server device for SLAM calculations targeting multiple robots. Note that the effects described here are not necessarily limited to those described herein, and may be any of the effects described in this disclosure.

FIG. 1 is a diagram illustrating an overview and a configuration example of a multiple-robot SLAM system 1 according to an embodiment of the present disclosure. FIG. 2 is a functional block diagram showing an example of a functional configuration of the sensing camera 20 according to the embodiment. 11A and 11B are diagrams illustrating a process of overlaying sensor data on a submap. FIG. 13 is a diagram showing an example of a matching process between the self trajectory and the self sub-map by the sensing camera 20-1. FIG. 13 is a diagram showing an overview of a matching process between the self trajectory and another submap. FIG. 1 illustrates a specific example of multiple robot SLAM processing. FIG. 1 is a functional block diagram showing an example of a conventional functional configuration of a multiple-robot SLAM system 1. FIG. 1 is a functional block diagram showing an example of the functional configuration of a proposed method in a multiple-robot SLAM system 1. 11 is a flowchart showing an upload process procedure executed by the sensing camera 20 in the proposed method. 10 is a flowchart showing a scan matching detection procedure executed by the sensing camera 20 in the proposed method. 10 is a flowchart showing a scan matching process procedure executed by the sensing camera 20 in the proposed method. 13 is a flowchart showing an optimization process performed by the server device 100 in the proposed method. FIG. 1 is a functional block diagram showing an example of a functional configuration of a first modified example of the multiple-robot SLAM system 1. 13 is a flowchart showing a pre-processing procedure related to a request for matching processing. 13 is a flowchart showing a procedure for requesting a matching process. 11 is a flowchart showing a procedure of a matching process executed by the server device 100 in response to a request for the matching process. FIG. 11 is a functional block diagram showing an example of a functional configuration of a modified example 2 in the multiple-robot SLAM system 1. 13 is a flowchart showing a procedure for requesting a matching process. FIG. 2 is a block diagram showing an example of a hardware configuration of a computer corresponding to an apparatus according to an embodiment of the present disclosure.

Below, an embodiment of the present disclosure will be described in detail with reference to the attached drawings. Note that in this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals to avoid redundant description.

One or more of the embodiments (including examples, variations, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from one another. Thus, these embodiments may contribute to solving different purposes or problems and may provide different effects.

The present disclosure will be described in the following order.
1. Introduction 2. Embodiment 2-1. Example of system configuration 2-2. Functional configuration of sensing camera 2-3. Overview of multiple robot SLAM 2-4. Overlay of sensor data 2-5. Matching process within a single device 2-6. Matching process between multiple devices 2-7. Specific example of multiple robot SLAM processing 2-8. Example of detailed system configuration (conventional method)
2-8-1. Functional configuration of terminal device 2-8-2. Functional configuration of information processing device 2-9. Detailed system configuration example (proposed method)
2-9-1. Functional configuration of terminal device 2-9-2. Functional configuration of information processing device 2-10. Operation procedure 2-10-1. Upload procedure of terminal device 2-10-2. Scan matching detection procedure of terminal device 2-10-3. Scan matching processing procedure of terminal device 2-10-4. Optimization processing procedure of information processing device 3. Modification 1
3-1. Detailed configuration example of the system in the modified example 1 3-1-1. Functional configuration of the terminal device 3-1-2. Functional configuration of the information processing device 3-2. Operation procedure 3-2-1. Request pre-processing procedure 3-2-2. Request processing procedure 3-2-3. Scan matching processing procedure of the information processing device 4. Modified example 2
4-1. Detailed configuration example of the system in the modified example 2 4-1-1. Functional configuration of the terminal device 4-1-2. Functional configuration of the information processing device 4-2. Operation procedure 5. Hardware configuration 6. Summary

1. Introduction
A SLAM system that realizes SLAM is composed of a front-end processing section that focuses on sensor signal processing, and a back-end processing section that focuses on pose graph optimization.

In a SLAM (multiple robot SLAM) system in which multiple robots operate simultaneously, the front-end processing unit may be placed on each robot on the edge side, and the back-end processing unit may be placed on a server device on the cloud side. In such a configuration, map data is integrated on the server device side, allowing each robot to use the same environmental map and reducing the processing load on the robots.

On the other hand, in a multi-robot SLAM system, as the number of robots increases, the load of the matching process performed by the server device between the robot's sensor data and map data increases at a rate squared with the number of robots, which is an issue.

This disclosure therefore proposes a method for suppressing an increase in the processing load of a server device when the number of robots increases by managing map data on a server device on the cloud side and using computing resources on the edge side for matching processing between map data and sensor data.

For example, according to the proposed method of the present disclosure, a terminal device on the edge side uploads map data to a server device on the cloud side, but does not upload sensor data, and instead performs matching processing within the device itself. For example, the terminal device performs matching processing between the sensor data it has acquired and the map data it has generated. The terminal device also performs matching processing between the sensor data it has acquired and the map data generated by a terminal device other than the terminal device itself. This processing is performed by each terminal device included in the multi-robot SLAM, and each terminal device transmits the matching results to the server device.

The server device performs optimization processing to optimize the map data acquired from each terminal device based on the matching results acquired from each server device, thereby generating a more accurate map. Each terminal device can then use this accurate map to estimate its own location.

2. Embodiments
Hereinafter, information processing according to an embodiment of the present disclosure will be described in detail with reference to the drawings. The information processing according to the embodiment of the present disclosure is realized in a SLAM system in which multiple robots operate simultaneously.

[2-1. Example of system configuration]
1 is a diagram illustrating an overview and a configuration example of a multiple-robot SLAM system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the multiple-robot SLAM system 1 according to the present embodiment includes a moving object 10, a sensing camera 20, and a server device 100.

The moving body 10 is an autonomously moving robot equipped with a sensing camera 20. Examples of moving bodies 10 include a robot vacuum cleaner and a drone. As shown in FIG. 1, an example of the multiple-robot SLAM system 1 includes multiple moving bodies 10, moving bodies 10-1...10-n. Note that there is no limit to the number of moving bodies 10 included in the multiple-robot SLAM system 1.

The sensing camera 20 is an example of a terminal device according to the embodiment, and is responsible for front-end processing in the multi-robot SLAM system 1, which is divided into a front-end and a back-end. Therefore, the sensing camera 20 can be said to be an edge-side terminal device.

Also, as shown in FIG. 1, the multiple robot SLAM system 1 includes multiple sensing cameras 20, such as sensing camera 20-1...sensing camera 20-n, and each sensing camera 20 is mounted on a moving body 10. FIG. 1 shows an example in which sensing camera 20-1 is mounted on moving body 10-1, and sensing camera 20-n is mounted on moving body 10-n. Note that the sensing camera 20 may be implemented as an external module attached to the moving body 10, or may be built into the moving body 10 during the manufacturing stage. Therefore, in the following embodiments, the processing described as being performed by the sensing camera 20 can be rephrased as processing of the moving body 10.

In addition, in the following embodiments, the expression "multiple robots" can be rephrased as "multiple sensing cameras 20", and the expression "own device" can include the meaning of both "the sensing camera 20 itself" and "the mobile body 10 on which the sensing camera 20 is mounted". Furthermore, the position and orientation of the sensing camera 20 essentially indicate the position and orientation of the mobile body 10.

As shown in FIG. 1, the sensing camera 20 executes front-end SLAM processing using sensor data sensed by its own sensor function (sensor unit 21 in FIG. 2), and transmits the results of the SLAM processing to the server device. Specifically, the sensing camera 20 observes sensor data of objects (obstacles, people, etc.) present in the environment as the mobile body 10 moves through the environment. The sensing camera 20 generates map data relating to an environmental map based on the sensor data. The sensing camera 20 uploads the generated map data to the server device 100. Furthermore, each sensing camera 20 receives map data from the server device 100 that has been error-corrected by the server device 100 based on the map data of the multiple sensing cameras 20, and estimates its own position based on the received map data.

The server device 100 is an example of an information processing device in the embodiment, and is responsible for back-end processing in the multi-robot SLAM system 1, which is divided into a front-end and a back-end. Therefore, in this embodiment, the server device 100 is a computer on the cloud side. On the other hand, the server device 100 may also be a computer on a local network. In other words, the server device 100 is not limited to being either cloud or local, as long as it is a computer that can be accessed via a network.

As shown in FIG. 1, the server device 100 executes back-end SLAM processing using the map data acquired from each sensing camera 20 to generate an integrated map in which the map data is integrated. Specifically, the server device 100 acquires map data related to an environmental map from each of the multiple sensing cameras 20. The server device 100 performs a matching process to match the map data acquired from each of the sensing cameras 20 with each other. The server device 100 also performs an error correction process on the map data based on the result of the matching process and the relative positional relationship between the map data. The server device 100 then distributes the error-corrected map data downstream to each of the multiple sensing cameras 20.

As shown in FIG. 1, the server device 100 may also perform navigation control of the mobile body 10, which is a robot, based on the integrated map, i.e., error-corrected map data. As a result, each sensing camera 20 can move the mobile body 10 mounted on its own device according to the contents of the navigation control.

[2-2. Functional configuration of sensing camera]
2 is a functional block diagram showing an example of a functional configuration of the sensing camera 20 according to the embodiment. The sensing camera 20 has a sensor unit 21.

2, the sensor unit 21 has a distance measurement sensor 211a and a distance measurement sensor signal processing unit 211b. The sensor unit 21 has an RGB camera 212a and an RGB camera signal processing unit 212b. The sensor unit 21 has an IMU 213a and an IMU signal processing unit 213b. Although not shown in FIG. 2, the sensor unit 21 may further have an odometry sensor. The configuration of the sensor unit 21 is not limited to the example of FIG. 2. For example, in SLAM processing, the distance measurement sensor 211a and the RGB camera 212a are required as external sensors (sensors that observe the outside) for creating an environmental map, while an internal sensor for measuring the internal state of the robot may be treated as an optional sensor. In the example of FIG. 2, the internal sensor is an IMU 213a and an odometry sensor, and may be selectively introduced to improve the accuracy of SLAM processing, for example.

Furthermore, according to the example of FIG. 2, the sensing camera 20 has a SLAM recognition processing unit 214, an obstacle recognition processing unit 215, and a person/object recognition processing unit 216.

The distance measurement sensor 211a may include an ultrasonic sensor (Sound Navigation and Ranging: SONAR), a ToF (Time of Flight) sensor, and a LiDAR (Light Detection and Ranging) sensor. For example, the distance measurement sensor 211a obtains information regarding the distance and direction to a measurement point as sensor data.

The ranging sensor signal processing unit 211b converts the sensor data acquired by the ranging sensor 211a into signal data that can be used by each recognition processing unit (SLAM recognition processing unit 214, obstacle recognition processing unit 215, human/object recognition processing unit 216). For example, the SLAM recognition processing unit 214 converts the measurement points into points in the local coordinate system based on the signal data acquired from the ranging sensor signal processing unit 211b. The SLAM recognition processing unit 214 then outputs the coordinate information of the multiple points (point cloud) obtained by the conversion as point cloud information.

The obstacle recognition processing unit 215 recognizes obstacles that exist in the environment in which the mobile body 10 moves, based on the signal data acquired from the distance measurement sensor signal processing unit 211b. The person/object recognition processing unit 216 recognizes people and objects that exist in the environment in which the mobile body 10 moves, based on the signal data acquired from the distance measurement sensor signal processing unit 211b.

The RGB camera 212a captures image data of the environment in which the mobile body 10 on which the device is mounted moves, and obtains this image data as sensor data.

The RGB camera signal processing unit 212b converts the sensor data acquired by the RGB camera 212a into signal data that can be used by each recognition processing unit (SLAM recognition processing unit 214, obstacle recognition processing unit 215, human/object recognition processing unit 216). For example, the SLAM recognition processing unit 214 estimates the position and attitude of the mobile body 10 on which the device is mounted based on the signal data acquired by the RGB camera signal processing unit 212b, and estimates the trajectory of the mobile body 10 moving through the environment from the estimation result.

The obstacle recognition processing unit 215 recognizes obstacles that exist in the environment in which the mobile body 10 moves, based on the signal data acquired from the RGB camera signal processing unit 212b. The person/object recognition processing unit 216 recognizes people and objects that exist in the environment in which the mobile body 10 moves, based on the signal data acquired from the RGB camera signal processing unit 212b.

The IMU 213a acquires the angles and acceleration of three axes that govern the motion of the moving body 10 on which the IMU 213a is mounted as sensor data. Specifically, the IMU outputs data including "time", "frequency", "orientation", "angular velocity", and "acceleration". "Time" indicates the time when the output data was generated. "Frequency" indicates the output frequency of the IMU. "Orientation" indicates the orientation of the moving body 10. "Angular velocity" indicates the angular velocity of the moving body 10. "Acceleration" indicates the acceleration of the moving body 10. Note that the data output by the IMU 213a may differ depending on the type of the IMU 213a. Therefore, the data output by the IMU 213a does not necessarily need to include a combination of "time", "frequency", "orientation", "each speed", and "acceleration".

The IMU signal processing unit 213b converts the sensor data acquired by the IMU 213a into signal data that can be used by each recognition processing unit (SLAM recognition processing unit 214, obstacle recognition processing unit 215, human/object recognition processing unit 216). For example, the SLAM recognition processing unit 214 generates self-position data of the mobile body 10 on which the device is mounted, based on the signal data acquired by the IMU signal processing unit 213b. For example, the SLAM recognition processing unit 214 generates self-position data in a local coordinate system.

The human/object recognition processing unit 216 recognizes people and objects present in the environment in which the mobile body 10 moves based on the signal data acquired from the IMU signal processing unit 213b.

The sensor unit 21 does not necessarily have to include all of the SLAM recognition processing unit 214, the obstacle recognition processing unit 215, and the person/object recognition processing unit 216. For example, an external sensor such as the distance measurement sensor 211a or the RGB camera 212a can substitute for the function of the obstacle recognition processing unit 215, so the sensor unit 21 does not have to include the obstacle recognition processing unit 215.

[2-3. Overview of Multi-Robot SLAM]
Next, an overview of multi-robot SLAM will be described. In multi-robot SLAM, to realize the matching process, a partial map of the SLAM is successively generated by each sensing camera 20. The partial map is called a submap, and is, for example, a lattice map (also called a grid map) that is two-dimensional matrix data.

The submap may be three-dimensional map data. For example, three-dimensional map data includes voxel maps and point cloud maps with added height information. Specifically, in a grid map, the probability of an obstacle's existence is set high for squares on the grid map that correspond to positions where an obstacle is observed to exist by a sensor mounted on a moving object, and the probability of an obstacle's existence is set low for squares where no obstacle is observed to exist.

For example, the sensing camera 20 sequentially generates submaps at any time using sensor data at any time so that matching processing can be realized. In addition, the sensing camera 20 superimposes on the submap representative points (trajectory nodes) of the movement path (trajectory) of the moving object 10 while it moves through the area included in the submap, and the sensor data observed at the positions of the trajectory nodes.

[2-4. Superimposition of sensor data]
3 is a diagram for explaining the process of superimposing sensor data on a submap. When the sensing camera 20 generates the submap sbm, the sensing camera 20 superimposes on the submap sbm the sensor data SD observed at a trajectory node tND on the moving path TR of the moving body 10 (the moving body 10 on which the device is mounted) that has moved through an area included in the submap sbm.

FIG. 3 shows an example in which nine trajectory nodes tND are generated on the travel route TR by the sensing camera 20. In this example, the sensing camera 20 superimposes, for each of the nine trajectory nodes tND, a point cloud of the sensor data SD observed at that trajectory node tND on the submap sbm.

[2-5. Matching process within a single device]
Each sensing camera 20 performs a matching process between the trajectory nodes generated by the own device and a submap as a front-end SLAM process. Specifically, the sensing camera 20 performs a matching process between the trajectory nodes generated by the own device and a self submap, which is a submap generated by the own device based on the trajectory nodes. Hereinafter, such a matching process is referred to as a "self trajectory-self submap matching process."

The sensing camera 20 also performs a matching process between the trajectory nodes generated by its own device and other submaps that are submaps generated by other sensing cameras 20. Hereinafter, this type of matching process will be referred to as "self-trajectory-other submap matching process."

Here, an example of the matching process between the own trajectory and the own sub-map will be described with reference to FIG. 4. FIG. 4 shows an example of the matching process between the own trajectory and the own sub-map by the sensing camera 20-1. For example, the sensing camera 20-1 performs a matching process between the sensor data SD observed at time t1 at a trajectory node tND on the movement route TR of the own device, and the sub-map smb generated by the own device in an observation time range including time t1 (for example, times t1 to t3) based on the sensor data SD.

In this way, in the matching process between self trajectory and self submap, since the sensor data on the trajectory is data for a certain moment, matching must be performed according to the time of the sensor data on the trajectory. This is also true for the matching process between self trajectory and other submaps.

In the example of FIG. 4, the sensing camera 20-1 superimposes the sensor data SD on the submap smb that it has generated based on the sensor data SD. Then, the sensing camera 20-1 matches the relative positional relationship between the submap node sND, which is the node indicating the reference point of the submap sbm, and the trajectory node tND, and rotates or translates the submap sbm and the trajectory node tND relative to each other so that the whole is consistent (so that the position of the obstacle in the submap sbm matches the position of the sensor data SD).

Furthermore, when the sensing camera 20-1 executes the matching process as described above, it generates a self-trajectory-self-submap constraint condition COT1 as a constraint condition indicating the relative positional relationship between the trajectory node tND and the submap sbm based on the matching result. In this way, the sensing camera 20-1 matches the relative positional relationship between its own trajectory position information and its own partial map information, thereby generating a constraint condition (self-trajectory-self-submap constraint condition COT1) indicating the relative positional relationship between them.

Then, the sensing camera 20-1 uploads the submap data, which is information about the submap sbm generated by the sensing camera itself, and the self-trajectory-self-submap constraint condition COT1 to the server device 100. On the other hand, the sensing camera 20-1 does not upload the trajectory node data (sensor data SD), which is information about the trajectory node tND that includes the sensor data SD, to the server device 100, but keeps it within the sensing camera itself.

So far, we have explained the self-trajectory-self-submap matching process using sensing camera 20-1 as an example, but in a multi-robot SLAM, the other sensing cameras 20 also perform the same self-trajectory-self-submap matching process.

For example, the sensing camera 20-2 performs a matching process between the sensor data SD observed at time t1 at the trajectory node tND on the movement route TR of the own device, and the submap smb generated by the own device for an observation time range including time t1 (for example, times t1 to t3) based on the sensor data SD. Then, the sensing camera 20-2 generates the self-trajectory-self-submap constraint condition COT2 as a constraint condition indicating the relative positional relationship between the trajectory node tND and the submap sbm.

Then, the sensing camera 20-2 uploads the submap data, which is information about the submap sbm generated by the camera itself, and the self-trajectory-self-submap constraint condition COT2 to the server device 100. On the other hand, the sensing camera 20-2 does not upload the trajectory node data (sensor data SD), which is information about the trajectory node tND that includes the sensor data SD, to the server device 100, but keeps it within the camera itself.

[2-6. Matching process between multiple devices]
4 has been described regarding the self-trajectory-self submap matching process, which is completed by each sensing camera 20 using only the information generated by the sensing camera itself. However, as described above, each sensing camera 20 also performs the self-trajectory-other submap matching process, which matches the trajectory nodes generated by the sensing camera itself with other submaps generated by other sensing cameras 20. Therefore, an overview of the self-trajectory-other submap matching process will be described in FIG. 5.

According to Figure 5, in multi-robot SLAM, the matching process within a single device (matching process between self-trajectory and self-submap) described in Figure 4 is expanded to matching process between multiple devices (matching process between self-trajectory and other submaps).

The left side of FIG. 5 shows a conceptual diagram of the self-trajectory-self sub-map matching process described in FIG. 4 using the sensing camera 20-1. Meanwhile, the right side of FIG. 5 shows a conceptual diagram of the self-trajectory-other sub-map matching process expanded between the sensing cameras 20-1 and 20-2. According to this example, the sensing camera 20-1 performs matching processing between the sensor data SD observed at time t1 at the trajectory node tND1 on the movement route TR1 of the own device, and the sub-map smb2 generated by the sensing camera 20-2 in an observation time range including time t1 (for example, time t2) based on the sensor data SD.

For example, the sensing camera 20-1 superimposes its own sensor data SD on the submap smb2. Then, the sensing camera 20-1 matches the relative positional relationship between the submap node sND2 indicating the reference point of the submap sbm2 and the trajectory node tND1, and rotates or translates the submap sbm2 and the trajectory node tND1 relative to each other so that the whole is consistent (so that the position of the obstacle in the submap sbm2 matches the position of the sensor data SD).

In addition, based on the matching results of the extended matching process, the sensing camera 20-1 generates self-trajectory-other submap constraint condition COT12 as a constraint condition indicating the relative positional relationship between the trajectory node tND1 and the submap sbm2. In this way, the sensing camera 20-1 matches the relative positional relationship between its own trajectory position information and other partial map information, thereby generating a constraint condition (self-trajectory-other submap constraint condition COT12) indicating these relative positional relationships.

Then, the sensing camera 20-1 uploads the submap data, which is information about the submap sbm1 generated by the sensing camera itself, and the constraint condition COT12 between the self-trajectory and other submaps, to the server device 100. On the other hand, the sensing camera 20-1 does not upload the trajectory node data (sensor data SD), which is information about the trajectory node tND1 that includes the sensor data SD, to the server device 100, but keeps it within the sensing camera itself.

In a multiple robot SLAM system, as shown in Figure 5, the same processing is performed on the sensing camera 20-2 side for the sensing camera 20-1.

For example, the sensing camera 20-2 performs a matching process between the sensor data SD observed at time t1 at the trajectory node tND2 on the movement route TR2 of the own device and the submap smb1 generated by the sensing camera 20-1 for an observation time range including time t1 (for example, time t2) based on the sensor data SD.

For example, the sensing camera 20-2 superimposes its own sensor data SD on the submap smb1. Then, the sensing camera 20-2 matches the relative positional relationship between the submap node sND1 indicating the reference point of the submap sbm1 and the trajectory node tND2, and rotates or translates the submap sbm1 and the trajectory node tND2 relative to each other so that the whole is consistent (so that the position of the obstacle in the submap sbm1 matches the position of the sensor data SD).

Furthermore, based on the matching results of the extended matching process, the sensing camera 20-2 generates self-trajectory-other submap constraint condition COT21 as a constraint condition indicating the relative positional relationship between the trajectory node tND2 and the submap sbm1. In this way, the sensing camera 20-2 matches the relative positional relationship between its own trajectory position information and other partial map information, thereby generating a constraint condition (self-trajectory-other submap constraint condition COT21) indicating these relative positional relationships.

Then, the sensing camera 20-2 uploads the submap data, which is information about the submap sbm2 generated by the camera itself, and the constraint condition COT21 between the self-trajectory and other submaps, to the server device 100. On the other hand, the sensing camera 20-2 does not upload the trajectory node data (sensor data SD), which is information about the trajectory node tND2 that includes the sensor data SD, to the server device 100, but keeps it within the camera itself.

[2-7. Specific examples of multiple robot SLAM processing]
Fig. 6 is a diagram showing a specific example of a multiple robot SLAM process. Fig. 6 shows a specific example of a matching process according to the embodiment and an optimization process based on the result of the matching process. Note that the matching process according to the embodiment is a general term for a matching process between a self trajectory and a self submap and a matching process between a self trajectory and another submap.

FIG. 6 also shows an example of the sensing camera 20-1 performing self-trajectory-self submap matching processing between submap sbmA1 generated at time t1, submap sbmA2 generated at time t2, and submap sbmA3 generated at time t3. FIG. 6 also shows an example of the sensing camera 20-1 performing self-trajectory-other submap matching processing between submap sbmA1, submap sbmA2, and submap sbm3, and submap sbmB2 generated by the sensing camera 20-2 at time t2.

Submap node sNDA1 is the center point defined for submap sbmA1, submap node sNDA2 is the center point defined for submap sbmA2, and submap node sNDA3 is the center point defined for submap sbmA3. Meanwhile, submap node sNDB2 is the center point defined for submap sbmB2.

In this state, as shown in FIG. 6(a), the sensing camera 20-1 can estimate the position and orientation tND and the movement trajectory TR of its own device based on its own sensor data SD, and therefore determines some representative points of the position and orientation tND as trajectory nodes tND and constructs a pose graph PG for each of the submaps sbmA1, sbmA2, and sbmA3.

Similarly, the sensing camera 20-2 can also estimate the position and orientation tND and the movement trajectory TR of its own device based on its own sensor data SD. For this reason, the sensing camera 20-2 also determines some representative points of the position and orientation tND as trajectory nodes tND, and constructs a pose graph PG for the submap sbmB2.

The pose graph PG is a graph structure that represents the movement trajectory TR, with the position and orientation of the object (sensing camera 20-1 and sensing camera 20-2 in the example of FIG. 6) as trajectory nodes tND, and the relative positions between the nodes tND as edges. For example, when the same landmark is observed between image data acquired at different times by the RGB camera 212a of the sensing camera 20-1 (sensing camera 20-2 as well), an edge can be created between the nodes tND that observed this landmark.

In this state, as shown in FIG. 6(b), the sensing camera 20-1 and the sensing camera 20-2 each perform the matching process according to the embodiment.

First, the self trajectory-self submap matching process will be described. According to the example of FIG. 6(b), the sensing camera 20-1 performs a self trajectory-self submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap node sNDA1 of the submap sbmA1. Then, based on the sensing results, the sensing camera 20-1 generates a self trajectory-self submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap sbmA1.

Similarly, the sensing camera 20-1 performs a self-trajectory-self-submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap node sNDA2 of the adjacent submap sbmA2. Then, based on the sensing results, the sensing camera 20-1 generates a self-trajectory-self-submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap sbmA2.

The sensing camera 20-1 also performs a self-trajectory-self-submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA2 and the submap node sNDA2 of the submap sbmA2. Then, based on the sensing results, the sensing camera 20-1 generates a self-trajectory-self-submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA2 and the submap sbmA2.

Similarly, the sensing camera 20-1 performs a self-trajectory-self-submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA2 and the submap node sNDA3 of the adjacent submap sbmA3. Then, based on the sensing results, the sensing camera 20-1 generates a self-trajectory-self-submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA3 and the submap sbmA3.

Next, in the example of FIG. 6(b), the sensing camera 20-2 performs a self-trajectory-self-submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmB2 and the submap node sNDA2 of the submap sbmB2. Then, based on the sensing result, the sensing camera 20-2 generates a self-trajectory-self-submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmB2 and the submap sbmB2.

Next, the matching process between the self-trajectory and other submaps will be described. According to the example of FIG. 6(b), the sensing camera 20-1 can use the submaps sbmA1 and sbmA3 of the adjacent device for the submap sbmB2 generated by the sensing camera 20-2.

For example, the sensing camera 20-1 performs a self-trajectory-other submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap node sNDB2 of the adjacent submap sbmB2. Then, based on the sensing results, the sensing camera 20-1 generates a self-trajectory-other submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA1 and the submap sbmB2.

The sensing camera 20-1 also performs a self-trajectory-other submap matching process that matches the relative positional relationship between each trajectory node tND included in the submap sbmA3 and the submap node sNDB2 of the adjacent submap sbmB2. Then, based on the sensing results, the sensing camera 20-1 generates a self-trajectory-other submap constraint condition COT that indicates the relative positional relationship between each trajectory node tND included in the submap sbmA3 and the submap sbmB2.

In the example of FIG. 6(a), the sensing camera 20-1 uploads the pose graph PG generated by itself, the constraint condition COT generated by itself, and the submap sbm (A1, A2, A3) generated by itself to the server device 100. On the other hand, the sensing camera 20-1 does not upload the sensor data SD associated with the trajectory node tND determined by itself to the server device 100.

The sensing camera 20-2 also uploads the pose graph PG generated by itself, the constraint condition COT generated by itself, and the submap sbmB2 generated by itself to the server device 100. On the other hand, the sensing camera 20-2 does not upload the sensor data SD associated with the trajectory node tND determined by itself to the server device 100.

As shown in FIG. 6(c), the server device 100 executes an optimization process to correct errors between submaps using information uploaded from each of the sensing cameras 20-1 and 20-2. In the optimization process, the errors in the positions of all nodes are corrected so that the relative positional relationships between all submap nodes sND and trajectory nodes tND are consistent.

Specifically, the server device 100 corrects the error between the submaps sbm (A1, A2, A3, B2) based on the submaps sbm (A1, A2, A3, B2) and the constraint condition COT. In this way, the server device 100 corrects the error in the overall map information constructed by combining the partial map information generated by each of the sensing cameras 20 based on the constraint condition COT.

In addition, the server device 100 corrects the error of the overall pose graph that combines each pose graph PG constructed for each of the submaps sbmA1, sbmA2, and sbmA3 with the pose graph PG constructed for the submap sbmB2 based on the constraint condition COT. In this way, the server device 100 corrects the error of the overall trajectory position information constructed by combining the trajectory position information generated by each of the sensing cameras 20 based on the constraint condition COT.

As a result of the optimization process described above, the server device 100 can generate a single environmental map (an accurate map) in which submaps sbmA, sbmA, sbmA3, and sbmB2 are integrated so as to correct the errors. FIG. 6(c) shows the environmental map MP in which submaps sbmA, sbmA, sbmA3, and sbmB2 are integrated through the optimization process.

[2-8. Detailed system configuration example (conventional method)]
Here, a detailed explanation will be given of an example of a conventional functional configuration of a multiple-robot SLAM system 1 with reference to Fig. 7. Fig. 7 is a functional block diagram showing an example of a conventional functional configuration of a multiple-robot SLAM system 1. The multiple-robot SLAM system 1 shown in Fig. 7 includes a moving body 10-1 (robot A) equipped with a sensing camera 20-1, a moving body 10-2 (robot B) equipped with a sensing camera 20-2, and a server device 100.

In FIG. 7, the functional configuration of the sensing camera 20 will be described with a focus on sensing camera 20-1, but sensing camera 20-2 may also have a similar functional configuration. Furthermore, in addition to sensing cameras 20-1 and 20-2, the multiple-robot SLAM system 1 may further include moving bodies 10 equipped with other sensing cameras 20, and all of the sensing cameras 20 may have the same functional configuration as sensing camera 20-1.

[2-8-1. Functional configuration of terminal device]
First, an example of the functional configuration of a sensing camera 20-1 (terminal device) according to a conventional method will be described. As shown in Fig. 7, the sensing camera 20-1 has a sensor unit 21, a pre-processing unit 22, and a SLAM processing unit 23. Although not described here, the sensing camera 20-2 also functions in the same way as the sensing camera 20-1.

(Sensor unit 21)
The sensor unit 21 has various sensor functions described in FIG. 2. For example, the sensor unit 21 has a distance measurement sensor 211a, a distance measurement sensor signal processing unit 211b, an RGB camera 212a, an RGB camera signal processing unit 212b, an IMU 213a, and an IMU signal processing unit 213b. The sensor unit 21 also has a SLAM recognition processing unit 214, an obstacle recognition processing unit 215, and a person/object recognition processing unit 216. The sensor unit 21 may further have an odometry sensor. As described in FIG. 2, the sensor unit 21 does not necessarily have to include all of the distance measurement sensor 211a, the RGB camera 212a, and the IMU 213a. The sensor unit 21 does not necessarily have to include all of the SLAM recognition processing unit 214, the obstacle recognition processing unit 215, and the person/object recognition processing unit 216.

(Pre-processing section 22)
The pre-processing unit 22 executes pre-processing for the SLAM processing performed by the SLAM processing unit 23. For example, the pre-processing unit 22 acquires sensor data SD from the sensor unit 21, and executes pre-processing for the SLAM processing based on the acquired sensor data SD. For example, the pre-processing unit 22 acquires output data of the IMU (e.g., acceleration data), output data of the odometry sensor, and output data of the LiDAR (e.g., point cloud). Note that the point cloud indicates surrounding two-dimensional or three-dimensional point cloud data (coordinates of each point) and data on the time of generation of the point cloud data. For example, the pre-processing includes a process of removing unnecessary features and noise from the point cloud, and a process of segmentation and downsampling.

(SLAM processing unit 23)
The SLAM processing unit 23 corresponds to a front-end processing unit in SLAM. For example, the SLAM processing unit 23 generates a pose graph PG based on the sensor data SD acquired from the sensor unit 21, and uploads pose graph data of the generated pose graph PG to the server device 100.

Note that the matching process requires a pose graph PG, but in the conventional method shown in FIG. 7, the matching process is performed on the server device 100 side. For this reason, for example, the SLAM processing unit 23 of the sensing camera 20-1 does not require the pose graph PG generated by the sensing camera 20-2 in the SLAM process. For this reason, the optimization unit 130 of the server device 100 only needs to transmit to the sensing camera 20-1 the portion of the pose graph PG corrected in the optimization process that corresponds to the sensing camera 20-1, and does not need to transmit the portion that corresponds to the sensing camera 20-2. Also, in this example, the SLAM processing unit 23 of the sensing camera 20-1 synchronizes between the pose graph PG generated by its own device and the pose graph PG corrected on the server device 100 side.

The same can be said about sensing camera 20-2. For example, the SLAM processing unit 23 of sensing camera 20-2 does not need the pose graph PG generated by sensing camera 20-2 in SLAM processing. For this reason, the optimization unit 130 of server device 100 only needs to transmit to sensing camera 20-2 the portion of the pose graph PG corrected in the optimization process that corresponds to sensing camera 20-2, and does not need to transmit the portion that corresponds to sensing camera 20-1. Also, in this example, the SLAM processing unit 23 of sensing camera 20-2 synchronizes between the pose graph PG generated by its own device and the pose graph PG corrected on the server device 100 side.

The SLAM processing unit 23 also estimates the trajectory of its own device based on the sensor data SD acquired from the sensor unit 21. For example, the SLAM processing unit 23 generates trajectory nodes TG based on the output data of the IMU 213 and the output data of the odometry sensor, and uploads the trajectory node data of the generated trajectory nodes TG to the server device 100. Specifically, the SLAM processing unit 23 uploads the sensor data SD corresponding to the trajectory nodes TG to the server device 100.

The SLAM processing unit 23 also generates a submap sbm based on the sensor data SD acquired from the sensor unit 21.

For example, the SLAM processing unit 23 generates a submap sbm based on the output data of the RGB camera 212a and the output data of the LiDAR, and uploads the submap data of the generated submap sbm to the server device 100.

[2-8-2. Functional configuration of information processing device]
Next, an example of the functional configuration of a server device 100 (information processing device) according to a conventional method will be described. As shown in FIG 7, the server device 100 includes a detection unit 110, a matching unit 120, an optimization unit 130, and a visualization unit 140.

(Detection Unit 110)
The detection unit 110 detects a combination of a sub-map sbm to be matched and a trajectory node tNG based on a pose graph PG acquired from each sensing camera 20.

As shown in FIG. 7, in an example of a multiple robot SLAM system 1 including sensing camera 20-1 and sensing camera 20-2, the detection unit 110 detects a combination of matching targets between the submap sbm1 generated by sensing camera 20-1 and trajectory node tND1. The detection unit 110 also detects a combination of matching targets between the submap sbm2 generated by sensing camera 20-2 and trajectory node tND2.

The detection unit 110 also detects a combination of matching targets between the submap sbm1 generated by the sensing camera 20-1 and the trajectory node tND2 generated by the sensing camera 20-2. Similarly, the detection unit 110 detects a combination of matching targets between the submap sbm2 generated by the sensing camera 20-2 and the trajectory node tND1 generated by the sensing camera 20-1.

The detection unit 110 registers the detected combinations in a queue and causes the matching unit 120 to perform scan matching for each combination.

(Matching unit 120)
The matching unit 120 performs scan matching. For example, the matching unit 120 performs scan matching based on a loop closing query. For example, the matching unit 120 performs scan matching based on point cloud information of trajectory node data (sensor data SD) acquired from each sensing camera 20 and submap sbm acquired from each sensing camera 20. For example, the matching unit 120 superimposes point cloud information from which unnecessary features and noise have been deleted by preprocessing such as downsampling on the submap sbm.

In addition, the matching unit 120 generates constraint conditions that indicate the relative positional relationship with the submap sbm as a result of the scan matching.

In the example of FIG. 7, the matching unit 120 matches the relative positional relationship between the trajectory node tND1 generated by the sensing camera 20-1 and the submap node sND1 of the submap sbm1 generated by the sensing camera 20-1, and generates a constraint condition COT1 that indicates this relative positional relationship.

The matching unit 120 also matches the relative positional relationship between the trajectory node tND2 generated by the sensing camera 20-2 and the submap node sND2 of the submap sbm2 generated by the sensing camera 20-2, and generates a constraint condition COT2 that indicates this relative positional relationship.

The matching unit 120 also matches the relative positional relationship between the trajectory node tND1 generated by the sensing camera 20-1 and the submap node sND2 of the submap sbm2 generated by the sensing camera 20-2, and generates a constraint condition COT12 that indicates this relative positional relationship. Similarly, the matching unit 120 matches the relative positional relationship between the trajectory node tND2 generated by the sensing camera 20-2 and the submap node sND1 of the submap sbm1 generated by the sensing camera 20-1, and generates a constraint condition COT21 that indicates this relative positional relationship.

(Optimization Unit 130)
The optimization unit 130 corrects errors in the map data based on each constraint condition COT generated by the matching unit 120. For example, the optimization unit 130 corrects errors in an overall submap sbm constructed by combining the submaps sbm generated by each sensing camera 20. For example, the optimization unit 130 corrects errors in an overall pose graph PG constructed by combining the pose graphs PG generated by each sensing camera 20 based on the constraint condition COT. According to the above example, the matching unit 120 corrects errors based on the constraint condition COT1, the constraint condition COT2, the constraint condition COT12, and the constraint condition COT21 to generate one environmental map MP in which errors have been corrected.

The optimization unit 130 also transmits the corrected overall pose graph PG (overall pose graph PG reflecting the constraint condition COT) to each sensing camera 20. In the example of FIG. 7, the optimization unit 130 transmits only the portion of the corrected pose graph PG that corresponds to the pose graph PG acquired from sensing camera 20-1 to the pose graph PG of sensing camera 20-1, thereby synchronizing the two pose graphs PG. As described above, this processing is performed because, in the conventional method shown in FIG. 7, sensing camera 20-1 does not perform matching processing and does not need to use the pose graph PG of sensing camera 20-2.

In the example of FIG. 7, the optimization unit 130 synchronizes the two pose graphs PG by sending only the portion of the corrected pose graph PG that corresponds to the pose graph PG acquired from the sensing camera 20-2 to the pose graph PG of the sensing camera 20-2. This process is performed because, in the conventional method shown in FIG. 7, the sensing camera 20-2 does not perform matching processing and does not need to use the pose graph PG of the sensing camera 20-1.

(Visualization Unit 140)
The visualization unit 140 generates a rendering map by using the map data error-corrected by the optimization unit 130. For example, the visualization unit 140 performs rendering processing on the submaps sbm error-corrected by the optimization unit 130 to generate one environmental map MP in which the submaps sbm are integrated. Then, the visualization unit 140 may output the environmental map MP to each sensing camera 20.

[2-9. Detailed system configuration example (proposed method)]
Next, a detailed description will be given of an example of the functional configuration of the proposed method according to the present disclosure in a multiple-robot SLAM system 1 with reference to FIG. 8. FIG. 8 is a functional block diagram showing an example of the functional configuration of the proposed method in a multiple-robot SLAM system 1. As in FIG. 7, the multiple-robot SLAM system 1 shown in FIG. 8 includes a moving body 10-1 (robot A) equipped with a sensing camera 20-1, a moving body 10-2 (robot B) equipped with a sensing camera 20-2, and a server device 100. In the following, the multiple-robot SLAM system 1 according to the proposed method will be referred to as a "multiple-robot SLAM system 1A."

In the multiple robot SLAM system 1A according to the proposed method of the present disclosure, matching processing is performed on the sensing camera 20 side. Specifically, the sensing camera 20 performs matching processing between its own sensor data SD and its own submap sbm. The sensing camera 20 also performs matching processing between its own sensor data SD and the submap sbm of other devices. That is, the matching processing described as being performed by the server device 100 in the conventional method of FIG. 7 is performed by each sensing camera 20. In this way, the proposed method employs a configuration in which matching processing is offloaded from the cloud server device 100 to the edge sensing camera 20. As a result, it is believed that the processing load on the server device for SLAM calculations can be appropriately reduced.

[2-9-1. Functional configuration of terminal device]
First, a functional configuration example of a sensing camera 20-1 (terminal device) according to the proposed method will be described. As shown in FIG. 8, the sensing camera 20-1 has a sensor unit 21, a preprocessing unit 22, and a SLAM processing unit 23, as in the conventional method, but further has a detection unit 24 and a matching unit 25. Although not described, the sensing camera 20-2 also functions in the same way as the sensing camera 20-1. Also, in FIG. 8, a new information processing flow is indicated by dotted arrows compared to the conventional method in FIG. 7.

(SLAM processing unit 23)
As in the conventional method, the SLAM processing unit 23 generates a pose graph PG based on the sensor data SD and uploads the pose graph data of the generated pose graph PG to the server device 100. The SLAM processing unit 23 also generates a submap sbm based on the sensor data SD and uploads the submap data of the generated submap sbm to the server device 100. On the other hand, in the proposed method, the SLAM processing unit 23 generates a trajectory node TG based on the sensor data SD, but does not upload the trajectory node data of the generated trajectory node TG to the server device 100. Specifically, the SLAM processing unit does not upload the sensor data SD corresponding to the trajectory node TG to the server device 100.

Here, in the proposed method shown in FIG. 8, the matching process is performed on the sensing camera 20 side. For this reason, for example, the SLAM processing unit 23 of the sensing camera 20-1 requires not only the pose graph PG generated by its own device in the SLAM process, but also the pose graph PG generated by the sensing camera 20-2. For this reason, the optimization unit 130 of the server device 100 transmits to the sensing camera 20-1 not only the part of the pose graph PG corrected in the optimization process that corresponds to the sensing camera 20-1, but also the part that corresponds to the sensing camera 20-2. Also, in this example, the SLAM processing unit 23 of the sensing camera 20-1 synchronizes not only the pose graph PG generated by its own device, but also the pose graphs PG generated by all the sensing cameras 20 included in the multi-robot SLAM system 1A.

The same can be said about sensing camera 20-2. For example, in SLAM processing, the SLAM processing unit 23 of sensing camera 20-2 needs not only the pose graph PG generated by its own device, but also the pose graph PG generated by sensing camera 20-1. For this reason, the optimization unit 130 of server device 100 will transmit to sensing camera 20-2 not only the part of the pose graph PG corrected in the optimization process that corresponds to sensing camera 20-2, but also the part that corresponds to sensing camera 20-1. Also, in this example, the SLAM processing unit 23 of sensing camera 20-2 synchronizes not only the pose graph PG generated by its own device, but also the pose graphs PG generated by all of the sensing cameras 20 included in the multi-robot SLAM system 1A.

(Detection Unit 24)
As shown in Fig. 8, the detection unit 24 detects a submap sbm to be registered in a queue as a matching query and a trajectory node tNG to be registered in a queue as a matching query based on the pose graph PG. For example, the detection unit 24 detects a combination of a submap sbm to be matched and a trajectory node tNG based on the pose graph PG. The pose graph PG used here may include a pose graph PG transmitted from the optimization unit 130. Taking the sensing camera 20-1 as an example, the pose graph PG transmitted from the optimization unit 130 may include not only a pose graph PG generated by the sensing camera itself, but also a pose graph PG generated by the sensing camera 20-2.

For example, the detection unit 24 detects a combination of matching targets between the trajectory node tND1 generated by the own device and the submap sbm1 generated by the own device. The detection unit 24 also detects a combination of matching targets between the trajectory node tND1 generated by the own device and the submap sbm2 generated by the sensing camera 20-2.

The detection unit 24 registers the detected combinations in a queue and causes the matching unit 25 to perform scan matching for each combination.

(Matching unit 25)
The matching unit 25 performs matching processing according to the embodiment. The matching unit 25 performs matching processing sequentially for each combination of the trajectory node tNG and the submap sbm registered in the queue as a matching query. Specifically, the matching unit 25 performs a matching processing between the self trajectory and the self submap, and a matching processing between the self trajectory and another submap.

When performing self-trajectory-other submap matching processing, the matching unit 25 acquires a submap sbm generated by another sensing camera 20 different from the self-device from the server device 100. For example, the matching unit 25 of the sensing camera 20-1 acquires the submap sbm2 generated by the sensing camera 20-2 from the server device 100. On the other hand, the matching unit 25 of the sensing camera 20-2 acquires the submap sbm1 generated by the sensing camera 20-1 from the server device 100.

In this state, in the self-trajectory-self-submap matching process, the matching unit 25 of the sensing camera 20-1 matches the relative positional relationship between the trajectory node tND1 generated by its own device and the submap node sND1 of the submap sbm1 generated by its own device, and generates a self-trajectory-self-submap constraint condition COT1 indicating this relative positional relationship. The matching unit 25 also matches the relative positional relationship between the trajectory node tND1 generated by its own device and the submap node sND2 of the submap sbm2 generated by the sensing camera 20-2, and generates a self-trajectory-other-submap constraint condition COT12 indicating this relative positional relationship.

Similar processing is also performed by the matching unit 25 of the sensing camera 20-2, and as a result, the self-trajectory-self submap matching processing generates the self-trajectory-self submap constraint condition COT2, and the self-trajectory-other submap matching processing generates the self-trajectory-other submap constraint condition COT21.

The matching unit 25 also uploads the constraint condition COT generated by the own device to the server device 100. According to the above example, the matching unit 25 of the sensing camera 20-1 uploads the constraint condition COT1 and the constraint condition COT12 to the server device 100. The matching unit 25 of the sensing camera 20-2 also uploads the constraint condition COT2 and the constraint condition COT21 to the server device 100.

[2-9-2. Functional configuration of information processing device]
Next, a functional configuration example of the server device 100 (information processing device) according to the proposed method will be described. In the proposed method, the matching process is performed by each sensing camera 20. For this reason, the server device 100 does not have the detection unit 110 and the matching unit 120 that are responsible for the matching process in the conventional method, as shown in FIG.

(Optimization Unit 130)
The optimization unit 130 corrects errors in the map data based on the constraint conditions COT generated by each sensing camera 20. The correction process may be the same as the conventional method. That is, the optimization unit 130 corrects errors in the overall submap sbm constructed by combining the submaps sbm generated by each sensing camera 20. For example, the optimization unit 130 corrects errors in the overall pose graph PG constructed by combining the pose graphs PG generated by each sensing camera 20 based on the constraint conditions COT. In the example of FIG. 8, the optimization unit 130 corrects errors based on the constraint conditions COT1, COT2, COT12, and COT21 to generate one environmental map MP in which errors have been corrected.

The optimization unit 130 also transmits the corrected overall pose graph PG (overall pose graph PG reflecting the constraint condition COT) to each sensing camera 20. In the example of FIG. 8, the optimization unit 130 transmits not only the portion of the corrected pose graph PG that corresponds to sensing camera 20-1, but also the portion that corresponds to sensing camera 20-2 to sensing camera 20-1. In this way, the optimization unit 130 causes the SLAM processing unit 23 of sensing camera 20-1 to synchronize the pose graphs PG generated by all of the sensing cameras 20 included in the multiple robot SLAM system 1A. As described above, such processing is performed because, in the proposed method shown in FIG. 8, sensing camera 20-1 needs to use the pose graph PG of sensing camera 20-2 in the matching process.

In the example of FIG. 8, the optimization unit 130 transmits to the sensing camera 20-2 not only the portion of the corrected pose graph PG that corresponds to the sensing camera 20-2, but also the portion that corresponds to the sensing camera 20-1. In this way, the optimization unit 130 causes the SLAM processing unit 23 of the sensing camera 20-2 to synchronize the pose graphs PG generated by all of the sensing cameras 20 included in the multi-robot SLAM system 1A. As described above, such processing is performed because, in the proposed method shown in FIG. 8, the sensing camera 20-2 needs to use the pose graph PG of the sensing camera 20-1 in the matching process.

(Visualization Unit 140)
The visualization unit 140 generates a rendering map by using the map data error-corrected by the optimization unit 130. Then, the visualization unit 140 may output the environmental map MP generated by rendering to each sensing camera 20.

[2-10. Operation procedure]
From here on, we will explain an example of the operational procedure realized by the multiple-robot SLAM system 1A according to the proposed method explained in Fig. 8. Specifically, we will explain the operational procedure in each scene in the multiple-robot SLAM system 1A using Figs. 9 to 12.

[2-10-1. Upload procedure of terminal device]
First, the upload procedure to the server device 100 by the sensing camera 20 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the upload processing procedure executed by the sensing camera 20 in the proposed method. Note that the upload procedure shown in Fig. 9 is performed individually by each sensing camera 20 using only its own device information.

The SLAM processing unit 23 determines whether or not new sensor data SD has been acquired from the sensor unit 21 (step S901). If the SLAM processing unit 23 has not been able to acquire new sensor data SD from the sensor unit 21 (step S901; No), the SLAM processing unit 23 waits until it can acquire new sensor data SD.

When the SLAM processing unit 23 has acquired new sensor data SD from the sensor unit 21 (step S901; Yes), it determines the type of the acquired sensor data SD (step S902). For example, when the SLAM processing unit 23 determines in step S902 that the acquired sensor data SD is output data from an IMU or an odometry sensor, it calculates the amount of movement of the device in a small amount of time (step S903) and returns to step S901.

If the SLAM processing unit 23 determines in step S902 that the acquired sensor data SD is camera image data, it matches (superimposes) the sensor data SD (point cloud information of the sensor data SD) on the submap sbm being created based on the amount of movement in the small amount of time calculated in step S903 (step S904).

The SLAM processing unit 23 also generates a trajectory node TG based on the sensor data SD and adds the generated trajectory node TG to the pose graph PG (step S905).

Then, the SLAM processing unit 23 determines whether the update of the submap sbm being created has been completed (step S906). If the update of the submap sbm has not been completed (step S906; No), the SLAM processing unit 23 returns to step S901.

When the update of the submap sbm is completed (step S906; Yes), the SLAM processing unit 23 adds the updated submap sbm to the pose graph PG (step S907).

Next, the SLAM processing unit 23 uploads the pose graph data and submap data to the server device 100 while retaining the trajectory node data within its own device (step S908).

[2-10-2. Scan matching detection procedure of terminal device]
Next, the scan matching detection procedure by the sensing camera 20 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the scan matching detection procedure executed by the sensing camera 20 in the proposed method. Note that the scan matching detection procedure shown in Fig. 10 is performed individually by each sensing camera 20.

First, the detection unit 24 determines whether data generated by the own device has been added to the pose graph PG of the own device (step S1001). If data generated by the own device has not been added to the pose graph PG of the own device (step S1001; No), the detection unit 24 waits until data is added.

On the other hand, if data generated by the own device is added to the pose graph PG of the own device (step S1001; Yes), the detection unit 24 determines the type of added data (step S1002). For example, if the added data is trajectory node data in step S1002, the detection unit 24 detects submaps sbm that are to be matched with the trajectory node TG generated by the own device, from among the submaps sbm previously generated by each sensing camera 20, including the own device (step S1003).

Then, the detection unit 24 lists combinations of the trajectory nodes TG and the submaps sbm to be matched (step S1004), and adds the listed combinations to the queue of the device (step S1005). For example, in step S1004, the detection unit 24 may list combinations of adjacent nodes.

If the added data is submap data in step S1002, the detection unit 24 detects trajectory nodes TG that are to be matched with the submap sbm generated by the own device from among the trajectory nodes TG previously generated by the own device (step S1006).

Then, the detection unit 24 lists combinations of the trajectory nodes TG and the submaps sbm to be matched (step S1007), and adds the listed combinations to the queue of the device (step S1008). For example, in step S1007, the detection unit 24 may list combinations of adjacent nodes.

[2-10-3. Scan matching processing procedure of terminal device]
Next, the scan matching process performed by the sensing camera 20 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the scan matching process performed by the sensing camera 20 in the proposed method. Note that the scan matching process shown in Fig. 11 is performed individually by each sensing camera 20.

First, the matching unit 25 determines whether an item for scan matching (a combination list of matching targets) has been added to the queue of the device itself (step S1101). While an item for scan matching has not been added to the queue (step S1101; No), the matching unit 25 waits until an item for scan matching is added to the queue.

On the other hand, when an item for scan matching is added to the queue (step S1101; Yes), the matching unit 25 determines whether or not the matching unit 25 has actual data for any of the submaps sbm listed as matching targets that were generated by a sensing camera 20 other than the own device (step S1102).

If the matching unit 25 does not have actual data of the submap sbm generated by the other sensing camera 20 (step S1102; No), it acquires the actual data of the submap sbm generated by the other sensing camera from the server device 100 (step S1103).

Then, the matching unit 25 executes scan matching (step S1104). Specifically, the matching unit 25 executes a matching process between the self trajectory and the self submap, and a matching process between the self trajectory and another submap. Note that if the matching unit 25 has actual data of a submap sbm generated by another sensing camera 20 (step S1102; Yes), it may skip step S1103.

The matching unit 25 generates a constraint condition COT indicating the relative positional relationship between the trajectory node tND and the submap sbm based on the matching result (step S1105). Then, the matching unit 25 uploads the constraint condition COT to the server device 100 (step S1106).

[2-10-4. Optimization process procedure for information processing device]
Next, the optimization process performed by the server device 100 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the optimization process performed by the server device 100 in the proposed method.

The optimization unit 130 determines whether or not the constraint conditions COT have been sufficiently accumulated in response to the constraint conditions COT being uploaded from each sensing camera 20 (step S1201). If the amount of accumulated constraint conditions COT is insufficient (step S1201; No), the optimization unit 130 waits until the constraint conditions COT have been sufficiently accumulated.

On the other hand, if the accumulated amount of the constraint condition COT is sufficient (step S1201; Yes), the optimization unit 130 executes an optimization process to correct the error between the sub-maps sbm uploaded from each sensing camera 20 based on the constraint condition COT (step S1202).

Next, the optimization unit 130 reflects the results of the optimization process in the pose graph PG (step S1203). Specifically, the optimization unit 130 transmits to each sensing camera 20 a corrected overall pose graph PG in which errors in the overall pose graph PG constructed by combining the pose graphs PG generated by each sensing camera 20 have been corrected. As a result, each sensing camera 20 reflects the pose graphs PG generated by the other sensing cameras 20 in its own pose graph PG. In other words, each sensing camera 20 synchronizes between the pose graph PG generated by its own device and the pose graphs PG generated by the other sensing cameras 20.

The optimization unit 130 also generates a rendering map (environment map) using the map data that has been error-corrected in the optimization process (step S1204). The optimization unit 130 may output the rendering map to each sensing camera 20.

[3. Modification 1]
Next, we will explain modified examples of the multiple-robot SLAM system 1. We have explained that the multiple-robot SLAM system 1A according to the proposed method of the present disclosure employs a configuration in which matching processing is offloaded from the cloud server device 100 to the edge sensing camera 20.

However, the sensing camera 20 often has inferior processing capabilities compared to the server device 100, and there may also be differences in processing capabilities between sensing cameras 20 depending on the type. In this way, if the sensing camera 20 has low processing capabilities, there is a problem that, for example, it cannot complete all matching processes within a period of time, and it takes a long time to obtain the results of self-position estimation. Therefore, in Variation 1 related to the proposed method of the present disclosure, a request unit 26 is newly introduced to each sensing camera 20 as a new component for solving such problems. Therefore, the functional configuration of Variation 1 related to the proposed method of the present disclosure will first be described below.

[3-1. Detailed configuration example of the system in the first modification]
Fig. 13 is a functional block diagram showing an example of the functional configuration of Variation 1 in a multiple-robot SLAM system 1. As with Fig. 8, the multiple-robot SLAM system 1 shown in Fig. 13 includes a moving body 10-1 (robot A) equipped with a sensing camera 20-1, a moving body 10-2 (robot B) equipped with a sensing camera 20-2, and a server device 100. Hereinafter, the multiple-robot SLAM system 1 according to Variation 1 of the proposed method will be referred to as a "multiple-robot SLAM system 1B."

The multiple robot SLAM system 1B according to the first variation of the proposed method is similar to the multiple robot SLAM system 1A in that the matching process is performed on the sensing camera 20 side. Therefore, the multiple robot SLAM system 1B can also appropriately reduce the processing load on the server device for SLAM calculations, but can also produce a new effect. Specifically, the multiple robot SLAM system 1B can complete the matching process within the allowable time even if a sensing camera 20 with a performance that makes it impossible to complete the matching process within the allowable time is introduced.

[3-1-1. Functional configuration of terminal device]
First, a functional configuration example of the sensing camera 20-1 (terminal device) according to the first modification will be described. As shown in Fig. 13, the sensing camera 20-1 has a sensor unit 21, a preprocessing unit 22, a SLAM processing unit 23, a detection unit 24, and a matching unit 25, as in the proposed method, but also has a request unit 26. Although not described here, the sensing camera 20-2 also functions in the same way as the sensing camera 20-1. Also, in Fig. 13, a new information processing flow is indicated by dotted arrows compared to the proposed method of Fig. 8.

(Request Unit 26)
The request unit 26 performs a process dispatcher process of allocating (requesting) matching processes between the device itself and an external device. Specifically, when an unprocessed matching process occurs among the matching processes for matching relative positional relationships that is predicted not to be calculated within a predetermined time, the request unit 26 requests an external device different from the device itself to perform the unprocessed matching process.

For example, when the detection unit 24 registers a combination of matching targets in a queue as a matching query, the request unit 26 extracts unprocessed matching processes that are predicted to be missed from the matching processes within a predetermined time period. Then, the request unit 26 requests the server device 100 to perform the extracted matching processes.

For example, the request unit 26 can perform the following processing when making a request to the server device 100. For example, as shown in FIG. 13, the request unit 26 registers the combination (combination of trajectory node tNG and submap sbm) that is the matching target in the unprocessed matching process in a queue of the server device 100. The request unit 26 also uploads trajectory node data that has not been uploaded in the multiple robot SLAM system 1A to the server device 100. Specifically, the request unit 26 uploads the trajectory node data (sensor data SD corresponding to trajectory node TG) of the trajectory node TG generated by the SLAM processing unit 23 to the server device 100.

In this way, in the multiple robot SLAM system 1B, part of the matching process that would normally be performed by the sensing camera 20 is performed by the server device 100. As a result, in the multiple robot SLAM system 1B, the matching unit 120 that was excluded from the server device 100 is reinstated.

In addition, if there is no unprocessed matching process that is predicted to be unable to be calculated within the specified time among the matching processes that match relative positional relationships, the request unit 26 inputs the original combination (the combination of the trajectory node tNG and the submap sbm) that is the matching target in the matching process to the matching unit 25, thereby executing sequential scan matching.

[3-1-2. Functional configuration of information processing device]
Next, a functional configuration example of the server device 100 (information processing device) according to Modification 1 of the proposed method will be described. As described above, in Modification 1, part of the matching process that should be performed by the sensing camera 20 is performed by the server device 100. Therefore, compared to the example of Fig. 8, the server device 100 has a matching unit 120. The server device 100 also has a queue.

(Matching unit 120)
When the matching unit 120 receives a request for matching processing from any of the sensing cameras 20 (request unit 26), it executes the unprocessed matching processing corresponding to the request. Specifically, the matching unit 120 extracts trajectory node data and submaps sbm to be used for the unprocessed matching processing from the storage unit of its own device based on a combination of matching targets registered in the queue of its own device by the requesting sensing camera (request unit 26). Then, the matching unit 120 matches the relative positional relationship between the trajectory node tND included in the extracted trajectory node data and the submap node sND of the extracted submap sbm, and generates a constraint condition COT indicating this relative positional relationship. In other words, the matching unit 120 executes the matching processing that could not be completed by the requesting sensing camera 20.

[3-2. Operation procedure]
From here on, an example of the operational procedure implemented by the multiple-robot SLAM system 1B according to the first modification example described with reference to Fig. 13 will be described. Specifically, the operational procedure in each scene in the multiple-robot SLAM system 1B will be described with reference to Figs. 14 to 16.

[3-2-1. Pre-processing procedure for requests]
First, a pre-processing procedure performed by the sensing camera 20 when requesting matching processing from the server device 100 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the pre-processing procedure related to the request for matching processing. Note that the pre-processing procedure shown in Fig. 14 is performed individually by each sensing camera 20.

First, the request unit 26 determines whether an item for scan matching (a combination list of matching targets) has been added to the queue of the device itself (step S1401). While an item for scan matching has not been added to the queue (step S1401; No), the request unit 26 waits until an item for scan matching is added to the queue.

On the other hand, when an item for scan matching has been added to the queue (step S1401; Yes), the request unit 26 determines whether any of the matching processing units (matching unit 25) of the own device has become available for processing (step S1402). When there is no matching processing unit that has become available for processing (step S1402; No), the request unit 26 waits until one of the matching processing units becomes available for processing.

On the other hand, if any of the matching processing units has an available process (step S1402; Yes), the request unit 26 assigns the matching task to the matching unit 25 that has an available process (step S1403).

Then, the request unit 26 measures the matching processing speed for each matching processing unit to which the matching task is assigned (step S1404).

[3-2-2. Request Processing Procedure]
Next, a request processing procedure in which the sensing camera 20 requests the server device 100 to perform matching processing will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the request processing procedure in which the matching processing is requested. Note that the request processing procedure shown in Fig. 15 is performed individually by each sensing camera 20. Also, the request processing procedure may be performed in parallel with the pre-processing procedure in Fig. 14, or may be performed sequentially with respect to the pre-processing procedure in Fig. 14.

The request unit 26 predicts the processing time required to complete matching for each matching processing unit assigned the matching task based on the queue size of the queue of the own device and the matching processing speed obtained as the measurement result in step S1404 (step S1501). For example, the request unit 26 may perform this process at regular intervals.

The request unit 26 determines whether the processing time is within the allowable range (step S1502). If the processing time is within the allowable range (step S1502; Yes), the request unit 26 returns to step S1501.

On the other hand, if the processing time is not within the allowable range (step S1502; No), the request unit 26 extracts a matching process to be requested from the server device 100 (step S1503). For example, the request unit 26 extracts, based on the processing time, a matching process that is registered in the queue of the own device and is predicted to fall outside the allowable range from among the matching processes of the matching processing unit to which the matching task has been assigned.

The request unit 26 then uploads the extracted trajectory node tND required for the matching process to the server device 100 (step S1504). The request unit 26 also deletes the matching task corresponding to the extracted matching process from the queue of the server device (step S1505).

[3-2-3. Scan matching processing procedure of information processing device]
Next, a procedure of the server device which operates upon receiving a request for matching processing will be described below. Fig. 16 is a flow chart showing the procedure of the matching processing which the server device 100 executes upon receiving a request for matching processing.

First, the matching unit 120 determines whether an item for scan matching (a list of combinations to be matched) has been added to the queue of the device itself (step S1601). For example, the matching unit 120 determines whether a combination to be matched in an unprocessed matching process that has been determined to be difficult to process by the sensing camera 20 because it has not been processed within the allowable range has been added to the queue. In other words, the matching unit 120 determines whether a request for an unprocessed matching process has been received from the sensing camera 20.

The matching unit 120 executes scan matching (step S1602). Specifically, the matching unit 120 extracts the trajectory node tND and the submap sbm corresponding to the combination of matching targets added to the queue of the own device, and matches the relative positional relationship between the extracted trajectory node tND and the submap sbm.

Then, the matching unit 120 generates a constraint condition COT indicating the relative positional relationship between the trajectory node tND and the submap sbm based on the matching result (step S1603), and saves the generated constraint condition COT (step S1604).

Note that, although not shown in FIG. 16, when the constraint condition COT is saved by the matching unit 120, the optimization unit 130 executes an optimization process to correct the error between the constraint condition COT generated by the device itself in the matching process in response to a request from the sensing camera 20 and the sub-map sbm uploaded from each sensing camera 20, based on the COT uploaded from the sensing camera 20.

The optimization unit 130 also reflects the results of the optimization process in the pose graph PG (step S1203). Specifically, the optimization unit 130 transmits to each sensing camera 20 the corrected overall pose graph PG in which the errors in the overall pose graph PG, which is constructed by combining the pose graphs PG generated by each sensing camera 20, have been corrected.

[4. Modification 2]
The matching process to be performed by one sensing camera 20 may be allocated to multiple computational resources on the edge side. In this way, the matching process to be performed by one sensing camera 20 may be distributed on the edge side, and if an unprocessed matching process occurs that cannot be handled within the allowable range, the unprocessed matching process may be requested to the server device 100. That is, in the modified example 2 according to the proposed method of the present disclosure, one sensing camera 20 may request another sensing camera 20 to perform a part of the matching process to be performed by itself, and if an unprocessed matching process occurs that cannot be handled within the allowable range, the unprocessed matching process is requested to the server device 100.

[4-1. Detailed configuration example of the system in the second modification]
Fig. 17 is a functional block diagram showing an example of the functional configuration of Variation 2 in a multiple-robot SLAM system 1. As with Fig. 13, the multiple-robot SLAM system 1 shown in Fig. 17 includes a moving body 10-1 (robot A) equipped with a sensing camera 20-1, a moving body 10-2 (robot B) equipped with a sensing camera 20-2, and a server device 100. Hereinafter, the multiple-robot SLAM system 1 according to Variation 2 of the proposed method will be referred to as a "multiple-robot SLAM system 1C."

In the example of FIG. 17, the sensing camera 20-2 is treated as a sub-machine of the sensing camera 20-1, and is configured to perform part of the matching process that should be performed by the sensing camera 20-1 in response to a request from the sensing camera 20-1.

In the example of FIG. 17, the sensing camera 20-2 has a matching unit 25-2 that performs matching processing on behalf of the sensing camera 20-1 in response to a request from the sensing camera 20-1.

[4-1-1. Functional configuration of terminal device]
First, an example of the functional configuration of the sensing camera 20-1 according to the modified example 2 will be described. As shown in Fig. 17, the sensing camera 20-1 has a sensor unit 21, a pre-processing unit 22, a SLAM processing unit 23, a detection unit 24, a matching unit 25, and a request unit 26, just like the modified example 1. Although not described here, the sensing camera 20-2 also functions in the same way as the sensing camera 20-1. Also, in Fig. 17, a new flow of information processing is indicated by dotted arrows, compared to the modified example 1 in Fig. 16.

When an unprocessed matching process occurs among the matching processes for matching relative positional relationships that is predicted to be unable to be calculated within a predetermined time, the request unit 26 requests the server device 100 to perform the unprocessed matching process. For example, the request unit 26 aims to complete all matching processes within an acceptable range by requesting the sensing camera 20-2 linked as a sub-device to perform part of the matching process that should be performed by the request unit 26. However, when an unprocessed matching process occurs that cannot be completed within the acceptable range despite the request for part of the matching process being from the sensing camera 20-2, the request unit 26 requests the server device 100 to perform the unprocessed matching process.

In this way, in the multiple robot SLAM system 1C, of the matching processes that should normally be performed by multiple computational resources on the edge side (sensing cameras 20-1 and 20-2 in the example of FIG. 17), the unprocessed matching processes that the multiple computational resources cannot handle within the allowable range are performed by the server device 100. For this reason, in the multiple robot SLAM system 1C as well, the matching unit 120 that was excluded from the server device 100 is reinstated.

The matching unit 25-2 of the sensing camera 20-2 also performs scan matching for each combination (combination of trajectory node tNG and submap sbm) that is the target of matching in the matching process requested by the request unit 26 of the sensing camera 20-1. The matching unit 25-2 also uploads the constraint condition COT generated based on the results of the matching process to the server device 100.

[4-1-2. Functional configuration of information processing device]
Next, a functional configuration example of the server device 100 according to the modified example 2 of the proposed method will be described. As described above, in the modified example 2, part of the matching process to be performed among a plurality of edge-side computing resources is performed by the server device 100. Therefore, compared to the example of Fig. 8, the server device 100 has a matching unit 120. The server device 100 also has a queue.

(Matching unit 120)
The matching unit 120 performs unprocessed matching processes that are to be performed among multiple edge-side computing resources and that cannot be handled within the allowable range by the multiple computing resources. For example, the matching unit 120 extracts trajectory node data and submaps sbm to be used for the unprocessed matching processes from the storage unit of the device itself based on a combination of matching targets registered in the queue of the device itself by the requesting sensing camera (request unit 26). Then, the matching unit 120 matches the relative positional relationship between the trajectory node tND included in the extracted trajectory node data and the submap node sND of the extracted submap sbm, and generates a constraint condition COT that indicates this relative positional relationship.

[4-2. Operation procedure]
From here, an example of an operational procedure realized by the multiple-robot SLAM system 1C according to the modified example 2 described in FIG. 17 will be described. Specifically, a request processing procedure in which the sensing camera 20 requests the server device 100 to perform matching processing will be described. FIG. 18 is a flowchart showing the request processing procedure for requesting matching processing. Note that the request processing procedure shown in FIG. 18 is performed individually by each sensing camera 20. The request processing procedure shown in FIG. 18 is a modification of the request processing procedure described in FIG. 15 in accordance with the multiple-robot SLAM system 1C according to the modified example 2.

The request unit 26 predicts the processing time required to complete matching for each matching processing unit assigned the matching task based on the queue size of the queue of the own device and the matching processing speed obtained as the measurement result in step S1404 (step S1801). For example, the request unit 26 may perform this process at regular intervals.

The request unit 26 determines whether the processing time is within the allowable range (step S1802). If the processing time is within the allowable range (step S1802; Yes), the request unit 26 determines whether an entity executing the matching process (matching processing unit) exists on the edge side other than the own device (step S1803). For example, if a configuration is adopted in which the matching process that should be performed by the own device is distributed among multiple computational resources including the own device, the request unit 26 can determine that an entity executing the matching process exists on the edge side other than the own device.

If the request unit 26 determines that there is another executing entity for the matching process other than its own device (step S1803; Yes), it excludes the other executing entity from the executing entities for performing the matching process (step S1804). In other words, if there is another executing entity for the matching process on the edge side other than its own device but its own device alone can complete all of the matching process within the allowable range, the request unit 26 does not request the matching process from another executing entity (matching processing unit) on the edge side.

If the request unit 26 determines that no other device than its own device is executing the matching process (step S1803; No), the request unit 26 returns to step S1801.

If the processing time is not within the allowable range (step S1802; No), the request unit 26 determines whether there is an entity executing the matching process on the edge side other than the device itself (step S1805).

If the request unit 26 determines that there are no other execution subjects of the matching process other than its own device (step S1805; No), it excludes other execution subjects that exist on the edge side from the execution subjects that perform the matching process (step S1806).

If the request unit 26 determines that the matching process is being performed by another entity besides its own device (step S1805; Yes), it causes the matching processes that can be processed within the allowable range to be performed among multiple computational resources including its own device, while extracting matching processes to be requested to the server device 100, i.e., unprocessed matching processes that cannot be processed within the allowable range by the multiple computational resources (step S1807). For example, the request unit 26 extracts matching processes that are registered in the queue of its own device and that are predicted to fall outside the allowable range from among the matching processes of the matching unit 25 that has been assigned a matching task, based on the processing time.

The request unit 26 then uploads the extracted trajectory node tND required for the matching process to the server device 100 (step S1808). The request unit 26 also deletes the matching task corresponding to the extracted matching process from the queue of the server device (step S1809).

5. Hardware Configuration
A hardware configuration example of a computer corresponding to the device according to each embodiment described above (e.g., the sensing camera 20, the server device 100) will be described with reference to Fig. 19. Fig. 19 is a block diagram showing a hardware configuration example of a computer corresponding to the device according to the embodiment of the present disclosure. Note that Fig. 19 shows an example of the hardware configuration of a computer corresponding to the device according to each embodiment, and is not necessarily limited to the configuration shown in Fig. 19.

As shown in FIG. 19, computer 1000 has a CPU (Central Processing Unit) 1100, RAM (Random Access Memory) 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input/output interface 1600. Each part of computer 1000 is connected by a bus 1050.

The CPU 1100 operates based on the programs stored in the ROM 1300 or the HDD 1400, and controls each component. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200, and executes processes corresponding to the various programs.

The ROM 1300 stores boot programs such as the Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000.

HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 records program data 1450. Program data 1450 is an example of an information processing program for realizing an information processing method according to an embodiment of the present disclosure, and data used by such information processing program.

The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input/output interface 1600. The input/output interface 1600 may also function as a media interface that reads programs and the like recorded on a specific recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase change rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.

For example, when the computer 1000 functions as the sensing camera 20, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200, thereby implementing the various processing functions executed by the processes shown in FIG. 8 and the like. In other words, the CPU 1100 and the RAM 1200, etc., work together with the software (the information processing program loaded onto the RAM 1200) to implement the information processing method by the terminal device according to an embodiment of the present disclosure.

Furthermore, when the computer 1000 functions as the server device 100, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200, thereby implementing various processing functions executed by each process shown in FIG. 8 and the like. In other words, the CPU 1100 and the RAM 1200, etc., work together with software (an information processing program loaded onto the RAM 1200) to implement an information processing method by an information processing device according to an embodiment of the present disclosure.

6. Summary
Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure. In addition, the components of the embodiments and modifications may be appropriately combined.

Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be present.

The present disclosure can also be configured as follows.
(1)
An information processing device,
providing, to one or more terminal devices, other partial map information generated by other terminal devices different from the terminal devices;
the terminal device acquires from the terminal device information on a matching result obtained by matching a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device moves in an area included in the partial map information generated by the terminal device and the other partial map information, and the partial map information;
performing an optimization process for correcting an error in overall trajectory position information constructed by combining the trajectory position information of the terminal device based on information of the matching result;
A method for processing information.
(2)
The information processing device,
acquiring, as information of the matching result, a constraint condition that indicates the relative positional relationship and that is generated by matching the relative positional relationship;
The information processing method according to (1), further comprising: executing a process.
(3)
The information processing device,
correcting an error in the entire trajectory position information based on the constraint condition;
The information processing method according to (2), further comprising: executing a process.
(4)
The information processing device,
correcting an error in overall map information constructed by combining the partial map information generated by each of the terminal devices based on the constraint condition;
outputting the corrected overall map information to each of the terminal devices;
The information processing method according to (2), further comprising: executing a process.
(5)
The information processing device,
For each of the terminal devices, the trajectory position information generated by the terminal device is synchronized with the corrected overall trajectory position information.
The information processing method according to (3), further comprising: executing a process.
(6)
The terminal device
Acquire other partial map information generated by a terminal device different from the own device from the information processing device on the cloud side;
Based on trajectory position information indicating a position on a trajectory along which the own device moves in an area included in the partial map information generated by the own device, point cloud information indicating a position of an object within the partial map information, and the other partial map information, a relative positional relationship between the trajectory position information and the other partial map information is matched;
uploading the trajectory position information, information on a matching result of a relative positional relationship between the trajectory position information and the other partial map information, and the partial map information to the information processing device that generates an environmental map corresponding to the partial map;
A method for processing information.
(7)
The terminal device,
a trajectory node including the point cloud information and indicating a representative point among the positions on the trajectory included in the trajectory position information, and a partial map node indicating a reference point of the other partial map information, based on which a relative positional relationship between the trajectory node and the partial map node is matched, thereby generating a constraint condition indicating the relative positional relationship as information of the matching result;
uploading the constraints to the information processing device;
The information processing method according to (6), further comprising: executing a process.
(8)
The terminal device,
acquiring from the information processing device, overall trajectory position information constructed by combining the trajectory position information generated by the own device and the trajectory position information generated by the other terminal device, the overall trajectory position information being corrected by the information processing device based on the constraint condition generated by the own device and the constraint condition generated by the other terminal device;
estimating a self-position in overall map information constructed by combining the partial map information based on the overall trajectory position information;
The information processing method according to (7), further comprising: executing a process.
(9)
The terminal device,
When an unprocessed matching process occurs among the matching processes for matching the relative positional relationship, the unprocessed matching process is requested to be completed by the information processing device.
The information processing method according to (6), further comprising: executing a process.
(10)
The terminal device,
requesting a part of a matching process for matching the relative positional relationship to another terminal device different from the own device;
Even if a part of the matching process has been requested to the other terminal device, when an unprocessed matching process occurs among the matching processes for matching the relative positional relationship, for which calculation cannot be completed within the predetermined time, the unprocessed matching process is requested to the information processing device.
The information processing method according to (9), further comprising: executing processing.
(11)
a providing unit that provides, to one or more terminal devices, other partial map information generated by other terminal devices different from the terminal devices;
an acquisition unit that acquires, from the terminal device, information on a matching result obtained by matching a relative positional relationship between trajectory position information indicating a position on a trajectory along which the terminal device moves within an area included in the partial map information generated by the terminal device and the other partial map information based on the trajectory position information and the other partial map information, and the partial map information;
and an optimization processing unit that corrects an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices based on information of the matching result.
(12)
A terminal device,
an acquisition unit that acquires other partial map information generated by a terminal device other than the own device from an information processing device on a cloud side;
a matching unit that matches a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the device moves in an area included in the partial map information generated by the device, point cloud information indicating a position of an object within the partial map information, and the other partial map information;
a processing unit that uploads the trajectory position information, information on a matching result of a relative position relationship between the trajectory position information and the other partial map information, and the partial map information to an information processing device that generates an environmental map corresponding to the partial map.
(13)
An information processing device and a plurality of terminal devices;
A system comprising:
The information processing device includes:
a providing unit for providing the terminal device with other partial map information generated by another terminal device different from the terminal device;
an acquisition unit that acquires, from the terminal device, information on a matching result obtained by matching a relative positional relationship between trajectory position information indicating a position on a trajectory along which the terminal device moves within an area included in the partial map information generated by the terminal device and the other partial map information, and the partial map information; and
an optimization processing unit that corrects an error in overall trajectory position information constructed by combining the trajectory position information of the terminal device based on information of the matching result,
The terminal device
an acquisition unit that acquires the other partial map information generated by another terminal device different from the own device from the information processing device on the cloud side;
a matching unit that matches a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the device moves in an area included in the partial map information generated by the device, point cloud information indicating a position of an object within the partial map information, and the other partial map information;
a processing unit that uploads the trajectory position information, information on a matching result of a relative position relationship between the trajectory position information and the other partial map information, and the partial map information to the information processing device that generates an environmental map corresponding to the partial map.

20 Sensing camera 21 Sensor unit 22 Preprocessing unit 23 SLAM processing unit 24 Detection unit 25 Matching unit 26 Request unit 100 Server device 110 Detection unit 120 Matching unit 130 Optimization unit 140 Visualization unit

Claims (13)

An information processing device,
providing, to one or more terminal devices, other partial map information generated by other terminal devices different from the terminal devices;
the terminal device acquires from the terminal device information on a matching result obtained by matching a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the terminal device moves in an area included in the partial map information generated by the terminal device and the other partial map information, and the partial map information;
performing an optimization process for correcting an error in overall trajectory position information constructed by combining the trajectory position information of the terminal device based on information of the matching result;
A method for processing information. The information processing device,
acquiring, as information of the matching result, a constraint condition that indicates the relative positional relationship and that is generated by matching the relative positional relationship;
The information processing method according to claim 1 , further comprising: executing a process. The information processing device,
correcting an error in the entire trajectory position information based on the constraint condition;
The information processing method according to claim 2 , further comprising: executing a process. The information processing device,
correcting an error in overall map information constructed by combining the partial map information generated by each of the terminal devices based on the constraint condition;
outputting the corrected overall map information to each of the terminal devices;
The information processing method according to claim 2 , further comprising: executing a process. The information processing device,
For each of the terminal devices, the trajectory position information generated by the terminal device is synchronized with the corrected overall trajectory position information.
The information processing method according to claim 3 , further comprising the step of: executing a process. The terminal device
Acquire other partial map information generated by a terminal device different from the own device from the information processing device on the cloud side;
Based on trajectory position information indicating a position on a trajectory along which the own device moves in an area included in the partial map information generated by the own device, point cloud information indicating a position of an object within the partial map information, and the other partial map information, a relative positional relationship between the trajectory position information and the other partial map information is matched;
uploading the trajectory position information, information on a matching result of a relative positional relationship between the trajectory position information and the other partial map information, and the partial map information to the information processing device that generates an environmental map corresponding to the partial map;
A method for processing information. The terminal device,
a trajectory node including the point cloud information and indicating a representative point among the positions on the trajectory included in the trajectory position information, and a partial map node indicating a reference point of the other partial map information, based on which a relative positional relationship between the trajectory node and the partial map node is matched, thereby generating a constraint condition indicating the relative positional relationship as information of the matching result;
uploading the constraints to the information processing device;
The information processing method according to claim 6 , further comprising: executing a process. The terminal device,
acquiring from the information processing device, overall trajectory position information constructed by combining the trajectory position information generated by the own device and the trajectory position information generated by the other terminal device, the overall trajectory position information being corrected by the information processing device based on the constraint condition generated by the own device and the constraint condition generated by the other terminal device;
estimating a self-position in overall map information constructed by combining the partial map information based on the overall trajectory position information;
The information processing method according to claim 7 , further comprising: executing a process. The terminal device,
When an unprocessed matching process occurs among the matching processes for matching the relative positional relationship, the unprocessed matching process is requested to be completed by the information processing device when the unprocessed matching process cannot be completed within a predetermined time.
The information processing method according to claim 6 , further comprising: executing a process. The terminal device,
requesting a part of a matching process for matching the relative positional relationship to another terminal device different from the own device;
Even if a part of the matching process has been requested to the other terminal device, when an unprocessed matching process occurs among the matching processes for matching the relative positional relationship, for which calculation cannot be completed within the predetermined time, the unprocessed matching process is requested to the information processing device.
The information processing method according to claim 9 , further comprising: executing a process. a providing unit that provides, to one or more terminal devices, other partial map information generated by other terminal devices different from the terminal devices;
an acquisition unit that acquires, from the terminal device, information on a matching result obtained by matching a relative positional relationship between trajectory position information indicating a position on a trajectory along which the terminal device moves within an area included in the partial map information generated by the terminal device and the other partial map information based on the trajectory position information and the other partial map information, and the partial map information;
and an optimization processing unit that corrects an error in overall trajectory position information constructed by combining the trajectory position information of the terminal devices based on information of the matching result. A terminal device,
an acquisition unit that acquires other partial map information generated by a terminal device other than the own device from an information processing device on the cloud side;
a matching unit that matches a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the device moves in an area included in the partial map information generated by the device, point cloud information indicating a position of an object within the partial map information, and the other partial map information;
a processing unit that uploads the trajectory position information, information on a matching result of a relative position relationship between the trajectory position information and the other partial map information, and the partial map information to an information processing device that generates an environmental map corresponding to the partial map. An information processing device and a plurality of terminal devices;
A system comprising:
The information processing device includes:
a providing unit for providing the terminal device with other partial map information generated by another terminal device different from the terminal device;
an acquisition unit that acquires, from the terminal device, information on a matching result obtained by matching a relative positional relationship between trajectory position information indicating a position on a trajectory along which the terminal device moves within an area included in the partial map information generated by the terminal device and the other partial map information, and the partial map information; and
an optimization processing unit that corrects an error in overall trajectory position information constructed by combining the trajectory position information of the terminal device based on information of the matching result,
The terminal device
an acquisition unit that acquires the other partial map information generated by another terminal device different from the own device from the information processing device on the cloud side;
a matching unit that matches a relative positional relationship between the trajectory position information and the other partial map information based on trajectory position information indicating a position on a trajectory along which the device moves in an area included in the partial map information generated by the device, point cloud information indicating a position of an object within the partial map information, and the other partial map information;
a processing unit that uploads the trajectory position information, information on a matching result of a relative position relationship between the trajectory position information and the other partial map information, and the partial map information to the information processing device that generates an environmental map corresponding to the partial map.

Images