JP7587645B2 - Information processing device - Google Patents

Description

The present invention relates to an information processing device.

Conventionally, a method (SLAM: Simultaneous Localization and Mapping) is known in which a moving object simultaneously performs self-location estimation and creates an environmental map. As one type of SLAM, a method (VSLAM: Visual SLAM) has been proposed that uses a video camera instead of a laser device (see, for example, Patent Document 1). In VSLAM as described in Patent Document 1, the use of a video camera aims to reduce costs.

JP 2014-222550 A

In VSLAM as described in Patent Document 1, feature points are extracted from each image (frame) captured by a video camera, and self-location estimation and environmental map creation are performed based on changes in feature points between each image. Point cloud data contained in such environmental maps is collected and integrated by an external server or other device, and the integrated point cloud data may be used for self-location estimation, etc.

However, the generated point cloud data is not always highly accurate. Therefore, if the multiple point cloud data to be integrated contain point cloud data with low reliability, the reliability of the integrated point cloud data may decrease. If the reliability of the integrated point cloud data is low, it will be difficult to use it for self-location estimation, etc.

Therefore, an example of an objective of the present invention is to provide an information processing device that can improve the reliability of the integrated point cloud data.

In order to solve the above-mentioned problems and achieve the object, the information processing device of the present invention described in claim 1 is characterized by comprising an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on a moving body, a driving condition acquisition unit that acquires driving condition information related to the driving conditions of the moving body at the time of imaging, a generation unit that generates point cloud data based on the imaging information, and an information assignment unit that assigns the driving condition information to the point cloud data.

The information processing device of the present invention described in claim 2 is characterized by comprising an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on a moving body, a driving condition acquisition unit that acquires driving condition information related to the driving conditions of the moving body at the time of imaging, a generation unit that generates point cloud data based on the imaging information, and an information assignment unit that assigns reliability information based on the driving condition information to the point cloud data.

The information processing method of the present invention described in claim 5 is characterized by including an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on a moving body, a driving condition acquisition step of acquiring driving condition information related to the driving conditions of the moving body at the time of imaging, a generation step of generating point cloud data based on the imaging information, and an information assignment step of assigning the driving condition information to the point cloud data.

The information processing method of the present invention described in claim 6 is characterized by including an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on a moving body, a driving condition acquisition step of acquiring driving condition information related to the driving conditions of the moving body at the time of imaging, a generation step of generating point cloud data based on the imaging information, and an information assignment step of assigning reliability information based on the driving condition information to the point cloud data.

1 is a block diagram showing an outline of an information processing device according to an embodiment of the present invention; 3 is a plan view showing an example of a frame of imaging information acquired by the information processing device; FIG. 11 is a plan view showing another example of a frame of imaging information acquired by the information processing device. FIG.

The following describes an embodiment of the present invention. An information processing device according to an embodiment of the present invention includes an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on a moving body, a driving condition acquisition unit that acquires driving condition information related to the driving conditions of the moving body at the time of imaging, a generation unit that generates point cloud data based on the imaging information, and an information assignment unit that assigns driving condition information to the point cloud data.

According to the information processing device of this embodiment, by adding driving condition information to the point cloud data, when collecting and integrating the point cloud data, the contribution of each point cloud data can be determined based on the driving condition information. If the driving conditions are such that highly reliable point cloud data can be obtained, the contribution of this point cloud data can be increased, and if the driving conditions are such that only unreliable point cloud data can be obtained, the contribution of this point cloud data can be decreased. Therefore, when integrating multiple point cloud data, the contribution of each point cloud data can be determined based on the driving condition information, thereby improving the reliability of the integrated point cloud data.

The imaging information acquisition unit, the driving condition acquisition unit, the generation unit, and the information provision unit do not have to be mounted on the same target, and may be mounted on different devices or equipment. In other words, each component of the information processing device may span multiple physically separated devices.

An information processing device according to another embodiment of the present invention includes an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on a moving body, a driving condition acquisition unit that acquires driving condition information related to the driving conditions of the moving body at the time of imaging, a generation unit that generates point cloud data based on the imaging information, and an information assignment unit that assigns reliability information based on the driving condition information to the point cloud data.

According to the information processing device of this embodiment, by adding reliability information based on driving condition information to the point cloud data, when collecting and integrating point cloud data, the contribution of each point cloud data can be determined based on the reliability information, thereby improving the reliability of the integrated point cloud data.

It is preferable that the driving condition acquisition unit acquires, as the driving condition information, at least one of self-information regarding the state or behavior of the moving body and external information regarding the environment around the moving body. The self-information includes, for example, driving stability. The lower the speed of the moving body, the smaller the acceleration (acceleration/deceleration), and the less wobbling, the higher the driving stability. The higher the driving stability, the easier it is to generate highly reliable point cloud data.

External information includes, for example, the difficulty of capturing images and the degree of travel obstruction. When it is dark around the moving body or when it is raining, it becomes difficult to capture images or clear image information cannot be obtained, and the difficulty of capturing images increases. When there is a traffic jam, when the distance to the moving body ahead is close, or when the moving body ahead is large, features around the moving body are hidden in the shadow of other moving bodies, and the difficulty of capturing images increases. In addition, when it is dark around the moving body or when it is raining, the stability of the moving body may decrease, and the degree of travel obstruction increases. When the difficulty of capturing images or the degree of travel obstruction are high, the reliability of the generated point cloud data is likely to decrease.

The driving condition acquisition unit may acquire driving condition information directly, or may calculate or estimate it by acquiring other parameters. For example, the brightness around the mobile body may be detected by a sensor, or may be estimated based on time information and date information. Information about whether or not there are street lights on the road along which the mobile body is traveling may also be taken into consideration. Information about the weather may be detected by a sensor to detect rain, or may be acquired from outside the mobile body via a network, etc.

In any of the above embodiments, the information processing device preferably further includes an output unit that outputs the point cloud data. This allows the point cloud data to be utilized at the output destination.

The information processing method according to the embodiment of the present invention includes an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on a moving body, a driving condition acquisition step of acquiring driving condition information related to the driving conditions of the moving body at the time of imaging, a generation step of generating point cloud data based on the imaging information, and an information assignment step of assigning driving condition information to the point cloud data. According to this information processing method, the reliability of the integrated point cloud data can be improved by assigning driving condition information to the point cloud data.

An information processing method according to another embodiment of the present invention includes an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on a moving body, a driving condition acquisition step of acquiring driving condition information related to the driving conditions of the moving body at the time of imaging, a generation step of generating point cloud data based on the imaging information, and an information assignment step of assigning reliability information based on the driving condition information to the point cloud data. According to this information processing method, the reliability of the integrated point cloud data can be improved by assigning reliability information based on the driving condition information to the point cloud data.

The above-mentioned information processing method may also be implemented as an information processing program that causes a computer to execute the information processing method. In this way, the reliability of the integrated point cloud data can be improved using a computer.

The above-mentioned information processing program may also be stored on a computer-readable recording medium. In this way, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades, etc., can be easily performed.

The following is a detailed description of an embodiment of the present invention. As shown in FIG. 1, the information processing unit 1 of this embodiment is composed of a first information processing device 1A, which is a VSLAM device mounted on a moving object 10, and a second information processing device 1B mounted on an external server 100. The first information processing device 1A includes an information acquisition unit 2, a generation unit 3, and an output unit 4. The moving object 10 is, for example, a vehicle, and is equipped with an imaging unit 20, a map information storage unit 30, a self-position estimation unit 40, a driving information sensor 50, and a date and time information management unit 60.

The second information processing device 1B includes a collection unit 11 and a generation unit 12. The external server 100 includes a communication unit 101 and a storage unit main body 102. The external server 100 is configured to communicate with multiple first information processing devices 1A.

The imaging unit 20 is, for example, a video camera, and is positioned to capture an image of the forward direction of the moving body 10. The video captured by the imaging unit 20 is composed of multiple pieces of imaging information (frames) and is transmitted to the information acquisition unit 2. The imaging unit may also be capable of capturing still images continuously at a predetermined time interval.

The map information storage unit 30 is composed of, for example, a hard disk or non-volatile memory, and stores existing map information. The map information stored in the map information storage unit 30 needs to include at least information about features. Note that information about features includes not only numerical values about the location and size of features, but also information about the attributes and types of features (such as distinction between residential and commercial facilities, and the type of business of commercial facilities).

The self-position estimation unit 40 estimates the current position (absolute position) of the mobile body 10, and may be, for example, a GPS receiving unit that receives radio waves transmitted from multiple GPS (Global Positioning System) satellites. The current position estimated by the self-position estimation unit 40 is transmitted to the information acquisition unit 2 as position information indicating the imaging position of the imaging information captured by the imaging unit 20.

The driving information sensor 50 is a sensor for measuring the amount of displacement of the moving body 10, and is composed of, for example, a vehicle speed pulse acquisition unit that acquires the vehicle speed pulse of the moving body 10, a gyro sensor that measures the amount of azimuth displacement of the moving body 10, and an acceleration sensor that acquires the acceleration of the moving body 10. The traveling direction of the moving body 10 is estimated based on the amount of displacement acquired by the driving information sensor 50. The imaging orientation information regarding the orientation of the imaging unit 20 at the time of imaging is determined based on the traveling direction of the moving body 10 and the mounting direction of the imaging unit 20 on the moving body 10. When the imaging unit 20 is facing the front of the moving body, the traveling direction and the imaging orientation are approximately the same. Therefore, the imaging orientation information is transmitted from the driving information sensor 50 to the information acquisition unit 2. It is sufficient that the driving information sensor 50 is capable of estimating at least the traveling direction of the moving body 10.

The date and time information management unit 60 has a calendar function and a clock function. The clock function is preferably a radio-controlled clock function that receives standard radio waves and automatically corrects errors. Date information and time information are transmitted from the date and time information management unit 60 to the information acquisition unit 2. Based on the date information and time information, solar altitude information and solar direction information can be identified. Therefore, the information acquisition unit 2 that acquires date information and time information can be considered to acquire solar altitude information and solar direction information.

The information acquisition unit 2 functions as an imaging information acquisition unit by acquiring imaging information from the imaging unit 20 as described above, functions as a position information acquisition unit by acquiring position information from the self-position estimation unit 40 as described above, functions as a map information acquisition unit by acquiring map information from the map information storage unit 30, functions as an imaging orientation information acquisition unit by acquiring imaging orientation information from the driving information sensor 50 as described above, and functions as a solar information acquisition unit by acquiring solar altitude information and solar orientation information from the date and time information management unit 60 as described above. The information acquisition unit 2 may acquire other information from a data center, etc., via a network such as the Internet or a public line.

The generating unit 3 extracts feature points from the imaging information acquired by the information acquiring unit 2 to generate a two-dimensional feature point group, and further generates point cloud data based on the plurality of feature point groups. The point cloud data generated by the generating unit 3 may be, for example, three-dimensional point cloud data. For example, the ORB (Oriented fast and Rotated Brief) algorithm or other algorithms may be used to extract the feature points.

The output unit 4 is composed of circuits, antennas, etc. for communicating with networks such as the Internet or public lines, and transmits the information (point cloud data) generated by the generation unit 3 by communicating with the communication unit 101 of the external server 100. As a result, the information is stored in the storage unit main body 102 of the external server 100.

The collection unit 11 of the second information processing device 1B acquires the point cloud data (individual point cloud data) generated by the first information processing device 1A via the communication unit 101. The collection unit 11 collects the individual point cloud data by acquiring individual point cloud data from multiple moving bodies or acquiring individual point cloud data multiple times from one moving body.

The generating unit 12 generates integrated point cloud data based on the multiple individual point cloud data collected by the collecting unit 11. The integrated point cloud data may be a superposition of multiple individual point cloud data, or may indicate the average position of the constituent points of the multiple individual point cloud data.

The thus generated total point cloud data can be used, for example, to estimate the self-position of a moving object. That is, when a feature is detected by an optical sensor (known as LIDAR; Laser Imaging Detection and Ranging) mounted on the moving object, the self-position of the moving object can be estimated by comparing it with the total point cloud data.

[Obtaining driving condition information]
The information acquisition unit 2 acquires driving condition information related to the driving conditions of the moving body 10 at the time of imaging, and functions as a driving condition acquisition unit. The timing at which the information acquisition unit 2 acquires the driving condition information is arbitrary, and for example, the imaging information and the driving condition information may be acquired simultaneously, or the driving condition information may be acquired after acquiring the imaging information and generating the point cloud data.

The driving condition information includes self-information about the state or behavior of the mobile body 10 and external information about the environment around the mobile body 10. Note that the driving condition information may include only one of the self-information and the external information.

The self-information includes, for example, driving stability. The lower the speed of the moving body 10, the smaller the acceleration (less acceleration/deceleration), and the less swaying, the higher the driving stability. The higher the driving stability, the easier it is to generate highly reliable point cloud data. The driving stability may be calculated based on the speed, acceleration, and swaying at the time of imaging, or may be calculated based on past driving history. In other words, if stability was high during past driving, it can be estimated that the driving stability is also high at the time of imaging. In addition, the driving stability may be calculated based on the past driving history of the moving body, or based on the past driving history of the driver.

The self-information may also include information about the performance and maintenance status of the mobile body 10. For example, if the acceleration and braking performance of the mobile body 10 is high, large acceleration and deceleration may occur. Furthermore, the likelihood of wobbling may change depending on the power steering settings. In other words, the performance and maintenance status of the mobile body 10 may change how easily the mobile body responds to the driver's operations, which may affect driving stability.

The external information includes, for example, the difficulty of capturing images and the degree of travel obstruction. When it is dark around the moving body 10 or when it is raining, it becomes difficult for the imaging unit 20 to capture images or clear imaging information cannot be obtained, and the difficulty of capturing images increases. When there is a traffic jam, when the distance to the moving body ahead is close, or when the moving body ahead is large, the features around the moving body are hidden by the other moving bodies, and the difficulty of capturing images increases.

For example, when a large vehicle C such as a truck is traveling ahead as shown in Figure 2, part or all of a building B located on the forward left side of the road R as shown in Figure 3 is hidden by the shadow of the large vehicle C, making it impossible to extract the feature points of building B.

In addition, when it is dark around the moving body 10 or when it is raining, the stability of the moving body 10 may decrease, and the degree of driving obstruction may increase. When the difficulty of capturing images is high or the degree of driving obstruction is high, the reliability of the generated point cloud data is likely to decrease.

The driving condition information may be acquired directly, or may be calculated or estimated by acquiring other parameters. For example, the brightness around the mobile body 10 may be detected by a sensor, or may be estimated by acquiring time information and date information from the date and time information management unit 60. Furthermore, by acquiring map information from the map information storage unit 30, the brightness may be estimated taking into consideration whether or not there are street lights on the road along which the vehicle is traveling. Furthermore, information about the weather may be detected by rain using a sensor, or may be acquired from outside the mobile body 10 by the information acquisition unit 2 via a network or the like.

[Adding information to point cloud data]
The generating unit 3 functions as an information adding unit by adding the driving condition information acquired as described above to the generated point cloud data. That is, the output unit 4 outputs the point cloud data to which the driving condition information has been added.

The generation unit 3 may function as an information adding unit by adding reliability information to the point cloud data instead of the driving condition information. That is, the reliability may be determined based on the driving condition information, and information about this reliability may be added to the point cloud data.

[Integration of individual point cloud data with annotations]
As described above, in the second information processing device 1B, the collection unit 11 collects individual point cloud data, and the generation unit 12 generates comprehensive point cloud data based on the plurality of individual point cloud data. At this time, if driving condition information is added to the individual point cloud data, the generation unit 12 determines the contribution degree of the individual point cloud data based on the driving condition information, and generates comprehensive point cloud data.

In this case, since the reliability of the individual point cloud data changes depending on the driving conditions as described above, the contribution degree of the individual point cloud data with high reliability can be increased based on the driving condition information. Note that the contribution degree includes 0, and by setting the contribution degree to 0, the individual point cloud data can be removed. Furthermore, if reliability information rather than driving condition information is assigned to the individual point cloud data, the generation unit 12 can determine the contribution degree of the point cloud data based on the reliability information.

The generation unit 12 may also generate comprehensive point cloud data based on the individual point cloud data for each driving condition. For example, the individual point cloud data may be classified according to the weather, and comprehensive point cloud data for sunny days, comprehensive point cloud data for cloudy days, and comprehensive point cloud data for rainy days may be generated. Furthermore, comprehensive point cloud data may be generated for each driver or vehicle type, or comprehensive point cloud data may be generated for each driving speed.

As described above, by generating comprehensive point cloud data for each driving condition, the reliability of the comprehensive point cloud data can be improved. In other words, when using the comprehensive point cloud data, by selecting comprehensive point cloud data that corresponds to the conditions at that time, the discrepancy between the conditions when the point cloud data was generated and the conditions when it is used can be reduced. In the above example, if it is raining when the comprehensive point cloud data is used, the comprehensive point cloud data generated based on the individual point cloud data for rainy weather can be used.

[Acquisition of driving condition information in second information processing device]
In the above example, the first information processing device 1A acquires driving condition information and assigns the information to the individual point cloud data, but the collection unit 11 of the second information processing device 1B may acquire individual point cloud data to which no information is assigned and driving condition information. That is, the collection unit 11 may function as a driving condition acquisition unit. In this case, the driving condition information acquired by the collection unit 11 may include at least one of self-information and external information, as in the above example.

The generation unit 12 determines the contribution of the individual point cloud data based on the driving condition information, and generates the overall point cloud data. At this time, the contribution of the individual point cloud data with high reliability may be increased based on the driving condition information. The generation unit 12 may also generate the overall point cloud data based on the individual point cloud data for each driving condition, as described above.

With the above configuration, by adding driving condition information or reliability information to the point cloud data, when collecting and integrating the individual point cloud data, the contribution of each individual point cloud data can be determined based on the driving condition information or reliability information, thereby improving the reliability of the overall point cloud data.

In addition, by outputting the point cloud data to the external server 100 using the output unit 4, the point cloud data can be utilized in the external server 100 to which the data is output.

In addition, even if the collection unit 11 of the second information processing device 1B acquires individual point cloud data to which no information is attached and driving condition information, when collecting and integrating the individual point cloud data, the contribution degree of each individual point cloud data can be determined based on the driving condition information, thereby improving the reliability of the overall point cloud data.

The present invention is not limited to the above-described embodiment, but includes other configurations that can achieve the object of the present invention, and the following modifications are also included in the present invention.

For example, in the above embodiment, the information acquisition unit 2 and the generation unit 3 are mounted on the moving body 10, but the imaging information acquisition unit, the driving condition acquisition unit, the generation unit, and the information provision unit constituting the first information processing device do not have to be mounted on the moving body, and may be mounted on different devices or equipment. In other words, each component of the information processing device may span multiple devices that are physically separated.

In addition, in the above embodiment, the collection unit 11 and the generation unit 12 are mounted on the external server 100, but the collection unit, generation unit, and driving condition acquisition unit constituting the second information processing device do not have to be mounted on the external server, and may be mounted on different devices or equipment. In other words, each component of the information processing device may span multiple physically separated devices.

Although the best configurations and methods for implementing the present invention have been disclosed above, the present invention is not limited thereto. That is, the present invention has been illustrated and described mainly with respect to specific embodiments, but those skilled in the art can make various modifications to the above-described embodiments in terms of shape, material, quantity, and other detailed configurations without departing from the scope of the technical idea and purpose of the present invention. Therefore, the descriptions limiting the shapes, materials, etc. disclosed above are provided as examples to facilitate understanding of the present invention, and do not limit the present invention. Therefore, descriptions of the names of components that have some or all of the limitations on the shapes, materials, etc. removed are included in the present invention.

1A First information processing device 2 Information acquisition unit (imaging information acquisition unit, driving condition acquisition unit)
3. Generation unit (information providing unit)
4 Output unit 10 Moving body 20 Imaging unit 1B Second information processing device 11 Collection unit (driving condition acquisition unit)
12 Generation unit

Claims (8)

An information processing device that generates point cloud data to be integrated into comprehensive point cloud data together with other point cloud data,
an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on the moving object;
a driving condition acquisition unit that acquires driving condition information regarding a driving condition of the moving object at the time of imaging;
a generation unit that generates a two-dimensional feature point group based on the imaging information, and further generates the point cloud data based on a plurality of the feature point groups ;
an information adding unit that adds the driving condition information to the point cloud data ,
The information processing apparatus , wherein the point cloud data is three-dimensional point cloud data . An information processing device that generates point cloud data to be integrated into comprehensive point cloud data together with other point cloud data,
an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on the moving object;
a driving condition acquisition unit that acquires driving condition information regarding a driving condition of the moving object at the time of imaging;
a generation unit that generates a two-dimensional feature point group based on the imaging information, and further generates the point cloud data based on a plurality of the feature point groups ;
an information providing unit that provides reliability information based on the driving condition information to the point cloud data ,
The information processing apparatus , wherein the point cloud data is three-dimensional point cloud data . The information processing device according to claim 1 or 2, characterized in that the driving condition acquisition unit acquires, as the driving condition information, at least one of self-information regarding the state or behavior of the moving body and external information regarding the environment surrounding the moving body. The information processing device according to any one of claims 1 to 3, further comprising an output unit that outputs the point cloud data. 1. An information processing method executed by an information processing device for generating point cloud data to be integrated with other point cloud data into comprehensive point cloud data, comprising:
an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on the moving object;
a driving condition acquisition step of acquiring driving condition information regarding a driving condition of the moving object at the time of imaging;
a generating step of generating a two-dimensional feature point group based on the imaging information, and further generating the point cloud data based on a plurality of the feature point groups ;
and an information imparting step of imparting the driving condition information to the point cloud data ,
The information processing method , wherein the point cloud data is three-dimensional point cloud data . 1. An information processing method executed by an information processing device for generating point cloud data to be integrated with other point cloud data into comprehensive point cloud data, comprising:
an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on the moving object;
a driving condition acquisition step of acquiring driving condition information regarding a driving condition of the moving object at the time of imaging;
a generating step of generating a two-dimensional feature point group based on the imaging information, and further generating the point cloud data based on a plurality of the feature point groups ;
and an information assignment step of assigning reliability information based on the driving condition information to the point cloud data ,
The information processing method , wherein the point cloud data is three-dimensional point cloud data . An information processing program for causing a computer to execute the information processing method according to claim 5 or 6. A computer-readable recording medium storing the information processing program according to claim 7.

Images