WO2025033237A1 - Information processing device, information processing system, information processing method, and recording medium - Google Patents

Abstract

This information processing device comprises a first acquisition unit, a second acquisition unit, an imparting unit, and an output unit. The first acquisition unit generates, on the basis of radar information, a two-dimensional point cloud obtained by converting a three-dimensional point cloud. The second acquisition unit acquires, on the basis of a captured image captured using light, a two-dimensional captured image represented using a coordinate system shared with the two-dimensional point cloud, and label information imparted to an object included in the two-dimensional captured image. The imparting unit imparts the label information to the two-dimensional point cloud on the basis of the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image. The output unit outputs output information in which label information is imparted to a point cloud based on the radar information.

Description

Information processing device, information processing system, information processing method, and recording medium

This disclosure relates to an information processing device, an information processing system, an information processing method, and a recording medium.

For example, Patent Document 1 discloses a point cloud information processing device for improving robustness in aligning multiple pieces of point cloud information. According to Patent Document 1, the point cloud information processing device includes an image analysis unit, a point cloud labeling unit, and a point cloud integration unit.

The image analysis unit analyzes image information captured from different viewpoints, recognizes different areas in each image, labels each area, and generates labeled image information. The point cloud labeling unit generates labeled point cloud information by labeling each point in the point cloud information from different viewpoints with a label of the corresponding area in the labeled image information based on the positional information of each point. The point cloud integration unit aligns the labeled point cloud information using labels that are common to multiple labeled point cloud information.

JP 2022-157660 A

In the point cloud information processing device described in Patent Document 1, image information captured from different viewpoints is used to assign label information to a point cloud. This poses the problem that it is difficult to assign label information to a point cloud.

The information processing device according to the present disclosure includes:
a first acquisition means for generating a two-dimensional point cloud by converting the three-dimensional point cloud based on the radar information;
a second acquisition means for acquiring, based on a captured image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image;
an assignment means for assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
and an output means for outputting output information in which the label information is added to a point cloud based on the radar information.

The information processing system according to the present disclosure includes:
a moving object that moves to generate the radar information by scanning a scanning area using a radar;
an imaging device for imaging the scanning region to generate the imaging image;
The information processing device includes:
The moving body is
a transmitting means for transmitting the radar to the scanning area;
a receiving means for receiving the reflected wave of the transmitted radar and generating the radar information relating to the reflected wave;
and transmitting means for transmitting the generated radar information.

The information processing method according to the present disclosure includes:
One or more computers
A two-dimensional point cloud is generated by converting the three-dimensional point cloud based on the radar information;
acquiring a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image based on a captured image captured using light;
assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
Output information is output in which the label information is added to a point cloud based on the radar information.

The recording medium in the present disclosure is
On one or more computers,
A two-dimensional point cloud is generated by converting the three-dimensional point cloud based on the radar information;
acquiring a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image based on a captured image captured using light;
assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
A program for executing the process of outputting output information in which the label information is added to a point cloud based on the radar information is recorded.

This disclosure makes it possible to easily assign label information to point clouds.

1 is a block diagram illustrating a configuration example of a first information processing system according to the present disclosure. 1 is a block diagram showing a configuration example of a first information processing device according to the present disclosure. 11 is a flowchart illustrating an example of a processing operation of the first information processing device according to the present disclosure. 1A is a diagram showing an example of an image including a two-dimensional point cloud obtained by projecting a three-dimensional point cloud onto a two-dimensional point cloud on a projection plane, and FIG. 1B is a diagram showing an example of an image including a two-dimensional point cloud to which a frame surrounding an object has been added. FIG. 13 is a diagram showing an example of a two-dimensional captured image to which a frame surrounding an object has been added. FIG. 2 is a diagram illustrating an example of a physical configuration of a first information processing device according to the present disclosure. 2 is a block diagram showing a configuration example of a first acquisition unit according to the present disclosure. FIG. 10 is a flowchart illustrating an example of a processing operation of a first acquisition unit according to the present disclosure. 4 is a block diagram showing an example configuration of a second acquisition unit according to the present disclosure. FIG. 10 is a flowchart illustrating an example of a processing operation of a second acquisition unit according to the present disclosure.

Hereinafter, in this disclosure, the drawings relate to one or more embodiments. In addition, in all drawings, similar components are given similar reference symbols and descriptions are omitted as appropriate.

[Embodiment 1]
(overview)
As shown in FIG. 1, the information processing system 100 includes a moving object 110, an image capturing device 120, and an information processing device 130.

The mobile body 110 moves to generate radar information by scanning a scanning area using a radar. The mobile body 110 includes a transmitter 111, a receiver 112, and a transmitter 113.

The transmitter 111 transmits radar waves into the scanning area. The receiver 112 receives the reflected waves of the transmitted radar and generates radar information related to the reflected waves. The transmitter 113 transmits the generated radar information.

The imaging device 120 is a device for capturing images of the scanning area and generating a captured image.

As shown in FIG. 2, the information processing device 130 includes a first acquisition unit 131, a second acquisition unit 132, an assignment unit 133, and an output unit 134.

The first acquisition unit 131 generates a two-dimensional point cloud by converting the three-dimensional point cloud based on the radar information.

The second acquisition unit 132 acquires, based on an image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud, and label information assigned to objects included in the two-dimensional captured image.

The assignment unit 133 assigns label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image.

The output unit 134 outputs output information in which label information is added to the point cloud based on the radar information.

This information processing system 100 can assign label information to a point cloud using a captured image. At least one captured image is sufficient. This makes it easy to assign label information to a point cloud.

In addition, this information processing device 130 can assign label information to a point cloud using a captured image. At least one captured image is sufficient. This makes it easy to assign label information to a point cloud.

The information processing device 130 executes information processing as shown in FIG. 3.

The first acquisition unit 131 generates a two-dimensional point cloud by converting the three-dimensional point cloud based on the radar information (step S101).

The second acquisition unit 132 acquires, based on the captured image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud, and label information assigned to objects included in the two-dimensional captured image (step S102).

The assignment unit 133 assigns label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image (step S103).

The output unit 134 outputs output information in which label information is added to the point cloud based on the radar information (step S104).

This information processing allows label information to be added to a point cloud using a captured image. At least one captured image is sufficient. This makes it easy to add label information to a point cloud.

(Detailed example)
A detailed example of the information processing system 100 etc. will be described below.

(Configuration example of moving body 110)
The moving body 110 is an air vehicle such as a drone that moves by remote control or operation by an operator or automatically according to a predetermined algorithm, etc. The moving body 110 may be equipped with devices, apparatuses, etc. that realize the functions of a transmitting unit 111, a receiving unit 112, a transmitting unit 113, etc.

Note that the moving body 110 is not limited to an aircraft, and may be a vehicle such as an automobile. The moving body 110 may further include a movement control unit (not shown) for controlling its movement.

For radar, millimeter waves, which are radio waves with wavelengths of 1 to 10 mm (millimeters), are preferably used. One characteristic of millimeter waves is that they penetrate materials better than light. "Light" includes visible light and infrared light, and this also applies below.

Note that radar can use radio waves of various wavelengths, not just millimeter waves. For example, microwaves can be used, or microwaves with wavelengths longer than light. Microwaves are radio waves with wavelengths of 1 meter or less, such as ultra-high frequency waves, centimeter waves, millimeter waves, and submillimeter waves. Ultra-high frequency waves, centimeter waves, and submillimeter waves have wavelengths of 0.1 to 1 m (meters), 1 to 10 cm (centimeters), and 0.1 to 1 mm, respectively.

The scanning area is an area that is predetermined to be scanned by the radar. The scanning area may be, for example, outdoors or indoors.

In more detail, for example, the scanning area may be an area where a metal object or the like may be placed on the ground surface or may be partially or completely buried underground. The metal object or the like can be detected using radar information obtained by scanning such a scanning area with a radar.

Also, for example, the scanning area may be a predetermined area of a building or the like. By using radar information obtained by scanning such a scanning area with a radar, it is possible to detect the condition of pipes, reinforcing bars, etc. arranged inside or outside the walls of the building or the like, and to discover abnormalities in the pipes, reinforcing bars, etc. Examples of buildings include, but are not limited to, buildings and bridges.

The mobile body 110, for example, moves while the transmitter 111 emits radar and the receiver 112 receives the reflected waves. This allows the radar to scan the scanning area and obtain radar information related to the reflected waves. In other words, radar information is information obtained using the mobile body 110, and more specifically, information obtained using an aircraft such as a drone, a vehicle, etc.

For example, if the mobile object 110 is a drone, it may transmit radar waves and receive the reflected waves while flying at a height of about 10 m. Various common methods may be used for the radar transmission method. Examples of radar transmission methods include frequency-controlled modulation (FMCW), pulse, continuous wave Doppler (CWD), two-frequency CW, and pulse compression.

Then, when the receiver 112 generates radar information regarding the received reflected waves, the transmitter 113 transmits the radar information to the information processing device 130, for example, via the network NT1. The network NT1 is typically a wireless line, but may also include at least a wired line in part. The transmitter 113 may transmit the radar information in real time, or may transmit multiple pieces of radar information generated at different times together.

Radar information is, for example, information about the reflected waves of the radar emitted into the scanning space. In detail, for example, radar information associates one or more of the intensity of the reflected waves, the observation position, the observation direction, the observation time, etc.

The observation position is the position in real space where the observation is performed, and may be at least one of the following: the radar transmission position, the reception position of the reflected wave, a position obtained using the transmission position and reception position, such as a position midway between the transmission position and reception position. The observation position is expressed, for example, by latitude, longitude, height, etc., and may be obtained by equipping the mobile body 110 with a GPS (Global Positioning System) function. Note that the observation position is not limited to the examples given here.

The observation direction may be at least one of the radar transmission direction, the reflected wave reception direction, a direction obtained using the transmission direction and reception direction, etc. Also, the receiver 112 may include one or more antennas to obtain the reception direction.

The observation time is information indicating the time of observation, for example, the observation time. The observation time may be at least one of the following: the radar transmission time (for example, the transmission time), the reception time of the reflected wave (for example, the transmission time), a time associated with the transmission time and the reception time, such as midway between the transmission time and the reception time. The observation time may be obtained by providing the mobile body 110 with a timekeeping function.

(Example of configuration of the imaging device 120)
The image capturing device 120 is a device for capturing an image of the operation area using light. As described above, the light includes visible light and infrared light. For example, when visible light is used, the image capturing device 120 is a visible light camera. For example, when infrared light, near infrared light, or far infrared light is used, the image capturing device 120 is an infrared camera, a near infrared camera, or a far infrared camera, respectively.

The imaging device 120 generates an image of the scanning area and transmits the image to the information processing device 130, for example, via the network NT2.

The captured image is, for example, a color image such as an RGB image. The captured image may also be a monochrome image.

Network NT2 is typically a wireless line, but may include at least a wired line. Note that network NT1 and network NT2 may be partly or entirely a common network, or partly or entirely different networks.

The image capturing device 120 may be fixed in position, or may be mounted on the above-mentioned mobile body 110 or a mobile body different from the mobile body 110. This mobile body may be an aerial vehicle such as a drone, or a vehicle such as an automobile. In addition, this mobile body may move by remote control or operation by an operator, or automatically by a predetermined algorithm, etc.

If the imaging device 120 is mounted on a moving body, it is preferable to image the scanning area by having the imaging device 120 capture images while moving.

For example, if the imaging device 120 is mounted on a moving object, the imaging device 120 may capture images of the scanning area while moving. In this case, the imaging device 120 may transmit captured images in real time, or may transmit multiple images captured at different times together.

The photographing device 120 may transmit photographing information that associates at least one of the photographing location and the photographing time with the photographed image.

The shooting position is the position in real space where the image was captured. The shooting position is expressed, for example, by latitude, longitude, and altitude, and may be obtained by equipping the image capture device 120 or a mobile object carrying the image capture device 120 with a GPS (Global Positioning System) function. Note that the shooting position is not limited to the example given here.

The shooting time is the time when the image was taken, for example the time of the image. Note that the shooting time is not limited to the examples given here.

(Example of functional configuration of information processing device 130)
(Regarding the first acquisition unit 131)
The first acquisition unit 131 generates a three-dimensional point cloud based on, for example, radar information transmitted from the transmission unit 113, and generates a two-dimensional point cloud by converting the three-dimensional point cloud.

In detail, for example, the first acquisition unit 131 generates three-dimensional information related to the three-dimensional point cloud based on the radar information transmitted from the transmission unit 113.

The three-dimensional point cloud is a cloud of points in three-dimensional space that corresponds to the scanned area. For example, as described above, the scanned area may be within an object, such as within the walls of a building, or may be underground. Thus, the three-dimensional point cloud may include a cloud of points that indicate within an object, such as within the walls of a building, or underground.

For example, the three-dimensional information is information in which a three-dimensional point cloud and a certainty factor are associated. The certainty factor is a value corresponding to the likelihood that an object exists in the associated three-dimensional point cloud. The certainty factor may be calculated according to the intensity of the reflected wave in the three-dimensional point cloud, and is, for example, a value indicating the probability that an object exists in the three-dimensional point cloud.

In more detail, for example, the three-dimensional point cloud may be represented by identification information for identifying each of the three-dimensional point clouds and the position of each of the three-dimensional point clouds. In this case, the three-dimensional information is information that associates the identification information for identifying each of the three-dimensional point clouds, the position of each of the three-dimensional point clouds, and the reliability.

The first acquisition unit 131 may, for example, calculate the distance at each point in the three-dimensional space using a fast Fourier transform (FFT) based on the radar information, and integrate the calculated distances to calculate the angle or position. Also, for example, the first acquisition unit 131 may determine, for each three-dimensional space, a reliability value that is larger the more an object exists at each point in the three-dimensional space, based on the radar information. This allows the first acquisition unit 131 to generate three-dimensional information.

Note that there may be multiple pieces of radar information. In this case, the multiple pieces of radar information may be generated and transmitted by each of the multiple mobile bodies 110. The mobile body 110 may also have multiple sets of transmitters 111 and receivers 112. The multiple pieces of radar information may be generated by each of the multiple sets of transmitters 111 and receivers 112 and transmitted from one or multiple transmitters 113.

The first acquisition unit 131 generates two-dimensional information related to the two-dimensional point cloud, for example, by converting the three-dimensional point cloud.

This transformation is a projection onto a predetermined projection plane. In other words, the two-dimensional point cloud is a point cloud in which a three-dimensional point cloud is projected onto a projection plane. Figure 4(a) shows an example of projecting a three-dimensional point cloud onto a two-dimensional point cloud on a projection plane.

The projection plane is a plane that corresponds to the ground surface included in the scanning area or a plane parallel to the ground surface at a predetermined distance in either the up or down direction from the ground surface.

For such a conversion from a three-dimensional point cloud to a two-dimensional point cloud, it is advisable to use a general technique for transforming an image, such as an affine transformation or a homography transformation. Note that the conversion from a three-dimensional point cloud to a two-dimensional point cloud is not limited to a general technique for transforming an image, and for example, a technique for transforming a coordinate system may also be used.

Note that the projection plane is not limited to the one exemplified here, and for example does not have to be parallel to the earth's surface, and may be defined using altitude, etc., instead of distance to the earth's surface.

For example, the two-dimensional information is information for associating, for each two-dimensional point cloud, at least a portion of the following: identification information for identifying each two-dimensional point cloud, a position on the projection plane, a corresponding three-dimensional point cloud, and a confidence level, which is a value corresponding to the likelihood that an object exists.

The corresponding 3D point cloud is information about the 3D point cloud from which it is projected.

For example, the corresponding three-dimensional point clouds may be identification information for identifying each of the three-dimensional point clouds. This makes it possible to associate each piece of information included in the three-dimensional information with each of the two-dimensional point clouds using the identification information and the three-dimensional information.

For example, the corresponding three-dimensional point cloud may include one or more of the pieces of information included in the three-dimensional information, namely, identification information for identifying each of the three-dimensional point clouds, the position of each of the three-dimensional point clouds, and the reliability. In this way, the two-dimensional information can directly include information on the corresponding three-dimensional point cloud, thereby associating information on the three-dimensional point cloud from which it is projected.

The degree of confidence contained in the two-dimensional information may be set based on the degree of confidence assigned to the corresponding three-dimensional point cloud. The degree of confidence contained in the two-dimensional information may be the same as the degree of confidence assigned to the corresponding three-dimensional point cloud, for example.

However, when projecting onto a projection plane, multiple 3D point groups may be projected onto a common position on the projection plane.

In such cases, the corresponding 3D point cloud may be the plurality of 3D point clouds, or may be a 3D point cloud that is representative of the plurality of 3D point clouds (e.g., the 3D point cloud closest to the projection plane, the 3D point cloud associated with the greatest confidence, etc.).

Furthermore, if the corresponding three-dimensional point cloud is a plurality of such three-dimensional point clouds, the confidence level included in the two-dimensional information may be, for example, the average of the confidence levels associated with the plurality of such three-dimensional point clouds. If the corresponding three-dimensional point cloud is a three-dimensional point cloud closest to the projection plane, the confidence level included in the two-dimensional information may be, for example, the confidence level associated with the closest three-dimensional point cloud. If the corresponding three-dimensional point cloud is a three-dimensional point cloud with which the maximum confidence level is associated, the confidence level included in the two-dimensional information may be, for example, the confidence level of the three-dimensional point cloud.

(Regarding the second acquisition unit 132)
The second acquisition unit 132 acquires a two-dimensional captured image and label information assigned to an object included in the two-dimensional captured image based on the captured image of the scanning area transmitted from the imaging device 120 .

A two-dimensional captured image is an image obtained by deforming the captured image of the scanning area so that the coordinate system used for the captured image of the scanning area is a common coordinate system for the two-dimensional point cloud. In other words, a two-dimensional captured image can also be said to be a captured image of the scanning area expressed in a common coordinate system for the two-dimensional point cloud. Note that the captured image may also be a two-dimensional image, but in this disclosure, in order to distinguish between images before and after deformation, the image before deformation is referred to as the captured image, and the image after deformation is referred to as the two-dimensional captured image.

The target object is a predefined object. For example, the target object may be any object that may be present in the scanning area.

The label information may include at least one of class identification information and class confidence.

Class identification information is information for identifying the class (type of object) to which an object belongs. The class (type of object) may be one or more of, for example, metal objects, plants, rocks, etc.

The class identification information may be represented, for example, by one or more combinations of letters, numbers, symbols, etc. that are assigned in advance to each class.

Note that the classes and the methods of expressing them are not limited to those exemplified here.

The class confidence is a value that indicates the likelihood that an object belongs to a class, and is expressed, for example, as a continuous value in a predetermined range.

Such label information may be obtained by human input or automatically using an object detection model.

The object detection model is a machine learning model for detecting objects and assigning class information to the objects, and when an image is input, it may output class information associated with the object contained in the image. The object detection model may perform learning using a learning image and ground truth data that includes class information of the object in the image.

Such an object detection model may be a general machine learning model for detecting objects from images. Examples of techniques that may be applied to such object detection models include R-CNN, YOLO, SSD, Fast R-CNN, and Faster R-CNN.

For example, if an object is buried underground, label information for the object may be assigned by placing a marker on the ground that indicates the object's position, and the label information may be assigned to the buried object through human input or automatic detection using an object detection model.

The label information may be associated with the position of a frame (e.g., a rectangular frame circumscribing the object) surrounding the object in the two-dimensional captured image. FIG. 5 is a diagram showing an example of a two-dimensional captured image to which a frame surrounding the object has been added. In FIG. 5, the area corresponding to the object in the two-dimensional captured image (the image of the second areas Qa to Qc) is shown by hatching. In this case, the label information may further include the position of the frame surrounding the object in the two-dimensional captured image. Note that the frame is not limited to a rectangle, and the shape, etc. may be changed as appropriate.

When label information is assigned to a two-dimensional captured image, the label information may be associated with each pixel corresponding to an object. The label information may be associated with the position of a frame that surrounds the object in the two-dimensional captured image (e.g., a rectangular frame that circumscribes the object). The label information may be associated with a region (mask) that corresponds to the object in the two-dimensional captured image.

(Regarding the assignment unit 133)
The assigning unit 133 assigns label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and an object in the two-dimensional captured image.

(Assignment of label information to 2D point clouds)
As described above, the two-dimensional point cloud and the two-dimensional captured image share a common coordinate system. Therefore, for example, the two-dimensional point cloud included in the region of the object in the two-dimensional captured image may be regarded as a point cloud corresponding to the object, and the assigning unit 133 may assign label information to the two-dimensional point cloud.

When label information is assigned to two-dimensional point clouds, the label information may be associated with each of the two-dimensional point clouds. The label information may be associated with the position of a frame (e.g., a rectangular frame circumscribing the object) that surrounds the object in the two-dimensional point cloud. In FIG. 4(b), the point cloud corresponding to the object (the point cloud of the first area Pa to Pc) is shown as a set of points. In this case, the label information may further include the position of the frame that surrounds the object in the two-dimensional point cloud. Note that the frame is not limited to a rectangle, and the shape, etc., may be changed as appropriate. Furthermore, the label information is not limited to a frame that surrounds the object, and may be associated with at least one of the two-dimensional point clouds in an appropriate manner, such as a region (mask) that corresponds to the object in the two-dimensional point cloud.

At this time, when the degree of certainty in the two-dimensional information is equal to or greater than a predetermined reference value, the assigning unit 133 may assign label information to the two-dimensional point cloud associated with the degree of certainty. In other words, the label information may be assigned to the two-dimensional point cloud whose degree of certainty is equal to or greater than a predetermined reference value.

(Assignment of label information to 3D point clouds)
As described above, the two-dimensional information associates the two-dimensional point group with the three-dimensional point group from which it is projected. Therefore, the assigning unit 133 may assign the label information to a three-dimensional point group corresponding to the two-dimensional point group to which the label information is assigned, using the two-dimensional point group to which the label information is assigned and the two-dimensional information. This allows the label information to be assigned to the three-dimensional point group. In other words, the label information may be assigned to the two-dimensional point group based on the positional relationship, and may be assigned to the three-dimensional point group associated with the two-dimensional point group by the two-dimensional information.

When label information is assigned to a three-dimensional point cloud, the label information may be associated with each of the three-dimensional point clouds. The label information may be associated with the position of a frame that surrounds an object in the three-dimensional point cloud (e.g., a rectangular frame that circumscribes the object). The label information may be associated with a region (mask) that corresponds to the object in the three-dimensional point cloud.

(Assigning label information to points within a specific area)
Furthermore, the assigning unit 133 may assign label information to a point group in a specific region different from the two-dimensional point group and the three-dimensional point group based on the two-dimensional point group to which the label information has been assigned. That is, the label information may be further assigned to a point group in a specified specific region based on the two-dimensional point group to which the label information has been assigned.

The specific area is an area that is appropriately specified. For example, a specific plane different from the projection plane may be specified, and the specific area may be defined based on the specific plane.

The specific plane may be specified, for example, using a distance from the ground surface (e.g., height). The specific area is, for example, an area in the vicinity of the specific plane. In detail, for example, the specific area is an area within a predetermined range in the vertical direction from the specific plane. This predetermined range may be a height specified by the user, or may be determined in advance.

When label information is assigned to a point cloud of a specific region, the label information may be associated with each point cloud of the specific region. The label information may be associated with the position of a frame that surrounds an object in the point cloud of the specific region (e.g., a rectangular frame that circumscribes the object). The label information may be associated with a region (mask) that corresponds to the object in the point cloud of the specific region.

Here, for example, among the two-dimensional point clouds corresponding to the point cloud of a specific region, there may be multiple two-dimensional point clouds to which label information has been assigned. In this case, the label information may be preferentially selected from the two-dimensional point clouds with the highest reliability associated with them, and assigned to the point cloud of the specific region corresponding to the selected two-dimensional point cloud.

(Regarding the output unit 134)
The output unit 134 outputs output information in which label information is added to a point cloud based on the radar information. This point cloud may be one or more of a two-dimensional point cloud, a three-dimensional point cloud, and a point cloud within a specific region.

In other words, the output information may be information in which label information is added to a two-dimensional point cloud. The output information may be information in which label information is added to a three-dimensional point cloud. The output information may be information in which label information is added to a point cloud within a specific region.

The destination of the output information may be one or more of another device connected via a network, a storage unit (not shown), a display unit (not shown), etc. Each of the storage unit and the display unit may be provided in the other device.

(Example of physical configuration of information processing device 130)
As shown in FIG. 6, the information processing device 130 physically includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070.

The bus 1010 is a data transmission path for the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070 to transmit and receive data to and from each other. However, the method of connecting the processor 1020 and other components to each other is not limited to bus connection.

The processor 1020 is a processor realized by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

Memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, or a read only memory (ROM). The storage device 1040 stores program modules for realizing the functions of the information processing device 130 that includes it. The processor 1020 loads each of these program modules into the memory 1030 and executes them to realize the function corresponding to that program module.

The network interface 1050 is an interface for connecting the information processing device 130 that is equipped with it to the network NT.

The input interface 1060 is an interface through which the user inputs information. The input interface 1060 is composed of, for example, a touch panel, a keyboard, a mouse, etc.

The output interface 1070 is an interface for presenting information to the user. The output interface 1070 is composed of, for example, a liquid crystal panel, an organic EL (Electro-Luminescence) panel, etc.

Note that the physical configuration of the information processing device 130 is not limited to this. For example, the information processing device 130 may be composed of multiple devices. In this case, each device may be, for example, a computer or the like having a similar physical configuration to the information processing device 130 shown in FIG. 6.

(Action and Effects)
As described above, according to this embodiment, the information processing device 130 includes the first acquisition unit 131, the second acquisition unit 132, the assignment unit 133, and the output unit 134.

The first acquisition unit 131 generates a two-dimensional point cloud by converting the three-dimensional point cloud based on the radar information. The second acquisition unit 132 acquires a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to objects included in the two-dimensional captured image based on an image captured using light. The assignment unit 133 assigns label information to the two-dimensional point cloud based on the positional relationship between the two-dimensional point cloud and the objects in the two-dimensional captured image. The output unit 134 outputs output information in which label information has been assigned to the point cloud based on the radar information.

This allows label information to be added to the point cloud using the captured image. At least one captured image is sufficient. This makes it easy to add label information to the point cloud.

According to this embodiment, the two-dimensional point cloud is a point cloud in which the three-dimensional point cloud is projected onto a projection plane.

This makes it easy to generate a two-dimensional point cloud to which the label information added to the captured image can be applied. Therefore, it becomes possible to easily add label information to the point cloud.

According to this embodiment, label information is further added to the 3D point cloud. The output information is information in which the label information is added to the 3D point cloud.

This makes it possible to output output information in which label information has been added to the 3D point cloud. Such output information can be used, for example, as training data when training a machine learning model to detect objects contained in a 3D point cloud using the 3D point cloud as input. Generally, it is often difficult for a human to identify where an object is located in a 3D point cloud, making it difficult to create such training data. By using output information in which label information has been added to the 3D point cloud, it becomes possible to facilitate training of a machine learning model to detect objects contained in a 3D point cloud.

According to this embodiment, label information is further assigned to the point cloud within the specified specific region based on the two-dimensional point cloud to which the label information has been assigned. The output information is information in which the label information has been assigned to the point cloud within the specific region.

This allows point cloud label information to be added within any area. Therefore, it becomes possible to easily add label information to a point cloud within any area.

According to this embodiment, the label information includes at least one of class identification information for identifying the class to which the object belongs and a class confidence indicating the likelihood that the object belongs to the class.

This allows the output information to be used as learning data and easily viewed on the display. This makes it possible to utilize the output information for a variety of purposes.

In this embodiment, by generating a two-dimensional point cloud, two-dimensional information is generated for each of the two-dimensional point clouds to associate a position on the projection plane, a corresponding three-dimensional point cloud, and a certainty value that corresponds to the likelihood that an object exists. Label information is assigned to the two-dimensional point cloud based on the positional relationship, and is assigned to the three-dimensional point cloud associated with the two-dimensional point cloud by the two-dimensional information.

This allows label information to be added to the 3D point cloud, making it possible to use the output information for a variety of purposes.

According to this embodiment, label information is assigned to two-dimensional point clouds whose confidence level is equal to or greater than a predetermined reference value.

As a result, 2D point clouds with low confidence are usually point clouds where it is ambiguous whether an object exists or not, and it is possible to reduce the number of such 2D point clouds to be assigned label information. This makes it possible to improve the accuracy of assigning label information.

According to this embodiment, when generating a two-dimensional point cloud, three-dimensional information is further generated based on radar information, in which a three-dimensional point cloud is associated with a certainty, which is a value corresponding to the likelihood that an object exists in the three-dimensional point cloud. When multiple three-dimensional point clouds are projected onto a common position on a projection plane, the certainty of the two-dimensional point cloud obtained by projecting the multiple three-dimensional point clouds onto the projection plane is one of the following (1) to (3): (1) The certainty of the three-dimensional point cloud that is closest to the projection plane among the multiple three-dimensional point clouds. (2) The average value of the certainty of the multiple three-dimensional point clouds. (3) The maximum value of the certainty of the multiple three-dimensional point clouds.

This makes it easy to associate an appropriate confidence level with a two-dimensional point cloud, and to assign label information to the two-dimensional point cloud using this confidence level. Therefore, it becomes possible to assign label information to a two-dimensional point cloud with high accuracy and ease.

[Embodiment 2]
In this embodiment, a detailed example of the first acquisition unit 131 and the first process executed by the first acquisition unit 131 will be described.

The first acquisition unit 131 includes a radar information acquisition unit 131a, a three-dimensional information acquisition unit 131b, and a projection unit 131c, as shown in FIG. 7, for example.

The radar information acquisition unit 131a acquires radar information.

The three-dimensional information acquisition unit 131b acquires three-dimensional information related to the three-dimensional point cloud based on the radar information acquired by the radar information acquisition unit 131a.

The projection unit 131c uses the three-dimensional information acquired by the three-dimensional information acquisition unit 131b to generate two-dimensional information about a two-dimensional point cloud by projecting the three-dimensional point cloud onto a projection plane.

The first acquisition unit 131 executes a first acquisition process (step S101) as shown in FIG. 8, for example.

The radar information acquisition unit 131a acquires radar information from the transmission unit 113, for example, via the network NT1 (step S101a).

The three-dimensional information acquisition unit 131b acquires three-dimensional information related to the three-dimensional point cloud based on the radar information acquired in step S101a (step S101b).

The projection unit 131c uses the three-dimensional information acquired in step S101b to generate two-dimensional information about the two-dimensional point cloud by projecting the three-dimensional point cloud onto the projection plane (step S101c), and returns to information processing (see FIG. 3).

For example, when the projection unit 131c receives information for identifying the projection plane, it projects the three-dimensional point group onto the projection plane to generate a two-dimensional point group on the projection plane. The projection unit 131c assigns identification information to each of the two-dimensional point groups according to a predetermined rule. The projection unit 131c associates this identification information with the position of the two-dimensional point group on the projection plane. The projection unit 131c also determines the certainty of the two-dimensional point group based on the certainty associated with the three-dimensional point group corresponding to the two-dimensional point group. The projection unit 131c then generates two-dimensional information in which each of the two-dimensional point groups is associated with the identification information, the position of the two-dimensional point group on the projection plane, the corresponding three-dimensional point group, and a certainty that is a value corresponding to the likelihood that an object exists in the two-dimensional point group. Generating two-dimensional information to generate two-dimensional information.

(Action and Effects)
As described above, according to this embodiment, the first acquisition unit 131 includes the radar information acquisition unit 131a, the three-dimensional information acquisition unit 131b, and the projection unit 131c.

The radar information acquisition unit 131a acquires radar information. The three-dimensional information acquisition unit 131b acquires three-dimensional information about a three-dimensional point cloud based on the radar information acquired by the radar information acquisition unit 131a. The projection unit 131c uses the three-dimensional information acquired by the three-dimensional information acquisition unit 131b to generate two-dimensional information about a two-dimensional point cloud by projecting the three-dimensional point cloud onto a projection plane.

This makes it possible to generate two-dimensional information about a two-dimensional point cloud by projecting a three-dimensional point cloud onto a projection plane. Therefore, it becomes possible to easily generate two-dimensional information about a two-dimensional point cloud.

[Embodiment 3]
In this embodiment, a detailed example of the second acquisition unit 132 and the second process executed by the second acquisition unit 132 will be described.

The second acquisition unit 132 includes a captured image acquisition unit 132a, an image transformation unit 132b, and an image label acquisition unit 132c, as shown in FIG. 9, for example.

The captured image acquisition unit 132a acquires the captured image taken using light.

The image transformation unit 132b transforms the captured image acquired by the captured image acquisition unit 132a into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud.

The image label acquisition unit 132c acquires the label information assigned to the object contained in the two-dimensional captured image acquired by the image transformation unit 132b.

The second acquisition unit 132 executes a second acquisition process (step S102) as shown in FIG. 10, for example.

The captured image acquisition unit 132a acquires captured images from the imaging device 120, for example, via the network NT2 (step S102a).

The image transformation unit 132b transforms the captured image acquired in step S101a into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud (step S102b).

For example, the image transformation unit 132b transforms the captured image into a two-dimensional captured image so that the positions in real space of the pixels contained in the captured image coincide with the positions in real space of the two-dimensional point cloud.

The process of step S102b may be performed as necessary. For example, the process of step S102b may be performed when the coordinate systems of the captured image acquired in step S101a and the two-dimensional point cloud generated in step S101c are different, but may not be performed when they are the same.

The image label acquisition unit 132c acquires the label information assigned to the object contained in the two-dimensional captured image acquired in step S102b (step S102c), and returns to information processing (see FIG. 3).

In step S102c, the label information may be obtained by human input as described above, or may be obtained using an object detection model. In addition, since the two-dimensional captured image is an image obtained by deforming the captured image, the objects contained in the two-dimensional captured image and the captured image are the same. Therefore, in step S102c, instead of obtaining label information that has been applied to objects contained in the two-dimensional captured image, label information that has been applied to objects contained in the captured image may be obtained.

(Action and Effects)
As described above, according to this embodiment, the second acquiring unit 132 includes the captured image acquiring unit 132a, the image transforming unit 132b, and the image label acquiring unit 132c.

The captured image acquisition unit 132a acquires a captured image captured using light. The image transformation unit 132b transforms the acquired captured image into a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud. The image label acquisition unit 132c acquires label information assigned to objects included in the two-dimensional captured image.

This makes it possible to obtain label information attached to objects contained in two-dimensional captured images. Therefore, it becomes possible to easily obtain label information attached to objects.

The present disclosure has been described above with reference to the embodiments, but the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

In addition, in the multiple flowcharts used in the above explanation, multiple steps (processing) are described in order, but the order in which the steps are executed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that does not interfere with the content.

Some or all of the above embodiments can be described as follows, but are not limited to the following:

1.
a first acquisition means for generating a two-dimensional point cloud by converting the three-dimensional point cloud based on the radar information;
a second acquisition means for acquiring, based on a captured image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image;
an assignment means for assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
and output means for outputting output information in which the label information is added to a point cloud based on the radar information.
2.
2. The information processing device according to 1., wherein the two-dimensional point cloud is a point cloud obtained by projecting the three-dimensional point cloud onto a projection plane.
3.
The label information is further assigned to the three-dimensional point cloud;
3. The information processing device according to 1. or 2., wherein the output information is information in which the label information is added to the three-dimensional point cloud.
4.
The label information is further assigned to a point cloud within a specified specific region based on the two-dimensional point cloud to which the label information is assigned;
3. The information processing device according to 1. or 2., wherein the output information is information in which the label information is added to a point cloud within the specific region.
5.
5. The information processing device according to any one of 1. to 4., wherein the label information includes at least one of class identification information for identifying a class to which the object belongs and a class confidence indicating a likelihood that the object belongs to the class.
6.
In generating the two-dimensional point cloud, two-dimensional information is generated for associating, for each of the two-dimensional point clouds, a position on the projection plane, the corresponding three-dimensional point cloud, and a certainty value that is a value corresponding to the likelihood that an object exists;
2. The information processing device according to 1, wherein the label information is assigned to the two-dimensional point cloud based on the positional relationship, and is assigned to the three-dimensional point cloud associated with the two-dimensional point cloud by the two-dimensional information.
7.
7. The information processing apparatus according to 6., wherein the label information is assigned to the two-dimensional point cloud whose certainty is equal to or greater than a predetermined reference value.
8.
In the generating of the two-dimensional point cloud, three-dimensional information is further generated in which the three-dimensional point cloud is associated with a certainty factor, which is a value corresponding to the likelihood that an object exists in the three-dimensional point cloud, based on the radar information;
8. The information processing device according to claim 2, 6 or 7, wherein, when a plurality of the three-dimensional point clouds are projected onto a common position on the projection plane, the certainty factor for the two-dimensional point cloud obtained by projecting the plurality of three-dimensional point clouds onto the projection plane is any one of the certainty factor of a three-dimensional point cloud that is closest to the projection plane among the plurality of three-dimensional point clouds, the average value of the certainty factors of the plurality of three-dimensional point clouds, and the maximum value of the certainty factors of the plurality of three-dimensional point clouds.
9.
The first acquisition means is
A radar information acquisition means for acquiring the radar information;
a three-dimensional information acquisition means for acquiring three-dimensional information regarding the three-dimensional point cloud based on the acquired radar information;
3. The information processing apparatus according to 2., further comprising: a projection means for generating, using the three-dimensional information, two-dimensional information related to the two-dimensional point cloud by projecting the three-dimensional point cloud onto the projection plane.
10.
The second acquisition means is
An image capturing means for capturing an image captured using the light;
an image transformation means for transforming the acquired photographed image into a two-dimensional photographed image expressed in a coordinate system common to the two-dimensional point cloud;
and an image label acquisition unit that acquires the label information assigned to the object included in the two-dimensional captured image.
11.
a moving object that moves to generate the radar information by scanning a scanning area using a radar;
an imaging device for imaging the scanning region to generate the imaging image;
The information processing device according to any one of claims 1 to 10,
The moving body is
a transmitting means for transmitting the radar to the scanning area;
a receiving means for receiving the reflected wave of the transmitted radar and generating the radar information relating to the reflected wave;
and a transmitting means for transmitting the generated radar information.
12.
One or more computers
A two-dimensional point cloud is generated by converting the three-dimensional point cloud based on the radar information;
acquiring a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image based on a captured image captured using light;
assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
and outputting output information obtained by adding the label information to a point cloud based on the radar information.
13.
On one or more computers,
A two-dimensional point cloud is generated by converting the three-dimensional point cloud based on the radar information;
acquiring a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image based on a captured image captured using light;
assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
A program for executing the step of outputting output information in which the label information is added to a point cloud based on the radar information.

This application claims priority based on Japanese Patent Application No. 2023-129864, filed on August 9, 2023, the entire disclosure of which is incorporated herein by reference.

REFERENCE SIGNS LIST 100 Information processing system 110 Mobile body 111 Transmission unit 112 Reception unit 113 Transmission unit 120 Photographing device 130 Information processing device 131 First acquisition unit 131a Radar information acquisition unit 131b Three-dimensional information acquisition unit 131c Projection unit 132 Second acquisition unit 132a Photographed image acquisition unit 132b Image deformation unit 132c Image label acquisition unit 133 Assignment unit 134 Output unit

Claims (10)

a first acquisition means for generating a two-dimensional point cloud by converting the three-dimensional point cloud based on the radar information;
a second acquisition means for acquiring, based on a captured image captured using light, a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image;
an assignment means for assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
and output means for outputting output information in which the label information is added to a point cloud based on the radar information. The information processing apparatus according to claim 1 , wherein the two-dimensional point cloud is a point cloud obtained by projecting the three-dimensional point cloud onto a projection plane. The label information is further assigned to the three-dimensional point cloud;
The information processing apparatus according to claim 1 , wherein the output information is information in which the label information is added to the three-dimensional point cloud. The label information is further assigned to a point cloud within a specified specific region based on the two-dimensional point cloud to which the label information is assigned;
The information processing apparatus according to claim 1 , wherein the output information is information in which the label information is added to a point cloud within the specific region. The information processing device according to claim 1 , wherein the label information includes at least one of class identification information for identifying a class to which the object belongs and a class confidence indicating a likelihood that the object belongs to the class. In generating the two-dimensional point cloud, two-dimensional information is generated for associating, for each of the two-dimensional point clouds, a position on the projection plane, the corresponding three-dimensional point cloud, and a certainty value that is a value corresponding to the likelihood that an object exists;
The information processing apparatus according to claim 2 , wherein the label information is assigned to the two-dimensional point cloud based on the positional relationship, and is assigned to the three-dimensional point cloud associated with the two-dimensional point cloud by the two-dimensional information. In the generating of the two-dimensional point cloud, three-dimensional information is further generated in which the three-dimensional point cloud is associated with a certainty factor, which is a value corresponding to the likelihood that an object exists in the three-dimensional point cloud, based on the radar information;
7. The information processing device according to claim 2 or 6, wherein when a plurality of the three-dimensional point clouds are projected onto a common position on the projection plane, the certainty factor for the two-dimensional point cloud obtained by projecting the plurality of three-dimensional point clouds onto the projection plane is any one of the certainty factor of a three-dimensional point cloud that is closest to the projection plane among the plurality of three-dimensional point clouds, an average value of the certainty factors of the plurality of three-dimensional point clouds, and a maximum value of the certainty factors of the plurality of three-dimensional point clouds. a moving object that moves to generate the radar information by scanning a scanning area using a radar;
an imaging device for imaging the scanning region to generate the imaging image;
The information processing device according to claim 1 or 2,
The moving body is
a transmitting means for transmitting the radar to the scanning area;
a receiving means for receiving the reflected wave of the transmitted radar and generating the radar information relating to the reflected wave;
and a transmitting means for transmitting the generated radar information. One or more computers
A two-dimensional point cloud is generated by converting the three-dimensional point cloud based on the radar information;
acquiring a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image based on a captured image captured using light;
assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
and outputting output information obtained by adding the label information to a point cloud based on the radar information. On one or more computers,
A two-dimensional point cloud is generated by converting the three-dimensional point cloud based on the radar information;
acquiring a two-dimensional captured image expressed in a coordinate system common to the two-dimensional point cloud and label information assigned to an object included in the two-dimensional captured image based on a captured image captured using light;
assigning the label information to the two-dimensional point cloud based on a positional relationship between the two-dimensional point cloud and the object in the two-dimensional captured image;
A recording medium having a program recorded thereon for executing the process of outputting output information in which the label information is added to a point cloud based on the radar information.

Images