WO2021210725A1 - Apparatus and method for processing point cloud information - Google Patents

Abstract

An apparatus and a method for processing point cloud information are disclosed. An apparatus for processing point cloud information, according to one embodiment, comprises: a point cloud information acquisition unit for acquiring 3D point cloud information about a 3D space; an additional information acquisition unit for acquiring at least one additional image obtained by photographing at least a part of the 3D space, and direction information indicating the direction of gravity in a coordinate system in which the at least one additional image is captured; and a processing unit for transforming, on the basis of the at least one additional image and the direction information, the coordinate system of the 3D point cloud information so that one axis thereof coincides with the direction of gravity, and displaying the 3D point cloud information by using the transformed coordinate system of the 3D point cloud information.

Description

Point cloud information processing apparatus and method

Disclosed embodiments relate to a technology for processing point cloud (Point Cloud) information for a three-dimensional space.

SfM (Structure from Motion) algorithm is used as one of the methods for reconstructing a 3D structure of a corresponding area from a 2D image taken of an area such as a large indoor space or an outdoor space. The SfM algorithm has a process of extracting feature points from images, a process of matching feature points between images, and a process of reconstructing a three-dimensional point cloud by triangulating the matched feature points. In addition, various types of SfM algorithms exist according to detailed differences in each process.

However, since the coordinates of each point in the 3D point cloud restored as a result of the above-described SfM algorithm only provide information on the relative positions of each other, there is a problem in that it is not possible to clearly indicate where each point points in real space. .

The disclosed embodiments are for processing the 3D point cloud information so that the previously generated 3D point cloud information can indicate a location in real space according to the direction of gravity.

An apparatus for processing point cloud information according to an embodiment includes a point cloud information acquisition unit configured to acquire 3D point cloud information for a 3D space, one or more additional images obtained by photographing at least a portion of the 3D space, and the one or more An additional information obtaining unit for obtaining direction information indicating the direction of gravity on the coordinate system in which the additional image is captured, and one axis of the coordinate system of the three-dimensional point cloud information based on the one or more additional images and the direction information to match the direction of gravity and a processing unit that transforms and displays the 3D point cloud information using a coordinate system of the transformed 3D point cloud information.

The additional information obtaining unit may obtain sensing information in which the direction of gravity is detected, and may obtain the direction information by reflecting the sensing information in a coordinate system in which the one or more additional images are captured.

The processing unit may extract a plurality of feature points from the one or more additional images, map each of the plurality of feature points and each point in the 3D point cloud information, and the one or more additional images are captured from the mapping result Transformation relation information between the coordinate system and the coordinate system of the 3D point cloud information may be calculated, and one axis of the coordinate system of the 3D point cloud information may be transformed based on the transformation relation information to match the direction of gravity.

The transformation relationship information may include a rotation relationship between a coordinate system in which the one or more additional images are captured and a coordinate system of the 3D point cloud information. The rotation relationship may be expressed as a three-dimensional rotation matrix or quaternion, or a Euler angle, or an Axis-angle, or other methods.

The method for processing point cloud information according to an embodiment includes obtaining three-dimensional point cloud (Point Cloud) information for a three-dimensional space; obtaining direction information indicating the direction of gravity on a photographed coordinate system; converting one axis of the coordinate system of the three-dimensional point cloud information to match the direction of gravity based on the one or more additional images and the direction information; and the transformation and displaying the 3D point cloud information using the coordinate system of the 3D point cloud information.

The obtaining of the additional image and the direction information may include obtaining sensing information in which the direction of gravity is detected, and obtaining the direction information by reflecting the sensing information in a coordinate system in which the one or more additional images are captured. have.

The converting may include extracting a plurality of feature points from the one or more additional images, mapping each of the plurality of feature points with each point in the 3D point cloud information, and capturing the one or more additional images from the mapping result. Calculating transformation relationship information between the coordinate system and the coordinate system of the three-dimensional point cloud information, and converting one axis of the coordinate system of the three-dimensional point cloud information to match the direction of gravity based on the transformation relationship information. can

The transformation relationship information may include a rotation relationship between a coordinate system in which the one or more additional images are captured and a coordinate system of the 3D point cloud information. The rotation relationship may be expressed as a three-dimensional rotation matrix or quaternion, or a Euler angle, or an Axis-angle, or other methods.

When a computer program stored in a non-transitory computer-readable storage medium according to an embodiment is executed by a computing device having one or more processors, it acquires three-dimensional point cloud information for a three-dimensional space, and the three-dimensional space At least one additional image obtained by photographing at least a part of and direction information indicating the direction of gravity on the coordinate system in which the one or more additional images are captured, and based on the at least one additional image and the direction information, the coordinate system of the three-dimensional point cloud information It may include one or more commands for transforming one axis of α to coincide with the direction of gravity, and displaying the 3D point cloud information using a coordinate system of the transformed 3D point cloud information.

According to the disclosed embodiments, by processing the point cloud information for the three-dimensional space of the corresponding area using the image taken in the target area and the gravitational direction information on the coordinate system in which the corresponding image is captured, the direction of gravity is determined. The point cloud information on the 3D space can be expressed through the coordinate system used as the axis, so that the point cloud information on the 3D space can be scalably utilized for various applications.

1 is a block diagram illustrating an apparatus for processing point cloud information according to an embodiment;

2 is a flowchart illustrating a method of processing point cloud information according to an embodiment;

3 is a flowchart illustrating step 206 in more detail according to an embodiment.

4 is a block diagram illustrating and describing a computing environment including a computing device suitable for use in example embodiments;

Hereinafter, specific embodiments will be described with reference to the drawings. The following detailed description is provided to provide a comprehensive understanding of the methods, apparatus, and/or systems described herein. However, this is merely an example and the disclosed embodiments are not limited thereto.

In describing the embodiments, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the disclosed embodiments, the detailed description thereof will be omitted. And, the terms to be described later are terms defined in consideration of functions in the disclosed embodiments, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification. The terminology used in the detailed description is for the purpose of describing the embodiments only, and should in no way be limiting. Unless explicitly used otherwise, expressions in the singular include the meaning of the plural. In this description, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, acts, elements, some or a combination thereof, one or more other than those described. It should not be construed to exclude the presence or possibility of other features, numbers, steps, acts, elements, or any part or combination thereof.

1 is a block diagram illustrating an apparatus 100 for processing point cloud information according to an exemplary embodiment. Referring to FIG. 1 , an apparatus 100 for processing point cloud information according to an embodiment includes a point cloud information acquisition unit 102 , an additional information acquisition unit 104 , and a processing unit 106 .

The point cloud information acquisition unit 102 acquires 3D point cloud information for a 3D space.

In the disclosed embodiments, the 'three-dimensional space' may mean an area having an arbitrary range in an outdoor or indoor environment. In addition, '3D point cloud information' refers to information obtained by reconstructing a corresponding 3D space based on a 2D image obtained by photographing the above-described 3D space.

In this case, the 3D point cloud information may include a plurality of points corresponding to structures such as buildings, objects, and living things in the above-described 3D space and a descriptor corresponding to each point. Specifically, the descriptor may be a vector for expressing peripheral characteristics of each point in a three-dimensional space.

In an embodiment, the point cloud information acquisition unit 102 may acquire 3D point cloud information for the 3D space using a predetermined point cloud information generation algorithm, for example, the aforementioned SfM algorithm. In another embodiment, the point cloud information acquisition unit 102 may acquire the 3D point cloud information by receiving the 3D point cloud information calculated by another computing device using a wired or wireless communication means.

The three-dimensional point cloud information acquired by the point cloud information obtaining unit 102 is expressed using a coordinate system for an arbitrary direction, not a direction coincident with the actual direction of gravity. That is, the coordinates of each point included in the 3D point cloud information have only relative meanings, and it is impossible to determine which position each point points to in real space from the coordinates of each point.

The additional information obtaining unit 104 obtains one or more additional images obtained by photographing at least a portion of the three-dimensional space, and direction information indicating a direction of gravity on a coordinate system in which the one or more additional images are photographed.

At this time, the 'coordinate system in which the additional image was captured' may be determined depending on which position and at what inclination (direction) the image capturing means for capturing the additional image took the photo, and the origin means the position at which the image was captured. can

In an embodiment, the additional information obtaining unit 104 captures at least a portion of the above-described three-dimensional space with an image capturing means such as a camera, downloads an image previously uploaded to a web site, or performs web crawling ( Additional images may be obtained by extracting images through web crawling) or by obtaining images through a satellite map service provided by a web site (eg, Google's Street View). That is, it is possible to use an image obtained from the web without directly photographing the above-described three-dimensional space.

In an embodiment, the additional information acquisition unit 104 may acquire the direction information by acquiring sensing information in which the direction of gravity is detected, and reflecting the sensing information in a coordinate system in which the one or more additional images are captured.

In this case, the 'sensing information' is information obtained by detecting the direction of gravity by using separate software or a separate sensor when an additional image is acquired, and may be a virtual coordinate system including information on the direction of gravity. Illustratively, the sensing information may be generated through the method (1) or (2) below, but is not limited thereto, and the method of generating the sensing information may be changed according to software or a sensor.

(1) The floor is detected using software that detects the floor in 3D space (eg, Apple's ARKit, Google's ARCore, etc.). Thereafter, when a coordinate system based on the detected floor is formed by the software, one axis of the formed coordinate system coincides with the direction of gravity.

(2) A direction of gravity is detected in a three-dimensional space using an accelerometer, a gyroscope sensor, or a geomagnetic sensor built into the image capturing means. Thereafter, the sensor may form a coordinate system such that the detected direction of gravity coincides with the direction of one axis, or may form an arbitrary coordinate system including a unit vector of the detected direction of gravity in a coordinate space.

When referring to the description of the 'coordinate system in which the additional image is captured' and the 'sensing information' described above, the principle that the additional information obtaining unit 104 reflects the sensing information to the coordinate system in which the additional image is captured is as follows.

The sensing information generated through the method (1) or (2) becomes a coordinate system in which one axis coincides with the direction of gravity or an arbitrary coordinate system including a unit vector in the direction of gravity in the coordinate space. Let's call this coordinate system a 'sensed coordinate system'.

When the 'sensed coordinate system' is a coordinate system in which one axis coincides with the direction of gravity, the additional information obtaining unit 104 determines that one axis of the 'coordinate system in which the additional image is captured' coincides with the direction of gravity of the 'sensed coordinate system' Transform so that it is in the same direction as the axis.

On the other hand, when the 'sensed coordinate system' is an arbitrary coordinate system including a unit vector in the direction of gravity in the coordinate space, the additional information obtaining unit 104 determines that one axis of the 'coordinate system in which the additional image is captured' is indicated by the unit vector. change to the direction.

The processing unit 106 converts one axis of the coordinate system of the three-dimensional point cloud information to coincide with the direction of gravity based on the one or more additional images and the direction information obtained through the additional information obtaining unit 104 .

In an embodiment, the processing unit 106 may extract a plurality of feature points from the one or more additional images.

Specifically, the processing unit 106 may extract an end point of a line segment, a corner of a polygon, or the like indicating the characteristics of the additional image as a feature point. At this time, in order to extract the feature points, the processing unit 106 includes a scale-invariant feature transform (SIFT), speeded-up robust features (SURF), features from accelerated segment test (FAST), oriented fast and rotated brief (ORB), etc. Any one of the feature point extraction algorithms may be used, but the present invention is not limited thereto.

Also, the processing unit 106 may map each of the plurality of feature points extracted from the one or more additional images and each point in the 3D point cloud information. In an embodiment, the processing unit 106 may perform the mapping by matching a descriptor of each of the plurality of extracted feature points with a descriptor of each point in the 3D point cloud information.

Subsequently, the processing unit 106 may calculate transformation relationship information between the coordinate system in which the one or more additional images are captured and the coordinate system of the 3D point cloud information from the mapping result.

Specifically, the transformation relation information may include, but is not limited to, relation information on the coordinate axis direction between the coordinate system in which the one or more additional images are captured and the coordinate system of the three-dimensional point cloud information, and the one or more additional images are It may further include relationship information about the position of the origin of each coordinate system of the captured coordinate system and the 3D point cloud information.

In more detail, the transformation relationship information may include a rotation relationship between a coordinate system in which the one or more additional images are captured and a coordinate system of 3D point cloud information.

In this case, the rotation relationship may be expressed in a three-dimensional rotation matrix or Quaternion, or Euler angle, or Axis-angle, or other methods. The 'dimensional rotation matrix' refers to a matrix that rotates one coordinate system in a three-dimensional space around an origin based on its element values.

For example, when the rotation relationship is a three-dimensional rotation matrix, the processing unit 106 may use a mapping pair of a feature point in the one or more additional images generated as a result of the mapping and each point in the three-dimensional point cloud information to the one or more A rotation matrix between the coordinate system in which the additional image is captured and the coordinate system of the 3D point cloud information may be calculated.

In an embodiment, the processing unit 106 may use various perspective-n-point (PnP) algorithms to calculate the transformation relation information from the mapping result. Illustratively, applicable PnP algorithms include, but are not limited to, a P3P algorithm, an efficient PnP (EPnP) algorithm, and the like, and any algorithms may be used if the transformation relationship information can be calculated from the mapping result. .

Subsequently, the processing unit 106 may transform one axis of the coordinate system of the 3D point cloud information to match the direction of gravity based on the transformation relationship information.

In an embodiment, the processing unit 106 may rotate the coordinate system of the 3D point cloud information based on the rotation relationship so that one axis of the coordinate system of the 3D point cloud information coincides with the direction of gravity of the coordinate system in which the additional image is captured.

After performing the transformation, the processing unit 106 displays the 3D point cloud information by using the coordinate system of the transformed 3D point cloud information to determine where the coordinates of each point in the 3D point cloud information represent in real space. can be made clear Accordingly, naturally, 3D point cloud information will be able to be easily utilized in actual industrial fields such as virtual reality and autonomous driving.

2 is a flowchart 200 for explaining a method of processing point cloud information according to an embodiment. The method illustrated in FIG. 2 may be performed, for example, by the above-described point cloud information processing apparatus 100 .

Referring to FIG. 2 , first, in step 202 , the point cloud information acquisition unit 102 acquires 3D point cloud information for a 3D space.

Subsequently, in step 204, the additional information obtaining unit 104 obtains one or more additional images obtained by photographing at least a portion of the three-dimensional space, and direction information indicating the direction of gravity on the coordinate system in which the one or more additional images are photographed.

Then, in step 206 , the processing unit 106 converts one axis of the coordinate system of the three-dimensional point cloud information to match the direction of gravity based on the additional image and sensor information obtained in step 204 .

Thereafter, in step 208, the processing unit 106 displays the 3D point cloud information by using the coordinate system of the transformed 3D point cloud information.

In the illustrated flowchart, the method is described by dividing the method into a plurality of steps 202 to 208, but at least some of the steps are performed in a different order, are performed in combination with other steps, are omitted, or are performed by dividing into detailed steps, or One or more steps not shown may be added and performed.

3 is a flowchart 300 for explaining step 206 in more detail according to an embodiment. The method shown in FIG. 3 may be performed, for example, by the above-described processing unit 106, but is not necessarily limited thereto.

Referring to FIG. 3 , first, in step 302 , the processing unit 106 may extract a plurality of feature points from the one or more additional images acquired by the additional information acquiring unit 104 .

Then, in step 304 , the processing unit 106 may map each of the extracted feature points and each point in the 3D point cloud information.

Subsequently, in step 306 , the processing unit 106 may calculate transformation relationship information between the coordinate system in which the one or more additional images are captured and the coordinate system of the 3D point cloud information from the mapping result.

Thereafter, in step 308 , the processing unit 106 may transform one axis of the coordinate system of the 3D point cloud information to coincide with the direction of gravity based on the transformation relationship information.

In the illustrated flowchart, the method is divided into a plurality of steps 302 to 308, but at least some of the steps are performed in a different order, are performed in combination with other steps, are omitted, or are performed in separate steps, or One or more steps not shown may be added and performed.

4 is a block diagram illustrating and describing a computing environment 10 including a computing device suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components in addition to those described below.

The illustrated computing environment 10 includes a computing device 12 . In an embodiment, the computing device 12 may be the point cloud information processing device 100 .

Computing device 12 includes at least one processor 14 , computer readable storage medium 16 , and communication bus 18 . The processor 14 may cause the computing device 12 to operate in accordance with the exemplary embodiments discussed above. For example, the processor 14 may execute one or more programs stored in the computer-readable storage medium 16 . The one or more programs may include one or more computer-executable instructions that, when executed by the processor 14, configure the computing device 12 to perform operations in accordance with the exemplary embodiment. can be

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 20 stored in the computer readable storage medium 16 includes a set of instructions executable by the processor 14 . In one embodiment, computer-readable storage medium 16 includes memory (volatile memory, such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash It may be memory devices, other forms of storage medium accessed by computing device 12 and capable of storing desired information, or a suitable combination thereof.

Communication bus 18 interconnects various other components of computing device 12 , including processor 14 and computer readable storage medium 16 .

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide interfaces for one or more input/output devices 24 . The input/output interface 22 and the network communication interface 26 are coupled to the communication bus 18 . Input/output device 24 may be coupled to other components of computing device 12 via input/output interface 22 . Exemplary input/output device 24 may include a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices, and/or imaging devices. input devices and/or output devices such as display devices, printers, speakers and/or network cards. The exemplary input/output device 24 may be included in the computing device 12 as a component constituting the computing device 12 , and may be connected to the computing device 12 as a separate device distinct from the computing device 12 . may be

Meanwhile, an embodiment of the present invention may include a program for performing the methods described in this specification on a computer, and a computer-readable recording medium including the program. The computer-readable recording medium may include program instructions, local data files, local data structures, etc. alone or in combination. The medium may be specially designed and configured for the present invention, or may be commonly used in the field of computer software. Examples of computer-readable recording media include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and program instructions specially configured to store and execute program instructions such as ROMs, RAMs, flash memories, and the like. Hardware devices are included. Examples of the program may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler.

Although representative embodiments of the present invention have been described in detail above, those of ordinary skill in the art to which the present invention pertains will understand that various modifications are possible without departing from the scope of the present invention with respect to the above-described embodiments. . Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the following claims as well as the claims and equivalents.

Claims (9)

a point cloud information acquisition unit that acquires 3D point cloud information for a 3D space; an additional information acquisition unit configured to acquire one or more additional images obtained by photographing at least a portion of the three-dimensional space and direction information indicating a direction of gravity on a coordinate system in which the one or more additional images are photographed; and One axis of the coordinate system of the 3D point cloud information is transformed to match the direction of gravity based on the one or more additional images and the direction information, and the 3D point cloud information is obtained using the coordinate system of the transformed 3D point cloud information. A point cloud information processing apparatus including a processing unit for indicating. The method according to claim 1, The additional information obtaining unit, A point cloud information processing apparatus for acquiring the sensing information in which the direction of gravity is detected, and acquiring the direction information by reflecting the sensing information to a coordinate system in which the one or more additional images are captured. The method according to claim 1, The processing unit, Extracting a plurality of feature points from the one or more additional images, mapping each of the plurality of feature points and each point in the 3D point cloud information, and the coordinate system in which the one or more additional images are captured and the 3D point cloud information from the mapping result A point cloud information processing apparatus for calculating transformation relationship information between coordinate systems of , and converting one axis of a coordinate system of the three-dimensional point cloud information to coincide with the direction of gravity based on the transformation relationship information. 4. The method according to claim 3, The transformation relationship information is A point cloud information processing apparatus including a rotational relationship between a coordinate system in which the one or more additional images are captured and a coordinate system of the three-dimensional point cloud information. obtaining 3D point cloud information for a 3D space; obtaining one or more additional images obtained by photographing at least a portion of the three-dimensional space and direction information indicating a direction of gravity on a coordinate system in which the one or more additional images are photographed; converting one axis of the coordinate system of the three-dimensional point cloud information to coincide with the direction of gravity based on the one or more additional images and the direction information; and and displaying the 3D point cloud information using a coordinate system of the converted 3D point cloud information. 6. The method of claim 5, The step of obtaining the additional image and the direction information comprises: A method for processing point cloud information, acquiring sensing information in which the direction of gravity is detected, and acquiring the direction information by reflecting the sensing information in a coordinate system in which the one or more additional images are captured. 6. The method of claim 5, The converting step is extracting a plurality of feature points from the one or more additional images; mapping each of the plurality of feature points and each point in the 3D point cloud information; calculating transformation relationship information between a coordinate system in which the one or more additional images are captured and a coordinate system of the 3D point cloud information from the mapping result; and and transforming one axis of the coordinate system of the three-dimensional point cloud information to coincide with the direction of gravity based on the transformation relationship information. 8. The method of claim 7, The transformation relationship information is A method for processing point cloud information, including a rotational relationship between a coordinate system in which the one or more additional images are captured and a coordinate system of the three-dimensional point cloud information. As a computer program stored in a non-transitory computer readable storage medium, The computer program includes one or more instructions, which, when executed by a computing device having one or more processors, cause the computing device to: Acquire 3D point cloud information about 3D space, At least one additional image obtained by photographing at least a portion of the three-dimensional space and direction information indicating the direction of gravity on the coordinate system in which the one or more additional images are photographed, Transform one axis of the coordinate system of the three-dimensional point cloud information to coincide with the direction of gravity based on the one or more additional images and the direction information; A computer program stored in a non-transitory computer-readable storage medium to represent the three-dimensional point cloud information by using the coordinate system of the converted three-dimensional point cloud information.

Images