JP7563921B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

The following Patent Document 1 discloses an information processing device that creates a three-dimensional model of a structure from an input image showing the structure. The three-dimensional model of the structure is useful for infrastructure inspection of the structure.

JP 2016-57792 A

When digital images of large infrastructure structures such as tunnels and bridges are taken at a resolution that allows damage to the concrete to be detected, creating a three-dimensional model based on the images will result in a very large amount of information. This will increase the costs of storing and archiving the model. Furthermore, to handle such a large amount of data, highly sophisticated information processing equipment will be required.

The present invention was made in consideration of these circumstances, and its purpose is to reduce the amount of information in a 3D model of a structure while maintaining the quality of data required for infrastructure inspection.

(1) One aspect of the present invention is an information processing device that creates a three-dimensional model of a structure from a two-dimensional image of the structure, the information processing device including: a model creation unit that, in creating the three-dimensional model, models abnormal locations of the structure using first point cloud data and models areas of the structure other than the abnormal locations using second point cloud data that is lower in density than the first point cloud data, and creates a three-dimensional model of the structure from the two-dimensional image of the structure using the first point cloud data; an acquisition unit that acquires damage point cloud data, which is the first point cloud data of the abnormal locations of the structure, from the three-dimensional model; a compression unit that generates the three-dimensional model of the second point cloud data by compressing the first point cloud data of the three-dimensional model after the damage point cloud data has been acquired by the acquisition unit; and a processing unit that reflects the damage point cloud data in the three-dimensional model generated by the compression unit .

( 2 ) In the information processing device of ( 1 ) above, the processing unit may reflect the damage point cloud data only in an area corresponding to the abnormal location among the entire area of the three-dimensional model generated by the compression unit.

( 3 ) The information processing device of ( 1 ) or ( 2 ) may further include an anomaly identification unit that identifies an abnormality location of a structure from a two-dimensional image of the structure, and the acquisition unit may acquire, as the damage point cloud data, the first point cloud data of a region of a three-dimensional model corresponding to the abnormality location identified by the anomaly identification unit.

( 4 ) One aspect of the present invention is an information processing method for a computer, comprising: a model creation step of creating a three-dimensional model, which is first point cloud data of a structure, from a two-dimensional image of the structure; an acquisition step of acquiring damage point cloud data, which is the first point cloud data of abnormal locations of the structure, from the three-dimensional model; a compression step of compressing the first point cloud data of the three-dimensional model after the damage point cloud data has been acquired in the acquisition step; and a processing step of reflecting the damage point cloud data in the three-dimensional model compressed in the compression step.

( 5 ) One aspect of the present invention is a program that causes a computer to function as the information processing device according to any one of (1) to ( 3 ) above.

As described above, according to the present invention, when creating a three-dimensional model of a structure, it is possible to reduce the amount of information in the three-dimensional model while maintaining the quality of data required for infrastructure inspection.

FIG. 2 is a diagram illustrating an example of the configuration of an information processing device 3 according to the present embodiment. 5A and 5B are diagrams showing an example of a digital image captured by an information processing device 3 according to the present embodiment. FIG. 2 is a diagram showing a three-dimensional model created by a model creation unit according to the present embodiment. FIG. 4 is an operation flow diagram of the information processing device 3 according to the present embodiment. FIG. 2 is a hardware configuration diagram of an information processing device 3 according to the present embodiment.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In the drawings, the same reference numerals may be used to denote identical or similar parts, and duplicate explanations may be omitted. In addition, the shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

Throughout the specification, when a part "includes," "has," or "includes" certain elements, this does not mean that it excludes other elements, but that it may further include other elements, unless specifically stated to the contrary.

The information processing device, information processing method, and program according to this embodiment will be described below with reference to the drawings.

Figure 1 is a diagram showing an example of a schematic configuration of an infrastructure management system 1 including an information processing device 3 according to this embodiment. As shown in Figure 1, the infrastructure management system 1 includes an imaging unit 2 and an information processing device 3.

The imaging unit 2 captures an image of the structure W that is the subject of the infrastructure inspection, thereby acquiring a two-dimensional image of the structure W. That is, the imaging unit 2 captures an image of the structure W, thereby acquiring a digital image of the structure W. The digital image captured by the imaging unit 2 may be transmitted directly or indirectly to the information processing device 3. For example, the imaging unit 2 may have a communication function for transmitting information to the information processing device 3 via a communication network. For example, the imaging unit 2 is a digital camera.

For example, the imaging unit 2 may be connected to the information processing device 3 by wire or wirelessly, and may transmit digital images to the information processing device 3 by wire or wirelessly. However, this is not limited, and the digital image captured by the imaging unit 2 may be input to the information processing device 3, and the input method is not particularly limited. For example, the user may retrieve the digital image from the imaging unit 2 using an external storage medium such as a USB (Universal Serial Bus) memory, and input the digital image by connecting the recording medium to the information processing device 3. The digital image captured by the imaging unit 2 may also be stored in a server device. The server device includes one or more physical servers. The multiple physical servers are connected to each other via a communication network, and thus can communicate with each other. The physical server may include one or more virtual servers. The virtual server may be a server on a cloud system, i.e., a cloud server.

For example, structure W is a structure that requires inspection (infrastructure inspection) of the condition of the structure, such as the presence or absence of abnormalities such as deterioration or damage, and the degree of progression of the abnormalities. Structure W is an infrastructure structure such as a road, dam, building, tunnel, or bridge. For example, structure W is a concrete structure.

The information processing device 3 digitizes and manages the state of the structure W. The information processing device 3 generates a three-dimensional model of the structure W from the digital image (input image) captured by the imaging unit 2. This three-dimensional model is three-dimensional point cloud data. The information processing device 3 only needs to generate a three-dimensional model of point cloud data from the digital image, and there is no particular limitation on the method of generation. For example, the information processing device 3 is a computer.

In the three-dimensional model, the information processing device 3 represents abnormal locations of the structure W with high-density point cloud data, and represents non-abnormal locations with low-density point cloud data. Specifically, when creating a three-dimensional model of the structure W from a digital image of the structure W, the information processing device 3 models the abnormal locations of the structure W using first point cloud data, and models the areas of the structure W other than the abnormal locations using second point cloud data that is lower density than the first point cloud data. Each functional unit of the information processing device 3 will be described below.

The information processing device 3 includes a reading unit 10, an anomaly identification unit 11, a model creation unit 12, an acquisition unit 13, a compression unit 14, and a processing unit 15. Note that the "unit" of the information processing device 3 means a unit that processes at least one function or operation, which may be embodied as hardware or software, or a combination of hardware and software.

The reading unit 10 reads one or more digital images captured by the imaging unit 2. For example, the digital image is of a concrete structure as shown in FIG. 2. The digital image is a two-dimensional image.

The anomaly identification unit 11 identifies an abnormal part of the structure W from a digital image of the structure W. For example, the anomaly identification unit 11 may identify a damaged part such as a crack as an abnormal part by applying a predetermined image processing to the digital image of the structure W. Note that the information processing device 3 of this embodiment is not particularly limited to a method of identifying an abnormal part, and may identify an abnormal part of the structure W from a digital image of the structure W using a known technology. In other words, the anomaly identification unit 11 is not particularly limited to a method of identifying an abnormal part of the structure W as long as it can identify an abnormal part of the structure W from a digital image of the structure W. As an example, the anomaly identification unit 11 may identify an abnormal part of the structure W from a digital image using a crack measurement method described in JP 2000-2523 A. Also, the anomaly identification unit 11 may identify an abnormal part of the structure W from a digital image using AI (artificial intelligence).

For example, the anomaly identification unit 11 has a trained model that, when a digital image is input, outputs an abnormal part of the structure W in the digital image. The trained model is generated by performing machine learning such as deep learning using a digital image of the structure W without an abnormal part and a digital image of the structure W with an abnormal part as training data. The trained model may be a convolutional neural network (CNN).

The model creation unit 12 generates a three-dimensional model of the structure W from one or more digital images of the structure W using the first point cloud data. As described above, the model creation unit 12 only needs to generate a three-dimensional model of point cloud data from a digital image, and the generation method is not particularly limited. As an example, the model creation unit 12 creates a three-dimensional model by restoring the three-dimensional shape of the structure W from multiple two-dimensional digital images using a technique called SfM (Structure from Motion). The model creation unit 12 of this embodiment generates a three-dimensional model of the structure W using SfM, which estimates the three-dimensional positions of feature points and the orientation of the imaging unit 2 from multiple digital images with different viewpoints, and MVS (Multi-View-Stereo), which generates a high-density point cloud (first point cloud data) from overlapping areas between images. Note that SfM and MVS are well-known techniques, so a detailed description will be omitted.

The acquisition unit 13 acquires damage point cloud data, which is first point cloud data of an abnormal location of the structure W, from the three-dimensional model created by the model creation unit 12. The acquisition unit 13 acquires the first point cloud data of the area of the three-dimensional model corresponding to the abnormal location identified by the abnormality identification unit 11 as damage point cloud data. Here, acquiring includes extracting the first point cloud data corresponding to the abnormal location from the three-dimensional model created by the model creation unit 12. Specifically, the acquisition unit 13 acquires three-dimensional coordinates of each point corresponding to the pixel position of the abnormal location identified by the abnormality identification unit 11 in the three-dimensional model created by the model creation unit 12. The three-dimensional coordinates of each point corresponding to the pixel position of the abnormal location are damage point cloud data.

After the damage point cloud data is acquired by the acquisition unit 13, the compression unit 14 generates a three-dimensional model of the second point cloud data by compressing the first point cloud data of the three-dimensional model created by the model creation unit 12. In this way, the compression unit 14 compresses the three-dimensional model consisting of high-density point cloud data created by the model creation unit 12 to create a lightweight three-dimensional model. The compression method by the compression unit 14 is not particularly limited, and known techniques can be used. As an example, the compression unit 14 thins out the first point cloud data while retaining characteristic information such as edges in the three-dimensional model created by the model creation unit 12. Then, the compression unit 14 creates a mesh for the thinned area to create a lightweight three-dimensional model. The lightweight three-dimensional model created by the compression unit 14 may be called a lightweight model.

The processing unit 15 reflects the damage point cloud data on the three-dimensional model (lightweight model) generated by the compression unit 14. The processing unit 15 registers damage information (damage point cloud data) using high-density point clouds on the lightweight model. The processing unit 15 reflects the damage point cloud data only on areas corresponding to abnormal locations among the entire area of the lightweight model generated by the compression unit 14. As an example, the processing unit 15 projects the damage point cloud data onto the lightweight model based on the relationship between the position of the imaging unit 2 acquired by SfM and the three-dimensional coordinates of the damaged location, and registers the damage point cloud data on the lightweight model, thereby reflecting the damage point cloud data on the lightweight model.

Next, the flow of operations of the information processing device 3 of this embodiment will be described with reference to FIG. 4. FIG. 4 is a flow diagram of operations of the information processing device 3 of this embodiment.

Multiple digital images of the structure W are input to the information processing device 3. The information processing device 3 reads the multiple input digital images (step S101). The anomaly identification unit 11 of the information processing device 3 identifies abnormal locations of the structure W from the multiple digital images of the structure W (step S102). Next, the model creation unit 12 generates a three-dimensional model of the structure W from the multiple digital images read in step S101 using the first point cloud data. As an example, the model creation unit 12 uses SfM to create a three-dimensional model of the structure W from the multiple two-dimensional digital images (step S103).

The acquisition unit 13 acquires damage point cloud data, which is the first point cloud data of the abnormal location of the structure W, from the three-dimensional model created in step S103. As an example, the acquisition unit 13 acquires the three-dimensional coordinates of each point corresponding to the pixel position of the abnormal location identified in step S102 from the three-dimensional model as damage point cloud data (step S104).

After step S104, the compression unit 14 compresses the first point cloud data of the three-dimensional model created by the model creation unit 12 to create a three-dimensional model of the second point cloud data, i.e., a lightweight model (step S105).

The processing unit 15 reflects the damage point cloud data in the lightweight model created in step S105 (step S106). As an example, the processing unit 15 projects the damage point cloud data onto the lightweight model based on the relationship between the position of the imaging unit 2 acquired by SfM in step S103 and the three-dimensional coordinates of the damaged area, and registers the damage point cloud data on the lightweight model, thereby reflecting the damage point cloud data in the lightweight model.

The processing unit 15 stores the lightweight model reflecting the damage point cloud data in the information processing device 3 (step S107).

Figure 5 shows an example of a hardware configuration of a computer 1000 functioning as an information processing device 3. A program installed on the computer 1000 can cause the computer 1000 to function as one or more "parts" of the information processing device 3 according to the above embodiment, or cause the computer 1000 to execute operations or one or more "parts" associated with the information processing device 3 according to the above embodiment, and/or cause the computer 1000 to execute a process or steps of the process according to the above embodiment. Such a program can be executed by the CPU 1012 to cause the computer 1000 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 1000 according to this embodiment includes a CPU 1012, a RAM 1014, and a graphics controller 1016, which are connected to each other by a host controller 1010. The computer 1000 also includes input/output units such as a communication interface 1022 and a storage device 1024, which are connected to the host controller 1010 via an input/output controller 1020. The storage device 1024 may be a hard disk drive, a solid state drive, or the like. The computer 1000 also includes input/output units such as a ROM 1030 and a touch panel, which are connected to the input/output controller 1020 via an input/output chip 1040.

The CPU 1012 operates according to the programs stored in the ROM 1030 and the RAM 1014, thereby controlling each unit. The graphics controller 1016 acquires image data generated by the CPU 1012 into a frame buffer or the like provided in the RAM 1014 or into itself, and causes the image data to be displayed on the display device 1018.

The communication interface 1022 communicates with other electronic devices over a network. The storage device 1024 stores programs and data used by the CPU 1012 in the computer 1000.

The ROM 1030 stores therein a boot program or the like that is executed by the computer 1000 upon activation, and/or a program that depends on the hardware of the computer 1000. The input/output chip 1040 may also connect various input/output units to the input/output controller 1020 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

The programs are provided by a computer-readable storage medium such as an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1024, RAM 1014, or ROM 1030, which are also examples of computer-readable storage media, and executed by the CPU 1012. The information processing described in these programs is read by the computer 1000, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by realizing the operation or processing of information according to the use of the computer 1000.

For example, when communication is performed between computer 1000 and an external device, CPU 1012 may execute a communication program loaded into RAM 1014 and instruct communication interface 1022 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1012, communication interface 1022 reads transmission data stored in a transmission buffer area provided in RAM 1014, storage device 1024, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 1012 may also cause all or a necessary portion of a file or database stored in the storage device 1024 or an external recording medium such as an IC card to be read into the RAM 1014, and perform various types of processing on the data on the RAM 1014. The CPU 1012 may then write back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1012 may perform various types of processing on the data read from the RAM 1014, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and writes back the results to the RAM 1014. The CPU 1012 may also search for information in a file, database, etc. in the recording medium. For example, when multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1012 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1000 or in the vicinity of the computer 1000. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1000 via the network.

The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and/or analog hardware circuits and may include integrated circuits (ICs) and/or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as, for example, field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, to cause the processor of the general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, to execute the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the present invention.

The imaging unit 2 described above may be fixed at a predetermined position or may be provided on a moving object. The moving object may be manned or unmanned. For example, the moving object is a small flying object such as a drone. The imaging unit 2 provided on the flying object may transmit the captured digital image to the information processing device 3 wirelessly or via a wired connection.

In the above-described information processing device 3, an example has been described in which the anomaly identification unit 11 identifies an abnormal part of the structure W from a digital image of the structure W. However, this is not limited to this, and the abnormal part of the structure W may be identified by a user. For example, the information processing device 3 displays a digital image on the display device 1018. The user selects an abnormal part of the structure W in the digital image using an operation device. The information processing device 3 may identify the abnormal part of the structure by setting the selected area as the abnormal part. The operation unit accepts input from the user and outputs information related to the input to the information processing device 3. For example, the operation unit is a touch panel, a hardware key such as a keyboard, or a pointing device such as a mouse. The operation unit may also accept operation input by voice from a microphone or the like. In this way, the abnormal part of the structure W may be identified manually from the digital image, or may be identified by computer software.

As described above, the information processing device 3 of this embodiment is an information processing device that creates a three-dimensional model of the structure W from a two-dimensional image of the structure W. In creating the three-dimensional model, abnormal areas of the structure W are modeled using first point cloud data, and areas of the structure W other than the abnormal areas are modeled using second point cloud data that is lower density than the first point cloud data.

With this configuration, when creating a three-dimensional model of a structure, the amount of information in the three-dimensional model can be reduced while maintaining the quality of data required for infrastructure inspection. Furthermore, by reducing the amount of information in the final three-dimensional model, the costs required for data visualization and editing can be reduced. Furthermore, the information processing device 3 can maintain the quality of data required for infrastructure inspection by controlling the density of the amount of information in the three-dimensional model.

Reference Signs List 1: Infrastructure management system 2: Imaging unit 3: Information processing device 10: Reading unit 11: Anomaly identification unit 12: Model creation unit 13: Acquisition unit 14: Compression unit 15: Processing unit

Claims (5)

1. An information processing device that creates a three-dimensional model of a structure from a two-dimensional image of the structure,
In creating the three-dimensional model, an abnormal portion of the structure is modeled using first point cloud data, and an area of the structure other than the abnormal portion is modeled using second point cloud data having a lower density than the first point cloud data ;
a model creation unit that creates a three-dimensional model of the structure from the two-dimensional image of the structure by using the first point cloud data;
an acquisition unit that acquires damage point cloud data, which is the first point cloud data of an abnormality location of the structure, from the three-dimensional model;
a compression unit that generates the three-dimensional model of the second point cloud data by compressing the first point cloud data of the three-dimensional model after the damage point cloud data is acquired by the acquisition unit;
a processing unit that reflects the damage point cloud data on the three-dimensional model generated by the compression unit;
An information processing device comprising : The processing unit reflects the damage point cloud data only in an area corresponding to the abnormality among the entire area of the three-dimensional model generated by the compression unit.
The information processing device according to claim 1 . The apparatus further includes an abnormality identification unit that identifies an abnormal portion of the structure from a two-dimensional image of the structure,
The acquisition unit acquires, as the damage point cloud data, the first point cloud data of a region of a three-dimensional model corresponding to the anomaly location identified by the anomaly identification unit.
3. The information processing device according to claim 1 or 2 . A computer information processing method, comprising:
a model creation step of creating a three-dimensional model, which is a first point cloud data of the structure, from the two-dimensional image of the structure;
an acquisition step of acquiring damage point cloud data, which is the first point cloud data of an abnormality location of the structure, from the three-dimensional model;
a compression step of compressing the first point cloud data of the three-dimensional model after the damage point cloud data is acquired in the acquisition step;
a processing step of reflecting the damage point cloud data on the three-dimensional model compressed in the compression step;
An information processing method comprising: A program for causing a computer to function as the information processing device according to any one of claims 1 to 3 .

Images