JP7605679B2 - System for diagnosing condition of inspection object, condition diagnosis device, and method for identifying image position - Google Patents

Description

The present invention relates to a technology for diagnosing the condition of an object to be inspected, and in particular to a condition diagnosis system, a condition diagnosis device, and a method for identifying the position of an image using an image of the object to be inspected.

Various methods for inspecting the condition of buildings have been proposed to prevent peeling and falling off of the exterior materials (exterior wall materials) of buildings. For example, Patent Document 1 discloses a condition assessment device that can diagnose large areas of exterior materials such as the exterior walls of buildings at low cost and efficiently. According to this device, a detection unit that performs inspection by striking the exterior wall is moved along the exterior wall plane, and the inspection position on the exterior wall is associated with the inspection results to generate condition assessment information for the entire exterior wall.

According to Patent Document 1, a camera installed above the frame holds in its field of view reference points installed at the four corners of the frame and an indicator installed on a detection unit that can move in the x and y directions within the frame, and captures an image using an impact on the detection unit as a trigger. Therefore, the inspection position of the detection unit can be identified from the positional relationship between the reference points and the indicator on the detection unit in the captured image, and the inspection position can be associated with its inspection result.

JP 2019-178953 A

In the above-mentioned Patent Document 1, the camera captures images according to the timing of the impact of the detection unit, so it is easy to obtain images synchronized with the inspection timing. However, when diagnosing the surface condition of an exterior wall using images, it is necessary to capture moving images by moving the camera close to the exterior wall to be inspected. No technology has been proposed to identify the inspection position based on moving images without a trigger signal such as an impact.

The object of the present invention is to provide a system, a device, and a method and program for diagnosing the condition of an object to be inspected, which can identify the imaging position based on a moving image of the object to be inspected.

In order to achieve the above-mentioned object, according to one embodiment of the present invention, a condition diagnosis system comprises a moving mechanism that moves a moving part along the surface of an object to be inspected within an imageable range, a camera attached to the moving part that captures a moving image of the surface of the object to be inspected, a coordinate acquisition unit that acquires the position coordinates of the moving part within the imageable range, and a data processing unit that detects an inspection image from the moving image, wherein the data processing unit detects a change point in the movement direction of the camera based on multiple images of the moving image, and detects an inspection image corresponding to the desired position coordinates from the moving image by matching the point where the change direction of the position coordinates of the moving part obtained from the coordinate acquisition unit turns around with the change point in the movement direction of the camera.
Furthermore, according to one embodiment of the present invention, the movement direction change point may be a point in the image of interest at which the movement direction of a specific image portion contained in the image of interest and two images that occur before and after the image of interest in time changes by more than a predetermined angle.
Furthermore, according to one embodiment of the present invention, an edge portion can be extracted from the inspection image, and a tile region of the inspection object can be extracted from the edge portion by a rectangle detection algorithm.
Furthermore, according to one embodiment of the present invention, the data processing unit can evaluate the surface condition of the object to be inspected based on tile regions extracted from the inspection image.
Furthermore, according to one embodiment of the present invention, the data processing unit can detect crack lines in at least one tile region from the inspection image by image processing.
In addition, according to one embodiment of the present invention, the data processing unit can connect the end points of the first crack line and the end points of the second crack line if they are within a predetermined distance and a predetermined angle from each other, and detect the first and second crack lines as a single crack line.
Moreover, according to one embodiment of the present invention, the device further includes a percussion unit attached to the movable part, which strikes the surface of the object to be inspected and outputs the striking sound to the data processing part as an inspection signal.
Moreover, according to one embodiment of the present invention, a condition diagnosis device is provided, comprising: a movement control unit that controls a movement mechanism that moves a moving unit along the surface of an object to be inspected within an imageable range; an interface that connects via wired or wireless connection to a camera that is attached to the moving unit and captures a moving image of the surface of the object to be inspected; a coordinate acquisition unit that acquires position coordinates of the moving unit within the imageable range; and a data processing unit that detects an inspection image from the moving image, wherein the data processing unit detects a change point in the movement direction of the camera based on a plurality of images of the moving image, and detects an inspection image corresponding to the desired position coordinates from the moving image by matching a point where the change direction of the position coordinates of the moving unit obtained from the coordinate acquisition unit turns around with the change point in the movement direction of the camera.
Moreover, according to one embodiment of the present invention, there is provided a method for identifying an image position in a condition diagnosis device having a movement control unit that controls a movement mechanism that moves a moving unit along the surface of an object to be inspected within an imageable range, and an interface that is attached to the moving unit and connects via a wired or wireless connection to a camera that captures moving images of the surface of the object to be inspected, wherein a coordinate acquisition unit acquires position coordinates of the moving unit within the imageable range, and a data processing unit detects a change point in the movement direction of the camera based on a plurality of images of the moving image, and identifies an image corresponding to the desired position coordinates from the moving image by matching a point where the change direction of the position coordinates of the moving unit obtained from the coordinate acquisition unit turns around with the change point in the movement direction of the camera.
Moreover, according to one embodiment of the present invention, a program is provided that causes a computer to function as a condition diagnosis device having a movement control unit that controls a movement mechanism that moves a moving unit along the surface of an object to be inspected within an imageable range, and an interface that connects via wired or wireless connection to a camera that is attached to the moving unit and captures moving images of the surface of the object to be inspected, and is characterized in that the program implements in the computer a function of acquiring position coordinates of the moving unit within the imageable range, a function of detecting a change point in the movement direction of the camera based on multiple images of the moving image, and a function of identifying an image corresponding to desired position coordinates from the moving image by matching a point where the change direction of the position coordinates of the moving unit turns around with a change point in the movement direction of the camera.

A condition diagnosis system according to one embodiment of the present invention detects points of change in the direction of movement from a moving image and corresponds them to points where the direction of change in the position coordinates obtained from the camera's position coordinates turns around , so that even in a system in which the moving image obtained from the camera and the movement control that moves the camera are not synchronized, it is possible to know at which position on the object being inspected the image in the moving image corresponds and to identify the shooting position of the image based on the image of the object being inspected.
According to one embodiment of the present invention, a change in the movement direction of a specific image portion included in a plurality of images can be easily detected based on an angle change.
According to one embodiment of the present invention, it is possible to reliably extract tile areas from an inspection image by detecting edges of the inspection image and extracting the tile areas using a rectangle detection algorithm.
According to one embodiment of the present invention, surface conditions can be assessed based on images of the extracted tile regions.
According to one embodiment of the present invention, crack lines can be detected by image processing.
According to one embodiment of the present invention, multiple crack lines on an image can be reconnected taking into account their directions, and multiple crack lines can be detected as a single crack line.
According to one embodiment of the present invention, a percussion unit capable of striking and inspecting the surface of the inspection object can be moved in the same manner as a camera, and the entire area of the inspection object 10 can be inspected efficiently.
A condition diagnosis device according to an embodiment of the present invention has an interface to which a camera can be connected, inputs video from the connected camera, and associates a movement direction change point detected from the video with a point where the change direction of the position coordinate obtained from the position coordinate of the camera body turns around , so that even in a system in which the video obtained from the camera and the movement control for moving the camera are not synchronized, it is possible to know which position of the inspection object the image in the video is, and to specify the shooting position of the image based on the image of the inspection object. In addition, since a camera can be connected via an interface, there is an advantage that inspection can be performed with a camera with performance suitable for the environment and the state of the inspection object.
A method for identifying the position of an image according to one embodiment of the present invention detects a point of change in the direction of movement from a moving image and corresponds it to a point where the direction of change in the position coordinates obtained from the position coordinates of the camera turns around , so that even in a system in which the moving image obtained from the camera and the movement control that moves the camera are not synchronized, it is possible to know at which position on the object to be inspected an image in the moving image corresponds and to identify the shooting position of the image based on the image of the object to be inspected.
A program according to one embodiment of the present invention detects points of change in the direction of movement from a moving image and associates them with points where the direction of change in position coordinates obtained from the camera's position coordinates turns around , making it possible to realize a condition diagnosis device on a computer that can identify the shooting position of an image based on an image of an object to be inspected, even in a system in which the moving image obtained from the camera and the movement control that moves the camera are not synchronized.

FIG. 1 is a block diagram showing the overall configuration of a condition diagnostic system according to a first embodiment of the present invention. FIG. 2 is a schematic plan view showing an example of an xy actuator used in the condition diagnosis system according to the first embodiment. FIG. 3 is a cross-sectional view of the xy actuator of FIG. 2 taken along line II. FIG. 4 is a schematic diagram showing an example of a carriage movement locus of the xy actuator in the first embodiment. FIG. 5 is a flow chart showing an example of a control operation during imaging in the condition diagnosis device according to the first embodiment. FIG. 6 is a schematic diagram showing an example of movement of the imaging range relative to the inspection object. FIG. 7 is a flowchart showing a method for identifying an image position in the condition diagnosis device according to the first embodiment. FIG. 8 is a schematic diagram showing the correspondence between a movement trajectory obtained from a moving image and inspection point coordinates acquired from an xy actuator in the condition diagnosis device according to the first embodiment. FIG. 9 is a schematic diagram showing an example of a method for detecting the movement direction of a captured image. FIG. 10 is a schematic diagram showing an example of a method for detecting a moving direction change portion of a captured image. FIG. 11 is a schematic diagram showing an example of a synchronization method in which a moving direction change portion of a captured image in the condition diagnosis device according to the first embodiment is associated with a turning point of an xy actuator. FIG. 12 is a schematic diagram illustrating the correspondence between the imaging frames and the turning point coordinates and times of the xy actuator in the condition diagnosis device according to the first embodiment. FIG. 13 is a block diagram showing the functional configuration of a condition diagnostic device according to the second embodiment of the present invention. FIG. 14 is a schematic plan view showing an example of an xy actuator used in the condition diagnosis device according to the second embodiment. FIG. 15 is a cross-sectional view of the xy actuator of FIG. 14 taken along line II. FIG. 16 is a schematic cross-sectional view showing the configuration of a percussion unit in the condition diagnosis device according to the second embodiment. FIG. 17 is a flowchart showing an example of a tile region detection method in the condition diagnosis device according to the second embodiment. FIG. 18 is a diagram showing an example of a tile wall surface for explaining a tile region detection method in the condition diagnosis device according to the second embodiment. 19A to 19D are diagrams showing an example of image processing for explaining a tile region detection method in the condition diagnosis device according to the second embodiment. FIG. 20 is a flow chart showing an outline of a crack inspection procedure in the condition diagnosis device according to the second embodiment. FIG. 21 is a schematic diagram showing an example of a binarization process in a crack inspection procedure in the condition diagnosis device according to the second embodiment. FIG. 22 is a diagram showing a schematic diagram of the result of image processing in the crack inspection procedure in the condition diagnosis device according to the second embodiment. FIG. 23 is a flowchart showing an example of a crack reconnection process in the condition diagnosis device according to the second embodiment. FIG. 24 is a schematic diagram showing an example of a crack reconnection process in the condition diagnosis device according to the second embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings. However, the components described in the following embodiment are merely examples and are not intended to limit the technical scope of the present invention.

1. First embodiment 1.1) System configuration As illustrated in FIG. 1, a condition diagnosis system according to a first embodiment of the present invention is a system for evaluating the condition of an inspection object 10, and includes a condition diagnosis device 100, a camera 200 as an example of an imaging unit, and an xy actuator 300 as an example of a moving mechanism. As described below, the condition diagnosis device 100 executes a predetermined program on a computer processor, thereby controlling the movement of the xy actuator 300, inputs moving images from the camera 200 that moves accordingly, extracts inspection images at the required timing from the moving images, and performs image processing to evaluate the condition of the inspection object 10. The condition diagnosis system according to this embodiment is an asynchronous system, and the movement of the camera 200 body and the moving images captured by the camera 200 are not necessarily synchronized.

The camera 200 can be connected to the condition diagnosis device 100 by a predetermined wired or wireless interface, and can output a captured video file of a predetermined format to the condition diagnosis device 100. A wired interface is used here in consideration of the transmission of video images and power supply, and the camera 200 is detachably connected to the condition diagnosis device 100. The frame rate during imaging can be selected, and can be set to a predetermined rate, for example, between 30 and 240 fps (frames per second). As illustrated in FIG. 1, the camera 200 can be set to a standard time such as the Internet, but imaging is performed at a predetermined frame rate according to its own timing CLK1. The video captured by the imaging unit 201 is transmitted to the condition diagnosis device 100 through the interface 202. It is desirable for the camera 200 to be in focus at close range and have a wide angle of view.

The xy actuator 300 moves a moving part in the xy directions under the control of the condition diagnosis device 100. The camera 200 is fixed to the moving part of the xy actuator 300, and is therefore movable in the xy directions within a predetermined imaging range. The configuration and operation of the xy actuator 300 will be described below with reference to Figures 2 to 4.

<Moving mechanism>
2 and 3, the xy actuator 300 includes a pair of parallel y frames 302 fixed to a rectangular frame 301, and an x stage 303 perpendicular to the y frames 302. The x stage 303 is fixed on a y carriage 304 movable on each y frame 302, and the pair of y carriages 304 move in the y-axis direction by the rotation of a y-axis stepping motor M1. Therefore, the x stage 303 can be moved to a desired position in the y-axis direction by the y-axis stepping motor M1.

The x-stage 303 is provided with a movable portion 305, which is an x-carriage, to which the camera 200 and the light-emitting portion 306 are fixed. The light-emitting portion 306 is a light source that illuminates at least the inspection object 10 in the imaging range 200a of the camera 200, and may be a light-emitting diode (LED) or the like. The movable portion 305 moves on the x-stage 303 by the rotation of the x-axis stepping motor M2. Therefore, the camera 200 fixed to the movable portion 305 can be moved to a desired position in the x-axis direction by the x-axis stepping motor M2.

By controlling the rotation of the y-axis stepping motor M1 and the x-axis stepping motor M2 in this manner, the camera 200 fixed to the moving part 305 can capture the entire imageable range 307 while moving at a predetermined speed (or every predetermined step) along a predetermined route through the imageable range 307. Casters 308 are provided at the four corners of the frame 301 to enable the frame 301 to be moved over the inspection object 10, and the frame 301 can also be moved over the inspection object 10 by a driving means (not shown).

More specifically, as illustrated in FIG. 4, the camera 200 can be moved a predetermined distance Δx in the x-axis direction by rotating the x-axis stepping motor M2 by a predetermined angle, and the camera 200 can be moved a predetermined distance Δy in the y-axis direction by rotating the y-axis stepping motor M2 by a predetermined angle. For example, if the upper left corner of the imageable range 307 is the scanning start point, T ₀ is the scanning start time, and the coordinate point is expressed as p (0,0;T ₀ ), a point moved by Δx in the x-axis direction has an x coordinate changed by +1 or -1, and a point moved by Δy in the y-axis direction has an y coordinate changed by +1 or -1. In this way, an arbitrary position of the camera 200 is expressed as coordinate p (x,y;T). Here, T is the elapsed time from the start point p (0,0;T ₀ ). It is assumed that the accuracy of the moving distances Δx and Δy can be obtained by the encoders attached to the y-axis stepping motor M1 and the x-axis stepping motor M2 with sufficient positioning accuracy.

As shown in Fig. 4, the camera 200 sequentially scans in the x-axis direction from the starting coordinate p(0,0; _T0 ), and when it reaches the end point p(m,0; _Tm(1) ) of the first x-axis scanning line, i.e., the right end of the imageable range 307, it moves in the y-axis direction to the right end p(m,2;Tm ₍₂₎ ) of the second x-axis scanning line, and then scans in the opposite direction to the left end of the imageable range 307, repeating this scanning operation. Therefore, this coordinate p(m,0; _Tm(1) ) or p(m,2; _Tm(2) ) can be regarded as a change point in the coordinate direction, i.e., a scan turn-around point. Hereinafter, the upper coordinate direction change point p(m,0; _Tm(1) ) is regarded as the scan turn-around point at the right end of each x-axis scanning line.

At the scan turnaround point, the camera 200 changes its direction of movement by an angle of approximately 90°. This 90° change in the direction of movement also appears as a 90° change in the moving image of the camera 200. Therefore, if it is detected that the direction of movement of the moving image has changed by nearly 90°, the frame information at that point in time can be regarded as the actual scan turnaround point of the camera 200, and the frame image of the camera 200 can be synchronized with the operation of the xy actuator 300 based on this as a reference.

However, as long as a valid change in the direction of movement can be detected, the angle of change does not need to be 90° as in FIG. 4, and may be 180° when returning to the starting point. According to this embodiment, as an example, synchronization is established between the camera 200 and the xy actuator 300 by detecting a 90° change in the direction of movement in a moving image.

The means for moving the y-carriage 304 and the moving part 305 by the y-axis stepping motor M1 and the x-axis stepping motor M2 can be of any type. It can be a feed screw type or a belt feed type, but in this embodiment, the belt feed type is used. A commercially available product can be used as this xy actuator 300. The size of the imaging range 307 of the camera 200 is, for example, about 60 x 60 cm. The moving mechanism is not limited to the xy actuator 300, and a robot arm or the like can be used as long as sufficient positioning accuracy can be obtained.

<Condition diagnosis device>
1, the condition diagnosis device 100 can realize various functions described below by executing a program stored in a memory (not shown) on a processor. The control operation of the condition diagnosis device 100 is executed according to timing CLK2, and starts operation in accordance with a standard time such as the Internet, but does not need to be synchronized with CLK1 of the camera 200.

The condition diagnosis device 100 can be connected to the camera 200 via a wired or wireless interface 101. The video storage unit 102 stores video files input from the camera 200 through the interface 101. The video files can be in well-known formats such as AVI or MOV, and here it is assumed that a timestamp is added to the video files in one-second increments.

The xy movement control unit 103 controls the y-axis stepping motor M1 and the x-axis stepping motor M2 of the xy actuator 300 according to timing CLK2, and moves the moving unit 305 to which the camera 200 is fixed in the xy directions as described above. The coordinate acquisition unit 104 outputs the position and time of the camera 200 on the moving unit 305 as coordinates p(x, y; T) to the image position synchronization unit 107 as shown in FIG. 4. In addition, the video file captured by the camera 200 is stored in the video storage unit 102 together with a timestamp.

The movement detection unit 105 sequentially reads out multiple frame images from the video file and detects the movement direction of the images, and therefore the movement direction of the camera 200, using a method such as optical flow. The movement direction change detection unit 106 determines whether the movement direction has changed by more than a predetermined angle using a method described below, and outputs the image frame number F and time information t that have changed by more than the predetermined angle to the image position synchronization unit 107 as a movement direction change point.

The image position synchronization unit 107 inputs the position information of the camera 200 as coordinates p(x, y; T) from the coordinate acquisition unit 104, and, as described later, associates the time t of the moving direction change point of the video with the time Tm of the coordinate change point p(m, _ym ; _Tm ) using the time stamp as a clue. In this way, the position coordinates of the camera 200 and the video can be synchronized, and the image frame of the video file corresponding to the specified coordinates can be identified.

The evaluation unit 108 inputs an image of a predetermined position from the image position synchronization unit 107, recognizes the area to be inspected by image processing, and evaluates the condition of the area. For example, if the object to be inspected 10 is a tiled wall surface, it is possible to detect one tile area from the input image and diagnose the condition of the tile area, such as cracks, by image recognition technology. The detection of tile areas will be explained in detail in the section on the second embodiment.

The condition diagnosis device 100 can realize the above-mentioned functions by executing a predetermined program on a computer. The overall operation of the condition diagnosis system according to the first embodiment will be described in detail below with reference to Figures 5 to 12.

1.2) Operation <Video image acquisition>
As illustrated in FIG. 5, the processor 109, which is the data processing unit of the condition diagnosis device 100, resets the timing CLK2 to the standard time by initial setting, starts the camera 200, and resets the moving unit 305 of the xy actuator 300 to the origin position p(0,0) (operation 501).

As described above, the camera 200 captures the inspection object 10 according to its own timing CLK1, and outputs video data together with time stamps at a predetermined interval (here, 1 second intervals) (operation 502). Here, it is assumed that the images are captured at a frame rate of 240 fps. The condition diagnosis device 100 receives the video data from the camera 200, and stores the video file consisting of the frame-decomposed frame images F(i) in the video storage unit 102 (operation 503).

Along with capturing an image by the camera 200, the processor 109 of the condition diagnosis device 100 controls the xy movement control unit 103 to move the moving unit 305 of the xy actuator 300 as shown in FIG. 4, thereby moving the imaging range 200a of the camera 200 on the inspection target 10 at a predetermined speed or in predetermined steps as shown in FIG. 6 (operation 504). At that time, the processor 109 inputs and records the coordinates p(x, y; T) of the moving unit 305 from the coordinate acquisition unit 104 (operation 505).

The processor 109 of the condition diagnosis device 100 monitors the coordinates of the moving unit 305 and determines whether the entire imageable range 307 has been scanned (operation 506). If scanning of the entire area has not been completed (NO in operation 506), the above operations 502 to 506 are repeated until scanning of the entire area is completed. If scanning of the entire area is completed (YES in operation 506), imaging by the camera 200 is stopped and processing is terminated (operation 507).

<Detection of change in movement direction>
7, the processor 109 of the condition diagnosis device 100 sequentially reads out a plurality of frame images from the video file in the video storage unit 102, and detects the movement direction of a specific image portion, and therefore the movement direction of the camera 200, by optical flow (operation 601). For example, in the case of a wall surface on which tiles are arranged as shown in FIG. 6, the movement direction of the camera 200 can be detected by following the movement of a specific tile image for each frame image.

Then, the processor 109 determines whether the movement direction has changed by a predetermined angle or more, and detects the image frame number F and time information t at which the movement direction has changed by the predetermined angle or more as a movement direction change point (operation 602). An example of a method for detecting a movement direction change point will be described below with reference to Figs. 8 to 10.

First, as illustrated in FIG. 8, an image movement trajectory 401a is acquired from a plurality of frame images by optical flow, and a coordinate trajectory 401b indicating the movement of the camera 200 is acquired from the coordinate p (x, y) input from the coordinate acquisition unit 104. The coordinate trajectory 401b corresponds to the scan trajectory in the coordinate system shown in FIG. 4. The movement trajectory 401a and the coordinate trajectory 401b are based on the same scan movement of the camera 200, but the moving image of the camera 200 and the coordinates indicating the camera position are not associated. For this reason, it is not possible to know which position (coordinate) of the inspection target 10 the frame image of the moving image is. Therefore, in this embodiment, for example, the movement direction change points 402a, 403a on the movement trajectory 401a and the coordinate direction change points (scan turnaround points) 402b, 403b on the coordinate trajectory 401b are detected, and the moving image and the coordinates are synchronized based on them. The correspondence between the movement direction change point and the coordinate direction change point (scan turnaround point) may be performed for each x-axis line, or may be performed for every predetermined number of x-axis lines, or may be performed when there is a deviation beyond a predetermined value, as in the case of movement direction change point 403a in FIG. 8. In other words, when the same movement pattern is repeated, once synchronization is achieved, synchronization is not performed even if there is a movement direction change point (corner) thereafter, and synchronization can be achieved at the movement direction change point (corner) when there is a large deviation from the repetition.

As shown in Figure 9, with respect to the position of the point of interest in a certain image frame F(i), the position of the point of interest in the image frame F(i-N) N frames before is vector m1, the position of the point of interest in the image frame F(i+N) N frames after is vector m2, and the angle between vectors m1 and m2 is θ (here, i>N, where i and N are integers equal to or greater than 0). In this case, the change in the direction of movement can be calculated by the inner product S of vectors m1 and m2 = |m1| · |m2| cos θ. In other words, if |m1| = |m2| = 1, when the point of interest is moving in the x-axis direction, θ = 180°, so the inner product S = -1, when it changes direction in the y-axis direction, θ = 90°, so the inner product S = 0, and when it turns around, θ = 0°, so the inner product S = +1. Therefore, if the S-value threshold value TH is set to detect only clear angle changes such as θ = 90° or 0° and ignore other angle changes, it is possible to detect movement direction change points 402a, etc. on movement trajectory 401a and coordinate direction change points 402b, etc. on coordinate trajectory 401b.

As shown in FIG. 10, the movement direction change points 402a(1) and 402a(2) on the movement trajectory 401a are both changed at an angle of about 90°. Therefore, in the image frames up to the change point, the focus point moves in the x-axis direction, so S is approximately -1, but at the first change point 402a(1), S rises to approximately 0, then returns to approximately -1, and at the second change point 402a(2), it rises again to approximately 0, and then moves along the next x-axis line, so S remains approximately -1. Therefore, by appropriately setting the threshold value TH, it is possible to identify the frame number and time of the image frame where the S value peaks (i.e., where the movement direction change point is indicated). Here, the movement direction change point 402a is detected at the first peak point 402a(1) of the S value, and the coordinate direction change point (scan turnaround point) 402b on the coordinate trajectory 401b is detected in a similar procedure.

<Image position synchronization>
7, when the processor 109 detects the image frame number F and the time t as the movement direction change point 402a by the above-mentioned operation 602, it is possible to synchronize the progress of the frame number of the moving image (i.e., the change in the image over time) with the change in the position coordinate over time by associating the movement direction change point 402a with the coordinate direction change point 402b (operation 603). Image position synchronization will now be described with reference to FIG.

As shown in FIG. 11, the S-value peak of the movement direction change point 402a detected from multiple frame images can be time-aligned with the S-value peak of the coordinate direction change point 402b to achieve correspondence. In this case, the timestamp of the video image can be used as follows.

As an example, the imaging range 307 of the xy actuator 300 is 60 x 60 cm, and the xy actuator 300 moves the camera 200 at 20 cm per second. The camera 200 captures images at a frame rate of 240 fps and adds time stamps at 1-second intervals. In this case, it takes 3 seconds for the camera 200 to move one x-axis line from one end of the imaging range 307 to the other, and for each time stamp, the camera 200 moves 20 cm as a scan distance, outputting 240 frames of images during that time. Therefore, the deviation between the time stamp time tm of the frame image with the movement direction change point 402a and the time Tm of the position coordinate of the coordinate direction change point 402b is a maximum of about ±1 second, and it is sufficient to match the movement direction change point 402a and the coordinate direction change point 402b that are closest within this range.

<Image acquisition>
7, when the processor 109 establishes image position synchronization by the above-mentioned operation 603, it outputs one or more frame images corresponding to each position coordinate p(x, y) by the xy actuator 300 as image data for inspection (operation 604). Image acquisition will be described below with reference to FIG.

As shown in FIG. 12, by associating the image frame F(n) indicating the movement direction change point with the time Tm of the position coordinate 402a, the image frames F(0) to F(n) can be associated with the times T0 to Tm of the position coordinate, and this makes it possible to identify the image frames F(0), F(i), F(j)... corresponding to the predetermined position coordinates p(0,0), p(1,0), p(2,0)... respectively. In this way, the position coordinates of the camera 200 and the captured image can be synchronized, and the captured image at the predetermined position coordinates of the inspection object 10 can be output. The evaluation unit 108 can diagnose the condition of the inspection object 10 using the captured image obtained in this way.

1.3) Effects As described above, according to the first embodiment of the present invention, even in a system in which the moving image obtained from the camera 200 and the movement control of the xy actuator 300 that moves the camera 200 body in a scan are not synchronized, it is possible to know which position on the inspection target 10 the frame image of the moving image is. As described above, the moving direction change point is detected from the moving image, the coordinate direction change point (scan turnaround point) is detected from the position coordinates of the camera 200, and the timing of the moving direction change point and the coordinate direction change point are matched to associate the moving image of the camera 200 with the coordinates indicating the camera position. This makes it possible to identify the image capture position based on the image captured of the inspection target.

2. Second embodiment 2.1) System configuration As illustrated in FIG. 13, the condition diagnosis system according to the second embodiment of the present invention includes a condition diagnosis device 700, a camera 200 as an example of an imaging unit, an xy actuator 300 as an example of a moving mechanism, and a diagnosis unit 800 that moves together with the camera 200. The condition diagnosis device 700 controls the xy actuator 300 by executing a predetermined program on a computer processor as in the first embodiment, performs condition evaluation of the inspection target 10 using a moving image, and further performs condition evaluation by striking the inspection target 10 with the diagnosis unit 800. Hereinafter, configurations and functions of the second embodiment that are different from those of the first embodiment will be described, and blocks having the same functions as those of the first embodiment will be given the same reference numbers and detailed description will be omitted.

The processor 701, which is the data processing unit of the condition diagnosis device 700, executes a program to realize functions consisting of a movement detection unit 105, a movement direction change detection unit 106, an image position synchronization unit 107, and an evaluation unit 108, similar to those in the first embodiment. The evaluation unit 108 in this embodiment consists of a tile area detection unit 702 and a crack detection unit 703.

The tile area detection unit 702 inputs an image of a predetermined position from the image position synchronization unit 107 and detects the tile area to be inspected by image processing, and the crack detection unit 703 detects cracks, etc. from the image of the predetermined position input from the image position synchronization unit 107 by image processing. The detection results of cracks, etc. are stored in the diagnosis information storage unit 704 together with the position coordinates of the tile area.

The diagnostic unit 800 is a sensor means that outputs an inspection signal by striking the surface of the inspection object 10, and may employ, for example, the detection unit disclosed in JP 2016-205900 A. The diagnostic unit 800 is controlled by a percussion control unit 705, which is controlled in synchronization with the scan control by the xy movement control unit 103 that controls the xy actuator 300. The diagnostic unit 800 is fixed to the movement unit 305 of the xy actuator 300, as described below.

The inspection signal generated by the impact obtained from the diagnostic unit 800 is evaluated by the percussion evaluation unit 706, and the evaluation result is stored in the diagnostic information storage unit 704 together with the position coordinates. Therefore, the surface condition of each position of the object to be inspected 10 is stored in the diagnostic information storage unit 704 as an image diagnosis result based on the image captured by the camera 200 and as an impact diagnosis result by the diagnostic unit 800. The configuration and operation of the xy actuator 300 are as described in the first embodiment. The configuration around the diagnostic unit 800 attached to the xy actuator 300 will be described below with reference to Figures 14 and 15.

As illustrated in Figures 14 and 15, the xy actuator 300 comprises a pair of parallel y frames 302 and an x stage 303 perpendicular to the y frames 302. As already explained, the x stage 303 can be moved in the y-axis direction by the y-axis stepping motor M1, and the moving part 305 can be moved in the x-axis direction on the x stage 303 by the x-axis stepping motor M2.

In this embodiment, the percussion unit 800 is fixed to the moving part 305 directly below the x-stage 303 so as to face the surface of the inspection target 10. The camera 200 and the light emitting part 306 are also fixed to the moving part 305 in a position adjacent to the percussion unit 800, and are installed so that the diagnostic unit 800 does not enter the imaging range 200a of the camera 200. Since the positional relationship between the camera 200 and the diagnostic unit 800 with respect to the moving part 305 is fixed, the offset between the imaging range 200a and the striking position by the diagnostic unit 800 can be corrected.

By controlling the rotation of the y-axis stepping motor M1 and the x-axis stepping motor M2 in this way, the camera 200 fixed to the moving part 305 captures images while scanning the imageable range 307, and the diagnostic unit 800 fixed to the moving part 305 can perform an impact inspection of the surface condition of the inspection object 10 while also scanning.

The diagnostic unit 800 moves according to the scan coordinates p(x, y; T) shown in FIG. 4, and performs impact inspections for each movement amount Δx and Δy, making it possible to associate the inspection results with the impact position.

As illustrated in FIG. 16, the percussion unit 800 in this embodiment has an impact section 802 fixed inside a housing 801, and the impact section 802 is made of a solenoid, which, for example, pushes the plunger 803 downward with a predetermined force when the solenoid is energized, and retracts the plunger 803 upward by means of a spring or other means when the current is stopped. An impact hammer 804 is provided at the lower end of the plunger 803, and movably passes through an opening 806 provided in the center of a base 805 on the bottom surface of the housing 801. When the solenoid is energized, the impact hammer 804 can strike the inspection surface of the inspection object 10 through the opening 806.

Microphones 809 are attached to the four sides of the bottom end of the housing 801 via anti-vibration rubber 808, and a microphone cover 810 is provided on top of these. The microphones 809 detect the impact sound from the inspection surface when struck. Note that, although four microphones 458 are provided in this example, the number is not limited to this.

When the percussion unit 800 is moved to the position to be inspected by the xy actuator 300, the striking hammer 804 strikes the inspection surface. The striking sound produced by striking the inspection surface at this time is detected by the microphone 458, and the condition of the inspection surface, such as peeling, can be evaluated from this striking sound detection signal.

Here, the camera 200 is placed about 15 cm away from the surface of the object 10 to be inspected, and is capable of capturing images of tile frames of sufficient size and number to allow detection of tile areas.

2.2) Tile Region Detection The tile region detection unit 702 receives an image at a predetermined position from the image position synchronization unit 107, and detects a tile region to be inspected by image processing. An example of a tile region detection procedure will be described below with reference to FIGS. 17 to 19.

In FIG. 17, the tile area detection unit 702 extracts horizontal and vertical edges from the input image, and detects the rough position of the tile frame from this edge information (operation 901). For example, as illustrated in FIG. 18, when the camera 200 captures an image of an inspection target 10 on which tiles are arranged in an imaging range 200a, a horizontal edge extraction image illustrated in FIG. 19(A) and a vertical edge extraction image illustrated in FIG. 19(B) are obtained. From these edge extraction images, a rough area where rectangular tiles exist, for example the area indicated by reference number 10a, can be obtained. As an alternative method, the rough area can be specified by detecting parts corresponding to the corners of the tiles from the input image.

Next, the tile region detection unit 702 applies a rectangle detection algorithm to the region 10a to accurately detect a rectangular region (operation 902). As the rectangle detection algorithm, an algorithm such as Harris' corner detection can be used. The rectangle detection algorithm can identify a tile region 10b as shown in FIG. 19(D).

Returning to FIG. 17, the tile region detection unit 702 determines whether the tile region 10b is the same as the region detected from the immediately preceding captured image (operation 903). Whether or not they are the same tile region can be determined, for example, by the degree of overlap (area) between the tiles. If there are identical tile regions, the best tile image is determined based on either image quality, brightness unevenness, or the size of the specified tile, or a combination of these (operation 904).

2.3) Percussion Information and Image Diagnosis Information Once each tile area has been determined from the imaging range 200a in this manner, the crack detection unit 703 performs crack detection as exemplified in the next section, and stores the detection results together with the position coordinates of the tiles in the diagnosis information storage unit 704. In addition, the percussion evaluation results obtained by the diagnosis unit 800 as described above are also stored in the diagnosis information storage unit 704 together with the position coordinates.

2.4) Crack Detection As described above, the crack detection unit 703 performs crack detection by image processing as exemplified in Figures 20 to 24.

Figure 20 shows an overview of the crack detection procedure. First, the crack detection unit 703 performs preprocessing to binarize the raster image of the imaging range 200a, which includes multiple tile areas (operation 910), and then vectorizes the binarized image (operation 911). The vectorized image is subjected to noise removal and, if multiple cracks are present, a process to reconnect them (operation 912). The crack detection results obtained in this manner are stored in the diagnostic information storage unit 704 together with the coordinates of the tile areas (operation 913). Each processing operation of the crack detection unit 703 is described in detail below.

<Binarization process>
21, a raster image 920 of the imaging range 200a is input to the crack detection unit 703. Next, the raster image 920 is grayscaled, and a background image 922 is generated by removing local black areas from the resulting grayscale image 921. Next, the background image 922 is subtracted from the grayscale image 921 to generate a difference image 923, which is then binarized to generate a binarized image 924. In the following, to simplify the explanation, the vectorization will be explained by taking up a raster image 924A near the crack.

<Vectorization process>
As shown in FIG. 22, the crack detection unit 703 thins the raster image 924A to generate a thinned image 924B, and then performs vectorization on the thinned image 924B. This results in a vector image 924C in which an arbitrary curve is expressed as a large number of vectorized line segments. Other parts of the raster image 924 are similarly vectorized. Here, if continuous lines C(1), C(2), C(3), etc. exist discretely in the vector image 924C, the crack detection unit 703 removes lines shorter than a predetermined value (here, C(4), C(5)) as noise by the crack reconnection process described below, and then detects a continuous curve from the discrete continuous lines as a crack. Hereinafter, an example of the crack reconnection process will be described with reference to FIG. 23 and FIG. 24.

<Reconnection of cracks>
In Fig. 23, the crack detection unit 703 preliminarily specifies a threshold value for the length of a vector, and after eliminating vectors shorter than the threshold value as noise, selects a crack line having an end point from the vector image of the imaging range 200a illustrated in the vector image 924C (operation 931). Hereinafter, the end point of the selected crack line is referred to as the target end point. Hereinafter, the curve _C0 in Fig. 24 is selected, and its end point _P0 becomes the target end point.

Next, the crack detection unit 703 searches for endpoints of other crack lines within a range from the target endpoint of the crack line that satisfies the following conditions a) and b) (operation 932).
Condition a) being on the extension side of the line segment of the target endpoint of the crack line and within a specified angle centered on the target endpoint, in other words, being located opposite the target endpoint; and Condition b) being within a specified distance from the target endpoint of the crack line.
24, the search range R that satisfies the conditions a) and b) is a range (condition a) that is less _than an angle α centered on a target end point _P0 of the curve _C0 with the extension direction of a line segment S _C0 of length L at the end of the curve _C0 as the center line, and is a range (condition b) within a distance (n×L) of n times (a real number of 1 or more) the length L of the line segment S _C0 from the target end point P0. In other words, the search range R has a sector shape with a central angle α centered on the target end point _P0 of the curve _C0 and a radius proportional to the length of the line segment S _C0 at the end of the curve _C0 .

The crack detection unit 703 determines whether or not there are other crack lines that match the results of the above search (operation 933), connects the target end point with the end point of the other crack line that matches, and registers them as one crack line (operation 934). Note that if there are multiple other end points within the search range, it is sufficient to select the other end point that is closest to the target end point.

24, the end point _P1 of curve _C1 and the end point _P2 of curve _C2 are present within the search range R, and therefore these. A portion of curve _C3 is within the search range R, but its end point _P3 is not present within the search range R and is therefore excluded from the search target. In this example, the end point _P2 of curve _C2 is closer to the target end point _P0 than the end point _P1 of curve _C1 , so the crack detection unit 703 connects the target end point _{P0 to} the end point _P2 of curve C2 and registers them as one crack line.

Returning to FIG. 23, if no other crack lines exist (NO in operation 933), or following operation 934, it is determined whether there are any unsearched end points (operation 935). If there are any unsearched end points (YES in operation 935), the crack detection unit 703 repeats the above operations 931 to 935 until the search is complete, and when the search for all end points has been completed (NO in operation 935), the crack reconnection process is terminated.

The crack detection unit 703 stores the registered crack detection results together with the coordinates of the tile area in the diagnostic information storage unit 704. In this way, it becomes possible to perform a comprehensive evaluation of cracks in the imaging range 200a. Here, as exemplified by the vector image 925 in FIG. 22, it is possible to detect one crack line C that spans multiple tiles, and it is possible to extract cracks that would have been overlooked if the cracks were evaluated for only one tile.

2.4) Effects As described above, according to the second embodiment of the present invention, even in an asynchronous system, it is possible to associate the video image of camera 200 with the coordinates indicating the camera position, as in the first embodiment, and further to detect and evaluate tile areas from the tile images corresponding to the position coordinates.

In addition, the image diagnosis results based on the image captured by the camera 200 and the impact diagnosis results by the diagnostic unit 800 are stored in the diagnostic information storage unit 704 together with the position coordinates, so that the surface condition of the inspection object 10 consisting of a tiled wall surface can be evaluated by images and tapping. In particular, by extracting cracks across multiple tiles, it is possible to extract, for example, long cracks caused by cracks on the structural body of the inspection object, which makes it easier to select a repair method. For example, this has the advantage of making it easier to select a method, such as whether to repair only the tiles or to repair to the base as well.

The present invention can be applied to a condition diagnosis system for the walls or floors of buildings.

10: Inspection object 100: Condition diagnosis device 101: Interface 102: Video image storage unit 103: xy movement control unit 104: Coordinate acquisition unit 105: Movement detection unit 106: Movement direction change detection unit 107: Image position synchronization unit 108: Evaluation unit 109: Processor (data processing unit)
200 Camera 200a Imaging range 201 Imaging unit 202 Interface 300 xy actuator 305 Moving unit 307 Imaging possible range 401a Movement trajectory obtained from moving image 401b Scan coordinate trajectory 402a, 403a Movement direction change point 402b, 403b Scan turnaround point (coordinate direction change point)
701 Processor (data processing unit)
702 Tile area detection unit 703 Crack detection unit 704 Diagnostic information storage unit 705 Percussion control unit 706 Percussion evaluation unit 800 Diagnostic unit

Claims (10)

a moving mechanism that moves the moving part along the surface of the inspection object within an imaging range;
a camera attached to the moving part for capturing a moving image of a surface of the inspection object;
a coordinate acquisition unit that acquires position coordinates of the moving unit within the image capture range;
a data processing unit for detecting a test image from the moving image;
Equipped with
The data processing unit:
Detecting a change point in the moving direction of the camera based on a plurality of images of the video;
a point at which the direction of change of the position coordinates of the moving part obtained from the coordinate acquisition unit turns around and a point at which the direction of movement of the camera changes, thereby detecting an inspection image corresponding to a desired position coordinate from the moving image;
A condition diagnosis system comprising: The condition diagnosis system according to claim 1, characterized in that the movement direction change point is a time point in the image of interest at which the movement direction of a specific image part contained in the image of interest and two images that are temporally before and after the image of interest changes by more than a predetermined angle. The condition diagnosis system according to claim 1 or 2, characterized in that an edge portion is extracted from the inspection image, and a tile area of the inspection object is extracted from the edge portion using a rectangle detection algorithm. The condition diagnosis system according to any one of claims 1 to 3, characterized in that the data processing unit evaluates the surface condition of the object to be inspected based on a tile area extracted from the inspection image. The condition diagnosis system according to claim 4, characterized in that the data processing unit detects crack lines in at least one tile region from the inspection image by image processing. The condition diagnosis system of claim 5, characterized in that the data processing unit connects the end points of the first crack line and the end points of the second crack line if they are within a predetermined distance and a predetermined angle, and detects the first and second crack lines as a single crack line. The condition diagnosis system according to any one of claims 1 to 6, further comprising a percussion unit attached to the moving part, the percussion unit striking the surface of the object to be inspected and outputting the striking sound as an inspection signal to the data processing part. a movement control unit that controls a movement mechanism that moves the movement unit along the surface of the inspection object within an image capturing range;
an interface for connecting, by wire or wirelessly, to a camera attached to the moving part and configured to capture a moving image of the surface of the inspection object;
a coordinate acquisition unit that acquires position coordinates of the moving unit within the image capture range;
a data processing unit for detecting a test image from the moving image;
Equipped with
The data processing unit:
Detecting a change point in the moving direction of the camera based on a plurality of images of the video;
a point at which the direction of change of the position coordinates of the moving part obtained from the coordinate acquisition unit turns around and a point at which the direction of movement of the camera changes, thereby detecting an inspection image corresponding to a desired position coordinate from the moving image;
A condition diagnosis device characterized by: 1. A method for identifying an image position in a condition diagnosis device having a movement control unit that controls a movement mechanism that moves a movement unit along a surface of an inspection object within an image capturing range, and an interface that is attached to the movement unit and connects, by wire or wirelessly, to a camera that captures a moving image of the surface of the inspection object, comprising:
a coordinate acquisition unit acquires position coordinates of the moving unit within the image capture range;
The data processing unit
Detecting a change point in the moving direction of the camera based on a plurality of images of the video;
identifying an image corresponding to a desired position coordinate from the video by associating a point at which the direction of change of the position coordinates of the moving part, obtained from the coordinate acquisition unit, turns around with a point at which the direction of movement of the camera changes;
13. A method for identifying an image location, comprising: A program for making a computer function as a condition diagnosis device having a movement control unit that controls a movement mechanism that moves a movement unit along a surface of an object to be inspected within an imaging range, and an interface that is attached to the movement unit and connects, by wire or wirelessly, to a camera that captures a video image of the surface of the object to be inspected,
a function of acquiring position coordinates of the moving unit within the image capture range;
a function of detecting a change point in the moving direction of the camera based on a plurality of images of the moving image;
a function of identifying an image corresponding to a desired position coordinate from the video by associating a point where a direction of change in the position coordinates of the moving part turns around with a point where a direction of movement of the camera changes;
A program for causing the computer to execute the above.

Images