WO2014073571A1 - Image processing device for self-propelled industrial machine and image processing method for self-propelled industrial machine - Google Patents

Abstract

The purpose of the present invention is to automatically set a caution-requiring range in which an obstacle is detected within a camera's field-of-view range, and even if the camera's field-of-view range is shifted afterwards, prevent the caution-requiring range from being shifted. The present invention is provided with: a camera (6) which is provided in a dump truck (1) and captures an image of a region in which a front structure (10) of the dump truck (1) is included in the field of view thereof; a structure detection unit which, on the basis of image data (P) captured by the camera (6), detects the front structure (10) in the image data (P); and a caution-requiring range setting unit (26) which, with a range in which the detection of an obstacle (M) is required within the image data (P) as a caution-requiring range (31), sets the caution-requiring range (31) within the image data (P) on the basis of the front structure (19) detected by the structure detection unit.

Description

Image processing apparatus for self-propelled industrial machine and image processing method for self-propelled industrial machine

The present invention relates to an image processing apparatus of a self-propelled industrial machine that performs predetermined image processing on image data captured by a photographing unit provided in a self-propelled industrial machine and an image processing method thereof.

The self-propelled industrial machines include transport machines such as dump trucks and working machines such as hydraulic shovels. Among these, the dump truck is provided with a vessel which can be undulated on the frame of the vehicle body. An excavated object excavated by a hydraulic shovel or the like is loaded on a vessel of the dump truck. After a predetermined amount of excavated material is loaded in the vessel, the dump truck is traveled to a predetermined accumulation site to excavate the loaded excavation material at the accumulation site.

The dump truck is a large transport vehicle, and there are many blind spots that can not be seen by the operator who gets into the cab. For this purpose, a camera is mounted on the dump truck, and an image captured by the camera is displayed on the monitor of the driver's cab. The camera takes an image of the area which is the blind spot of the operator. Then, by displaying the image captured by the camera on the monitor, the operator can supplementarily obtain a blind spot. Thereby, the operator can recognize the situation around the dump truck in detail.

A self-propelled industrial machine is a hydraulic shovel having a lower traveling body. A camera is also mounted on this hydraulic shovel to check surrounding conditions in detail. Patent Document 1 discloses a technique for performing predetermined image processing on an image acquired by a camera of a hydraulic shovel to recognize an obstacle. In the technology of Patent Document 1, it is detected by image processing whether or not an intruding moving body is in the working range of the hydraulic shovel. By performing this image processing, the distance of the intruding moving body and the degree of certainty as a human body are evaluated.

Japanese Patent Application Laid-Open No. 10-72851

By the way, a predetermined visual field range exists in the camera, and an image of this visual field range is displayed on the screen of the monitor as image data. Then, as in the technique of Patent Document 1 described above, by performing predetermined image processing on image data, an obstacle such as a worker, a service car, or another work machine is detected. As a result, the operator who gets in the driver's cab can know that when an obstacle is detected.

Here, a range close to a self-propelled industrial machine such as a dump truck needs to be alerted, and this range is considered as a critical alert range. This alert range can be set as a range in which the dump truck and the obstacle may come in contact with each other. Therefore, all of the alert range needs to be included in the camera view range. That is, the alert range is a partial range of the field of view of the camera. For this reason, a range other than the alert range is generated in the visual field range of the camera. The range outside the vigilance range is a vigilance area, and even if an obstacle exists in this range, no problem occurs. That is, it is not necessary to know the operator.

If image processing is performed up to this cautionary unnecessary range and an obstacle is detected, the operator is notified of information which does not need to be recognized by the operator, which causes the operator to be confused. This reduces the work efficiency. Further, the caution-unwanted range is a far range in the visual field range of the camera, and the pixels when this range is imaged become rough. As a result, even non-obstacles may be falsely detected as obstacles, and normal obstacles can not be detected. Furthermore, when the image processing is performed to the unwarranted range, the amount of information to be processed increases, and the time required for processing also increases.

Therefore, it is important to set an alert range within the field of view of the camera. However, it is very difficult to manually set the alert range based on the image data acquired from the camera, and the effort for setting becomes extremely complicated. In addition, since a self-propelled industrial machine such as a dump truck is operated at a severe site, there is a possibility that the attachment position or angle of the camera may be shifted due to the vibration of the vehicle body. As a result, even if the warning range is initially set, the warning range deviates after the fact, and the obstacle can not be detected normally.

Therefore, according to the present invention, the alert range for detecting an obstacle in the camera's field of view is automatically set, and even if the camera's field of view deviates afterward, the alert range is not deviated. With the goal.

In order to solve the above problems, the image processing apparatus for a self-propelled industrial machine according to the present invention is provided in a self-propelled industrial machine, and a part of the structure constituting the self-propelled industrial machine is included in the field of view. An imaging unit for imaging a peripheral region, a structure detection unit for detecting the structure in the image data based on image data captured by the imaging unit, and a position in the peripheral region of the image data To be detected, and the predetermined range including the obstacle is set as the required alert range, and the required alert range is set in the image data based on the structure detected by the structure detector. And a warning range setting unit.

The image processing apparatus may further include an obstacle detection unit configured to detect the obstacle only in the alert range in the image data.

In the image data, a range other than the alert range is set as the alert unnecessary range, and the difference between the pixel values of each pixel in the alert unnecessary range which is temporally adjacent to each other is calculated. The image processing apparatus may further include a shift determination unit that determines whether or not the view range of the imaging unit is shifted more than an allowable range based on the number of pixels which is equal to or greater than a threshold.

The self-propelled industrial machine may include a traveling speed detection unit that detects a traveling speed, and the alert range may be changed based on the traveling speed detected by the traveling speed detection unit.

The image picked up by the photographing unit is displayed on the image display means. In addition to the display mode described above, the image of the imaging unit is subjected to viewpoint conversion by signal processing to be a bird's eye view with the upper viewpoint It is good also as composition.

In the image processing method of a self-propelled industrial machine according to the present invention, a photographing unit is provided in the self-propelled industrial machine, and a surrounding area including a part of the structure of the self-propelled industrial machine is photographed, Based on the image data captured by the imaging unit, the structure in the image data is detected, and an obstacle located in the peripheral area of the image data is detected, and a predetermined range including the obstacle is detected. Setting the required alert range in the image data based on the structure detected by the structure detection unit, and setting the failure with respect to only the required alert range in the image data Perform processing to detect objects.

In the present invention, the imaging unit performs imaging so that the structure of the self-propelled industrial machine is included in the field of view, detects the structure edge, and sets the alert range based on the structure edge. As a result, it is possible to automatically set an alert range for detecting an obstacle within the visual field range of the camera. Further, even if the view range of the camera is deviated afterward, the caution range does not shift.

It is a top view of a dump truck. It is a side view of a dump truck. It is a block diagram showing composition of an image processing device of a 1st embodiment. It is a flowchart which shows operation | movement of the image processing apparatus of 1st Embodiment. It is a figure which shows an example of image data. It is a figure which shows an example of the image data which performed distortion correction. It is a figure explaining an example of the smoothing process with respect to image data. It is a figure which shows an example of the image data which performed the smoothing process. It is a figure explaining an example of edge extraction processing. It is a figure which shows an example of the image data by which the edge extraction process was performed, (a) is the image data from which the edge pixel was extracted, (b) is the image data after the image conversion process of the image data shown by (a). . FIG. 6A is a diagram showing an example of a vigilance range and a vigilance unnecessary range, wherein (a) is image data in which a vigilance range is set based on edge line segment data, and (b) is image data shown in (a) Image data to which distortion restoration processing has been added. It is a figure which shows an example of the display image of 1st Embodiment. It is a block diagram showing composition of an image processing device of a 2nd embodiment. It is a flowchart which shows operation | movement of the image processing apparatus of 2nd Embodiment. It is a figure which shows an example of a caution range image. It is a figure which shows an example of the detection method of a mobile using the block matching method. It is a figure which shows an example when scanning a comparison range in a search range. It is the figure which demonstrated the block matching method three-dimensionally. It is a figure which shows an example of the display image of 2nd Embodiment. It is a figure which shows an example for obtaining a difference image from a mask image. It is a figure which shows an example of the display image on which the shift | offset | difference information was superimposed. It is a figure showing an example which specifies the front structure in case a gap has arisen in a visual field range, (a) is image data from which an edge pixel was extracted, (b) is an image conversion shown in (a) It is image data after processing. The figure which shows an example of a vigilance range and a vigilance unnecessary range in case a visual field range has a gap, (a) is image data in which a vigilance range is set based on edge line segment data, (b) is It is image data obtained by adding distortion restoration processing to the image data shown in (a). It is a figure which shows an example of the caution range in case the shift | offset | difference has arisen in visual field range. It is a block diagram which shows the structure of the image processing apparatus of 3rd Embodiment. It is a figure which shows an example of the image data of 4th Embodiment. It is a figure which shows an example of the image which binarized the image data of FIG. It is a figure which shows an example of the display image of 4th Embodiment. It is explanatory drawing which shows the principle which carries out a bird's-eye view imaging process of a camera image. It is a figure which shows an example which displayed the display image of 2nd Embodiment as a bird's-eye view, (a) shows the area | region over which it is over the image data of FIG. 19, (b) is an example of the image which looked over. It is a figure which shows an example of the display image of 3rd Embodiment.

A: First Embodiment of the Present Invention Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. Here, a dump truck as a transport vehicle is applied to a self-propelled industrial machine. Of course, any self-propelled industrial machine other than a dump truck, such as a hydraulic shovel having a lower traveling body or the like may be applied. There are a rigid type and an articulated type as a dump truck, either of which may be applied. The point is that the present embodiment can be applied to any self-propelled industrial machine that performs a predetermined operation (such as transportation or digging). In the following, “left” is the left side when viewed from the driver's cab, and “right” is the right side when viewed from the driver's cab.

FIG. 1 shows an example of the dump truck 1 of the present embodiment. The dump truck 1 includes a vehicle body frame 2, front wheels 3 (3L and 3R), rear wheels 4 (4L and 4R), a vessel 5, a camera 6, a cab 7, an image processing device 8, and a monitor 9 There is. The vehicle body frame 2 forms the main body of the dump truck 1, and front wheels 3 and rear wheels 4 are provided in front of the vehicle body frame 2. The front wheel 3R is the right front wheel 3 and the front wheel 3L is the left front wheel 3. The rear wheel 4R is a right rear wheel 4 and the rear wheel 4L is a left rear wheel 4. A vessel 5 is a loading platform for loading soil, minerals, and the like. The vessel 5 is configured to be able to be undulated.

A camera 6 as an imaging unit can be installed at an arbitrary position on the dump truck 1. FIG. 2 shows a side view of the dump truck 1, and the camera 6 is mounted in front of the cab 7. The camera 6 performs imaging of the peripheral area of the dump truck 1 in a field of view range 6A (the range of the broken line in the drawing) which looks diagonally downward and forward of the dump truck 1. In the visual field range 6A, the worker M is present as an obstacle to avoid contact. Other obstacles include work machines and service cars, which may be included in the visual field range 6A.

The field of view range 6A of the camera 6 is obliquely downward in front of the dump truck 1 so that part of the front structure 10 is included in the field of view range 6A. As shown by the broken line in FIG. 2, the visual field range 6A includes a part of the front structure 10. Note that the visual field range 6A which is the peripheral area photographed by the camera 6 may be a direction other than the front of the dump truck 1, or any direction such as the rear, left or right of the dump truck 1 may be a visual field. By setting the visual field range 6A to the rear of the dump truck 1, it is possible to display an image of the rear situation when moving the dump truck 1 backward. At this time, the visual field range 6A includes a part of the rear structure.

The operator's cab 7 is provided with various operation means for the operator to ride and operate the dump truck 1. For example, a shift lever or the like for moving the dump truck 1 forward or backward is provided as the operating means. An image processing device 8 and a monitor 9 are provided in the driver's cab 7, and the image processing device 8 performs predetermined image processing on image data generated by the camera 6 taking a picture. The image data subjected to the image processing is displayed on the monitor 9. The monitor 9 is a display device, and basically, an image taken by the camera 6 is displayed on the monitor 9.

The image processing device 8 is an image processing controller that performs image processing on image data of a video captured by the camera 6. Then, the image data subjected to the image processing is displayed on the monitor 9 as a display image. As shown in FIG. 3, the image processing apparatus 8 includes a distortion correction unit 21, a smoothing processing unit 22, an edge extraction unit 23, a structure edge determination unit 24, an edge line segment data determination unit 25, and a caution range setting unit 26. And a distortion restoration unit 27 and a display image generation unit 28.

The image captured by the camera 6 is input to the distortion correction unit 21 as image data. The distortion correction unit 21 corrects distortion of image data using lens parameters of the camera 6 (focal length, optical axis center of image, lens distortion coefficient, etc.). The image reflected by the lens of the camera 6 is not distorted at the center, but is distorted as it goes away from the center. Therefore, the image data obtained by photographing by the camera 6 is distorted as a whole. On the other hand, image data with high linearity can be obtained by performing correction using the above-mentioned parameters.

The smoothing processing unit 22 receives the image data corrected by the distortion correction unit 21 and performs smoothing processing. The smoothing process is a process of reducing a change in pixel value between each pixel of image data to generate a smooth image. By performing the smoothing process on the image data, it is possible to smooth the change of the pixel value, and it is possible to make the image data smoothed as a whole. Although any method can be used for the smoothing process, for example, a moving average filter, a median filter, a Gaussian filter or the like can be used.

The edge extraction unit 23 performs edge extraction processing on the image data that has been subjected to the smoothing processing by the smoothing processing unit 22. For this reason, pixels having a large change in pixel value between adjacent pixels are extracted as edge pixels, and a shape in which the edge pixels are continuously formed is set as an edge. Thus, an edge having a large change in pixel value is extracted from the image data. Although any method can be used for the edge extraction process, for example, a method such as Sobel filter or Canney's algorithm can be used.

Among the edges extracted by the edge extraction unit 23, the structure edge determination unit 24, which is a structure detection unit, uses an edge whose shape is close to the boundary of the front structure 10 in the image data as the edge of the front structure 10. judge. For this purpose, the structure edge determination unit 24 straightens the edges in the image data using a linearization method such as Hough transformation. Then, a straightened edge is determined as an edge line segment, and an edge line segment having high connectivity is determined as a boundary (edge) of the front structure 10. The edge line segment can be expressed using the coordinates in the image data, and here, the intersection coordinates of the edge line segments, the edge line segment, and the lower end of the image data (closest to the dump truck 1 among the image data P The coordinates of the point of intersection with the existing end) are stored as edge line segment data.

The edge line segment data determination unit 25 compares an arbitrary number of edge line segment data, including the latest edge line segment data. Since the camera 6 has a predetermined imaging cycle and image data is periodically acquired, edge line segment data is also periodically acquired. The edge line segment data determination unit 25 stores the past edge line segment data, and calculates an average value with the latest edge line segment data. Then, the average value of the calculated edge line segment data is determined as the determined edge line segment data. Each part from the distortion correction unit 21 to the edge line segment data determination unit 25 is a structure detection unit for detecting the front structure 10.

The caution range setting unit 26 sets a caution range based on the edge line segment data input from the edge line segment data determination unit 25. The relative positional relationship between the vigilance range and the front structure 10 of the dump truck 1 is constant, and the vigilance range and the front structure are based on the mounting height or depression angle of the camera 6, the number of pixels of the camera 6, the angle of view, etc. The relative positional relationship with 10 can be recognized as known. Therefore, the caution range setting unit 26 sets a predetermined range as the caution range based on the boundary of the front structure 10 obtained by the edge line segment data.

The distortion restoration unit 27 restores distortion using the camera lens parameters distortion-corrected by the distortion correction unit 21. The image data up to the distortion restoration unit 27 is data in which distortion of the camera lens of the camera 6 is corrected, and distortion occurs in the image of the original camera 6. Therefore, when displaying the image of the camera 6 inherently, the distortion restoration unit 27 performs processing to restore distortion to the image data. Due to the restoration of this distortion, distortion also occurs in the required alert range in the image data.

The display image generation unit 28 generates a display image to be displayed on the monitor 9. The display image at this time is image data in which distortion is restored, and a caution range is set in this. The display image creation unit 28 outputs the display image in which the alert range is set to the monitor 9 and displays the display image on the monitor 9. As a result, the operator boarding the cab 7 can recognize the image of the camera 6 by visually recognizing the monitor 9.

Next, the operation will be described based on the flowchart of FIG. The camera 6 installed on the dump truck 1 captures a view range 6A with a view obliquely downward, and performs shooting at a predetermined shooting cycle (for example, frame / second). As a result, image data of one frame obtained for each shooting cycle is periodically output to the image processing device 8 (step S1). The camera 6 is installed so that the front structure 10 is included in the visual field range 6A, and the front structure 10 is included in the image data.

An example of the image data obtained by the camera 6 photographing the visual field range 6A is shown in FIG. The camera 6 has a field of view obliquely downward, and as shown in the figure, the ground in the shooting direction of the camera 6 is projected on the image data P as a background. The site where the dump truck 1 is operated is under a severe environment, the ground is not leveled, and there are many irregularities. In the image data P, it is projected as the uneven portion R. Further, the field of view of the camera 6 is set so that the front structure 10 of the dump truck 1 is projected in the field of view range 6A. Therefore, the front structure 10 is also included in the image data P.

The image data P described above is output to the distortion correction unit 21 of the image processing device 8 and is also output to the display image generation unit 28. The image data P input to the distortion correction unit 21 is subjected to distortion correction based on the lens parameters of the camera 6 (step S2). By performing distortion correction of the image data P using the lens parameters of the camera 6, distortion of a portion separated from the center of the image data P is corrected. Thereby, the image data P is corrected to an image having high linearity as shown in FIG.

The image data P subjected to distortion correction processing is input to the smoothing processing unit 22. The smoothing processing unit 22 performs a smoothing process on the image data P (step S3). As described above, the smoothing processing unit 22 generates smooth image data P by reducing the difference in pixel value between adjacent pixels for each pixel of the image data P. As described above, the edge extraction unit 23 performs edge extraction processing. At this time, the pixel values of the image data P become sparse values due to unevenness of the ground, a difference in illuminance, or the like, and these may be detected as edges. However, these are noise components, and the smoothing processing unit 22 performs smoothing processing in order to remove noise components in advance. By performing this smoothing process, the sparse pixel values are smoothed and noise components are removed. Here, the smoothing processing is performed using the smoothing filter shown in FIG. 7, but any method can be used as long as the difference in pixel value between adjacent pixels can be reduced.

FIG. 7 shows an example in which a Gaussian filter is applied as the smoothing filter. The Gaussian filter is a filter in which the filter coefficients of pixels close to one pixel (central pixel) in the image data P are large and the filter coefficients of pixels far from the central pixel are small in consideration of the spatial arrangement of pixels. The Gaussian filter then takes a weighted average of each filtered pixel. For example, the following Gaussian function (Expression 1) can be used as the filter coefficient. In addition, (sigma) in Formula 1 is a standard deviation.

In the example of FIG. 7, a Gaussian filter of 3 pixels × 3 pixels in total of 9 pixels is used. The number of pixels of the Gaussian filter is not nine but may be 25 pixels in total of 5 pixels × 5 pixels, or more. The Gaussian filter using the Gaussian function of Equation 1 described above is configured of a central pixel and peripheral pixels (eight pixels). The filter coefficient K of the central pixel is the largest, the filter coefficients L of the four pixels above, below, and to the left of the central pixel are the second largest, and the filter coefficients M of the other four pixels are the smallest. That is, "K>L>M". The filter coefficients are weighted, and the sum of the filter coefficients is "1".

The smoothing processing unit 22 performs the smoothing processing using the above-described Gaussian filter on all the pixels of the image data P. Specifically, the pixel value of the central pixel is multiplied by the filter coefficient K, the pixel values of the four pixels above, below, left, and right of the central pixel are multiplied by the filter coefficient L, and the pixel values of the other four pixels are multiplied. Then, the filter coefficient M is multiplied. Then, with respect to the pixel value of the central pixel multiplied by the filter coefficient K, the pixel values of the surrounding eight pixels multiplied by the filter coefficients L and M are added. The pixel value of the central pixel when this addition is performed becomes the smoothed value. This process is performed on all the pixels of the image data P.

By the way, as described above, in the image data P, the uneven portion R on the ground and the front structure 10 are shown. The pixel of the uneven portion R has a difference between its surrounding pixels and the pixel value. Therefore, when the smoothing processing unit 22 performs the smoothing process on the image data P, the difference in pixel value between the uneven portion R and the surrounding pixels is reduced, and the image data P is smoothed. Then, the values of the filter coefficients K, L and M described above are values such that the boundary of the front structure 10 in the image data P is detected as an edge, but the other uneven portion R etc. is not detected as an edge. Set to As a result, as shown in FIG. 8, the uneven portion R in the image data P can be significantly reduced. Since the values of the filter coefficients K, L, and M are determined by the standard deviation σ of the Gaussian function, the standard deviation σ is set such that the uneven portion R is not detected.

Of course, the difference between the pixel of the uneven portion R and the surrounding pixels may become large depending on the size, the illumination condition, etc. in the uneven portion R. In this case, even if the smoothing process is performed, the uneven portion R can not be completely removed from the image data P, and a portion of the uneven portion R remains. However, as shown in FIG. 8, the number can be significantly reduced.

On the other hand, for the front structure 10, the boundary between the background and the front structure 10 is clearly distinguishable. Generally, the dump truck 1 is often colored in a color that can be easily distinguished from the ground or the surrounding environment from the viewpoint of grasping the surrounding situation. Along with this, the front structure 10 also has a clearly different color from the background such as the ground and the surrounding environment. Therefore, the difference between the pixel value of the front structure 10 and the pixel value of the background of the boundary between the background and the front structure 10, that is, the front structure 10, is large. Thereby, since the difference of the pixel value in the boundary of front structure 10 is very large, even if it performs smoothing processing, most remain.

Next, the process of the edge extraction unit 23 will be described. As shown in FIG. 8, in the image data P subjected to the smoothing processing, the boundary between the concavo-convex portion R and the front structure 10 whose number is greatly reduced is shown. The edge extracting unit 23 extracts, as an edge pixel, a pixel having a difference in pixel value between adjacent pixels equal to or larger than a predetermined threshold (a pixel having a large difference) from the image data P subjected to the smoothing process (step S4) .

Here, a Canny algorithm is used as edge extraction processing for extracting edge pixels. Of course, other methods may be used to extract edge pixels. The Canny algorithm uses two types of Sobel filters, a Sobel filter in the horizontal direction (horizontal direction: X direction) and a Sobel filter in the vertical direction (longitudinal direction: Y direction) as shown in FIG. The Sobel filter in the X direction emphasizes the contour in the X direction of the image data P, and the Sobel filter in the Y direction emphasizes the contour in the Y direction. Edge pixels are extracted for all pixels of the image data P using these two Sobel filters.

By applying Sobel filters in the X and Y directions to all the pixels of the image data P, the differential values (fx, fy) between the pixels and the surrounding pixels for all the pixels of the image data P are applied. Extract). fx is a differential value in the X direction, and fy is a differential value in the Y direction. Then, the edge strength g is calculated as “g = (fx ² + fy ² ) ^1/2 ” based on fx and fy for all pixels of P of the image data. The edge strength g indicates the difference in pixel value between one pixel and the surrounding pixels, and edge pixels are detected based on the calculated edge strength g.

The edge extraction unit 23 compares the edge intensities g of all the pixels of the image data P subjected to the Sobel filter using the first threshold T1 and the second threshold T2 (where T1> T2). . Focusing on one pixel of the image data P, if the edge strength g of the pixel is equal to or greater than the first threshold T1, the pixel is detected as an edge pixel. On the other hand, if the edge strength g of the focused pixel is less than the second threshold T2, the pixel is detected as not being an edge pixel. When the edge strength of the focused pixel is equal to or more than the first threshold T1 and less than the second threshold T2, when the edge pixel exists in the vicinity of the pixel (in particular, the pixel adjacent to the pixel) It is detected as a pixel, otherwise it is not detected as an edge pixel.

When the above edge extraction processing is performed, image data P in which edge pixels are extracted as shown in FIG. 10A is generated. Then, a shape in which edge pixels are continuous is extracted as an edge E. The front structure 10 is determined from the image data P by performing the structure edge determination process on the image data P from which the edge E is extracted (step S5). In the edge extraction process, pixels having a large difference between adjacent pixels are extracted as edge pixels. As a result, since the pixels that originally projected the uneven portion R are hardly detected as edge pixels, the accuracy of detecting the front structure 10 is dramatically improved.

On the other hand, the boundary of the front structure 10 is likely to be detected as an edge pixel because the difference in pixel value is large. However, by performing edge extraction processing using a Sobel filter, as shown in FIG. 10A, the edge E may not exhibit linearity and may exhibit irregular changes. For example, the boundary of the front structure 10 which originally has linearity may lose linearity, and part of the boundary may not be detected.

The structure edge determination unit 24 extracts a straight line from the edge pixel of the image data P. Although arbitrary means can be used as a means to extract a straight line, Hough transform is used as a known method here. Of course, if straight lines can be extracted from edge pixels, methods other than Hough transform may be used. By performing arithmetic processing of Hough transform on the image data P shown in FIG. 10 (a), line segments indicating irregular changes are straight lines L and L1 to L4 as shown in FIG. 10 (b). become. Among them, attention is paid to straight lines L1 and L2 extending in the X direction and straight lines L3 and L4 extending in the Y direction. Then, the structure edge determination unit 24 selects the longest straight line L1 among the straight lines L1 and L2, and selects two straight lines L3 and L4.

The structure edge determination unit 24 extends straight lines L1, L3, and L4 in the image data P. The straight line L1 is an extending straight line in the X direction, and the straight lines L3 and L4 are straight lines extending in the Y direction. By extending these lines, the straight line L1 intersects with the straight line L3 and the straight line L4. The intersection of the straight line L1 and the straight line L3 is C1, and the intersection of the straight line L1 and the straight line L4 is C2. Further, an intersection between the straight line L3 and the lower end of the image data P is C3, and an intersection between the straight line L4 and the lower end of the image data P is C4. Intersection points C1 to C4 indicate the coordinates of pixels in the two-dimensional image data P.

When the intersection points C1 to C4 are connected by a straight line, an edge line segment of the front structure 10 is formed. Therefore, the information of the intersections C1 to C4 is data for specifying the edge line segment of the front structure 10, and this is used as edge line segment data. An area surrounded by the edge line segment data (coordinate information of the intersections C1 to C4) is the front structure 10. The structure edge determination unit 24 determines the edge of the front structure 10 based on the edge line segment data. Although the front structure 10 may be specified based on edge line segment data obtained from one image data P, in order to improve the accuracy of detecting the front structure 10 in the image data P, the edge line The minute data determination unit 25 compares edge line segment data (step S6).

As described above, the image data P is input from the camera 6 to the image processing device 8 at a predetermined cycle, and a plurality of pieces of image data P can be obtained. The edge line segment data determination unit 25 stores edge line segment data based on past image data P, and calculates an average value of edge line segment data based on the latest image data P and past edge line segment data. . The edge line segment data determination unit 25 sets the calculated edge line segment data (average value) as the determined edge line segment data. That is, the front structure 10 in the image data P is identified by the determined edge line segment data.

When the edge line segment data determination unit 25 calculates the average value of edge line segment data of a plurality of image data P, the edge line segment data determination unit 25 stores edge line segment data of a predetermined number of image data P. The determination of the edge line segment data is not performed until it is done. In addition, by calculating the average value of many edge line segment data, the accuracy of the edge line segment data is improved, but the processing time becomes longer. Therefore, in consideration of the balance between the accuracy of the edge line segment data and the processing time, the number of edge line segment data to be calculated is determined.

The edge line segment data determined by the edge line segment data determination unit 25 is output to the caution range setting unit 26. The caution range setting unit 26 sets a caution range based on edge line segment data (step S7). The critical range is a range for detecting an obstacle, and is a predetermined range from the dump truck 1. As described above, the relative positional relationship between the front structure 10 of the dump truck 1 and the caution range can be recognized as known. Therefore, if the front structure 10 in the image data P can be identified, the alert range can be automatically recognized.

FIG. 11A shows a state in which the caution range 31 is set based on edge line segment data. The determined edge line segment data is information of the intersections C1 to C4, and the front structure 10 is identified by connecting the intersections C1 to C4. Therefore, the area surrounded by the intersections C1 to C4 in FIG. The alert range 31 is set in the image data P based on the relative positional relationship between the front structure 10 and the alert range 31.

On the other hand, in the image data P, an area other than the alert range 31 becomes an alert-unnecessary range 32 that does not require alerting. Therefore, when an obstacle is detected from the image data P, the obstacle is detected only in the alert range 31, and the obstacle is not detected in the alert unnecessary range 32. The method of obstacle detection can be any method. Then, when detecting an obstacle, the alarm-unnecessary area 32 is a mask area, and by masking the area, it is possible to narrow down the alarm area 31 and detect an obstacle.

The caution range 31 is set based on the image data P subjected to distortion correction by the distortion correction unit 21. The display image displayed on the monitor 9 is image data P output from the camera 6 and is an image not subjected to distortion correction. Therefore, the distortion restoration unit 27 restores the distortion correction for the image data P for which the caution range 31 has been set (step S8). The distortion restoration unit 27 restores the distortion correction of the image data P using the lens parameters of the camera 6 used by the distortion correction unit 21. Thus, the alert range 31 is deformed as shown in FIG. Information on the alerting range 31 subjected to the distortion restoration processing is output to the display image creation unit 28.

The display image generation unit 28 generates an image (display image) to be displayed on the monitor 9 (step S9). The display image generation unit 28 periodically inputs image data P from the camera 6. The image data P is an image that has not been subjected to distortion correction. Further, the display image creation unit 28 inputs the information of the alert range 31 of which the distortion restoration processing has been performed from the distortion restoration unit 27. Then, the display image creation unit 28 superimposes the caution range 31 on the image data P input from the camera 6. The state is shown in FIG. That is, in the image data P shown in the figure, the caution range 31 is an area surrounded by virtual lines. The caution area 31 is an area for detecting an obstacle, and it is not necessary to display the caution area 31 exceptionally as shown in FIG.

Then, the display image creation unit 28 outputs the image data P input from the camera 6 to the monitor 9. An operator who gets in the cab 7 can obtain an auxiliary visual field by visually recognizing the image data P displayed on the monitor 9.

As described above, in the present embodiment, the front structure 10 of the dump truck 1 is specified based on the image data P, and the alert range 31 is set from the front structure 10. As a result, the alerting range 31 for detecting an obstacle can be automatically set without manual operation, and the time and effort for setting can be largely omitted. Further, even if the dump truck 1 vibrates violently to cause a shift in the visual field range 6A of the camera 6, there is no shift in the caution range 31. That is, the relative positional relationship between the front structure 10 and the caution range 31 in the image data P is constant as the visual field range 6A is shifted, so the caution range 31 is determined based on the front structure 10 By setting, the occurrence of a shift in the alert range 31 is avoided.

In the above, the structure edge determination unit 24 detects the linearity of the edge pixel to determine the edge line segment of the front structure 10, but if the boundary between the front structure 10 and the background can be detected A method other than the method of detecting the edge line segment of the front structure 10 may be used. For example, when the difference in pixel value between the front structure 10 and the background is extremely large, the front structure 10 can be detected based on the difference in pixel value. In this case, it is not necessary to perform distortion correction by the distortion correction unit 21, nor is it necessary to perform distortion correction by the distortion restoration unit 27. However, since the front structure 10 has a unique linearity, the accuracy of detecting the boundary between the front structure 10 and the background is improved by adopting a method of detecting the edge line segment of the front structure 10 Do.

Further, the smoothing processing unit 22 performs smoothing processing on the image data P, thereby reducing the uneven portion R in the image data P. In this respect, in the case where the ground taken by the camera 6 is leveled to a certain extent, the uneven portion R in the image data P is reduced. Therefore, the edge E of the front structure 10 can be extracted without performing the smoothing process. However, in order to improve the detection accuracy of the edge E of the front structure 10, it is preferable to perform the smoothing process.

In addition, although the edge line segment data determination unit 25 improves the accuracy of the edge line segment data, the edge line segment data determination unit may be used when the edge line segment can be clearly identified based on one image data P. 25 may omit the process. However, by determining the edge line segment data by calculating the average value of the edge line segment data of a plurality of image data P, the accuracy of the edge line segment data is improved. For this reason, it is desirable that the processing by the edge line segment data determination unit 25 be performed.

B: Second Embodiment of the Present Invention Next, a second embodiment will be described. FIG. 13 shows the configuration of the image processing apparatus 8 of the second embodiment. As shown in FIG. 13, the alert range image generation unit 41, the obstacle detection unit 42, the mask image generation unit 43, and the deviation determination unit 44 are newly added to the first embodiment, but other than that Since the configuration of is the same as that of the first embodiment, the description will be omitted.

Image data P is periodically output to the image processing apparatus 8 from the camera 6 which is photographing the visual field range 6A. The image data P is output not only to the distortion correction unit 21 and the display image generation unit 28 but also to the caution range image generation unit 41. The caution range image generation unit 41 generates an image of only the caution range 31 in the image data P, as shown in FIG. For this purpose, the alert restoration range 31 in which the distortion is restored is input from the distortion restoration unit 27, and the image data P is input from the camera 6. Then, of the image data P, only the image within the area of the caution area 31 is generated as the caution area image.

The caution range image is input to the obstacle detection unit 42. The obstacle detection unit 42 detects an obstacle in the alert range image. Workers, service cars and the like are mainly assumed as obstacles for which contact must be avoided, but, for example, other work machines and the like also become obstacles. Arbitrary methods can be used for the method of detecting an obstacle. The obstacles include a stationary state and a moving state, and a stationary obstacle is a stationary body, and a moving obstacle is a moving body.

An arbitrary method can be used as a method of detecting a stationary body, for example, a method of detecting a region of a stationary body by compositely judging an outline, luminance, color or the like in an image can be used. The stationary body has a certain area, and when the contour of the certain area has a predetermined shape, when the luminance value has a predetermined value, when the color shows a specific color, etc. The above-mentioned certain area can be detected as a stationary body.

As a specific method of detecting a stationary body, for example, pattern matching method of determining an obstacle by matching using shape information learned in advance based on information of contours of a background and a stationary body can be used. . Also, use a method of extracting an image feature having specific luminance and color information but extracting only pixels, extracting the shape feature such as area, perimeter, and moment of the extracted pixel area, and identifying the shape as an obstacle. Can.

Arbitrary methods can be used also as a method of detecting a moving object, and for example, when comparing temporally different image data P, the moving object is moving when a certain area is moving to a different position To detect that. For example, a block matching method can be applied to the detection of a moving object. The block matching method will be described later. In addition, gradient data detected using the relationship between the spatial gradient and the temporal gradient of brightness at each pixel of the image data P, and image data and latest image data when there is no area of the moving object It is possible to use a background subtraction method for detecting a difference between the two and a frame subtraction method for generating two difference images using three pieces of image data of different times to detect a moving object region.

The mask image generation unit 43 generates a mask image. As described in the first embodiment, the alert range setting unit 26 sets the alert range 31 in the image data P, and sets the other area as the alert unnecessary range 32. Among these, the mask image generation unit 43 generates a mask image with the alert unnecessary area 32 as a mask area. The mask image is an image of the same size as the image data P, and the number of pixels is also equal. Then, in the mask image, the mask area (the area 32 not requiring alarming) and the other area (the area 31 requiring alarming) are clearly distinguished. For example, all mask regions in the mask image can be configured in black (zero pixel value), and all regions in the critical range 31 can be configured in white (pixel value 256).

The shift determination unit 44 receives the mask image from the mask image generation unit 43 and determines the shift. The shift determination unit 44 holds a past mask image, and compares the newly input mask image with the past mask image. Then, the difference between the new mask image and the past mask image is calculated for all the pixels, and when there are a predetermined number or more of pixels having a difference of pixel values equal to or more than a predetermined value, the camera 6 is shifted. In the other cases, it is determined that no deviation has occurred. The determination result is output to the display image creation unit 28.

Next, the operation of the second embodiment will be described using the flowchart of FIG. Steps S1 to S8 shown in FIG. 14 are the same as in the first embodiment, and thus the description thereof is omitted. In step S8, the alert range 31 in which distortion is restored is generated. The information of the alerting range 31 is input to the alerting range image generation unit 41. The alert range image generation unit 41 receives the image data P from the camera 6, and an area other than the alert range 31 of the image data P, that is, the alert unnecessary area 32 is an insensitive region 33 P's information has been deleted. The insensitive region 33 has the same function as the mask region described above. Here, the insensitive regions 33 are all composed of black pixels (zero pixel value) (note that black pixels are shaded in FIG. 15 for convenience of the drawing).

Thereby, the alert range image P1 as shown in FIG. 15 is generated (step S10). The worker M is included as an obstacle in the alert range image P1. The caution range image P1 has the same number of pixels as the image data P in the X and Y directions, and the image data P is displayed in the caution range 31 portion, but black in the insensitive region 33. Only the pixels are displayed. In other words, the black pixels in the insensitive area 33 are a mask, and the unmasked area in the image data P is the caution area 31. As shown in the figure, a large number of uneven portions R exist in the alert range 31 and a worker M exists as an obstacle. Such an alert range image P1 is output to the obstacle detection unit 42.

The obstacle detection unit 42 detects an obstacle from the alert range image P1 (step S11). Although any method can be used to detect an obstacle, the case where a block matching method is used will be described here. FIG. 16 shows an example of a moving object detection method using the block matching method. The obstacle detection unit 42 stores the past alert range image P1 and compares the latest alert range image P1 with the past alert range image P1. Here, the alert range image P1 one cycle before the imaging cycle of the camera 6 is taken as the past alert range image P1.

In the example of FIG. 16, a total of 16 pixels of 4 pixels in the X direction and 4 pixels in the Y direction is taken as a region of interest A1. Of course, the attention area A1 may adopt the number of pixels other than 16 pixels. Then, a total of 64 pixels of 8 pixels in the X direction and 8 pixels in the Y direction are set as the search range A2 with the attention area A1 as the center. Of course, the number of pixels in the search range A2 can also be set arbitrarily.

The obstacle detection unit 42 sets the search range A2 to a predetermined position of the image data P, for example, an end in the X direction and the Y direction. Then, the search range A2 is searched. The search method in the search range A2 will be described with reference to FIG. The comparison area A3 is set in the search range A2. The comparison area A3 has four pixels in the X direction and four pixels in the Y direction, as in the case of the attention area A1, and has a total of 16 pixels. Then, the comparison area A3 is set to the end in the X direction and the Y direction.

The obstacle detection unit 42 calculates the difference in pixel value between corresponding pixels of the attention area A1 in the past caution range image P1 and the comparison area A3 in the latest caution range image P1. The attention area A1 and the comparison area A3 are both configured by 4 pixels in the X direction and 4 pixels in the Y direction, and one pixel in the attention area A1 and one pixel in the comparison area A3 correspond 1: 1. doing. Therefore, as described above, the difference calculation of the pixel values between corresponding pixels is performed. By performing this operation, it is possible to obtain difference values for 16 pixels. Then, the difference value for 16 pixels is respectively squared, and the operation is performed to add the squared 16 difference values. The value obtained by performing this calculation is taken as the addition value.

The obstacle detection unit 42 determines whether the added value is equal to or less than a predetermined threshold. The threshold value at this time is a value capable of determining whether or not the attention area A1 and the comparison area A3 can be regarded as identical. When the added value is equal to or less than a predetermined threshold value, the obstacle detection unit 42 determines that the past attention area A1 matches the latest comparison area A3. On the other hand, in other cases, it is determined that the past attention area A1 and the latest comparison area A3 do not match.

If the past attention area A1 matches the latest comparison area A3 and both do not completely match within the search range A2, then the attention area A1 moves within the search range A2 Can be recognized. As shown in FIG. 17, the comparison area A3 is shifted by one pixel in the X direction for comparison. Let this be a search (scan) in the X direction. When scanning in the X direction is completed, next, the comparison region A3 is shifted by one pixel in the Y direction, and scanning in the X direction is performed again. Performing scanning in the X direction while shifting one pixel in the Y direction is referred to as scanning in the Y direction. Therefore, scanning is performed in the X direction and the Y direction. FIG. 18 shows the method of this scan three-dimensionally.

The scanning in the X direction and the Y direction is performed until it is detected that the past attention area A1 matches the latest comparison area A3. Therefore, if it is detected that the past attention area A1 matches the latest comparison area A3, the scan is ended at that point. On the other hand, when all the scans in the X direction and the Y direction are completed, if a matching past focused area A1 and the latest comparison area A3 are not detected, a mobile object (a failure Object: It is recognized that there is no worker M).

The search range A2 is a limit range that can be moved when the worker M normally walks until the latest alert range image P1 is acquired since the past alert range image P1 is acquired. Determined as a standard. As described above, the past alert range image P1 is an image one cycle before the latest alert range image P1. Therefore, the search range A2 is set based on the limit amount when the worker M normally walks during one cycle of the photographing cycle of the camera 6.

By the way, the camera 6 has a case where the imaging cycle is fast and a case where it is slow. At this time, when the shooting cycle of the camera 6 is high speed, even if the worker M is moving, it is recognized that it hardly moves. On the other hand, when the photographing cycle is low, even if the worker M is moving at low speed, it is recognized as moving at high speed. Therefore, when the shooting cycle of the camera 6 is fast, the vigilant range image P1 relatively in the past, which is a plurality of cycles earlier, is set as the vigilance range image P1 of the past. On the other hand, when the shooting cycle of the camera 6 is slow, the relatively latest (for example, one cycle before) the alert range image P1 is set as the past alert range image P1.

The obstacle detection unit 42 causes the search within the search range A2 to be performed on all the pixels of the alert range image P1. Therefore, when the search range A2 is at the end of the X direction and Y direction, when the search is completed, the next search area A2 is shifted in the X direction by the pixel (8 pixels) in the X direction, Do a search for Then, when the search for all the pixels in the X direction of the alert range image P1 is completed, the search range A2 is shifted in the Y direction by the pixel in the Y direction (8 pixels), and the same search is performed again. Then, the search within the search range A2 is performed on all the pixels of the caution required range image P1.

Here, as also shown in FIG. 16, the alert range image P <b> 1 is distinguished into an alert range 31 for detecting an obstacle and a blind area 33. The obstacle detection unit 42 recognizes the insensitive area 33 in the alert range image P1. That is, the pixels included in the insensitive area 33 are recognized. Therefore, the obstacle detection unit 42 excludes the dead area 33 when performing a search based on the search range A2. As a result, the worker M as an obstacle can be detected only in the alert range 31. Then, the obstacle detection unit 42 outputs the detected information on the worker M as the obstacle to the display image generation unit 28.

The display image creation unit 28 receives the image data P from the camera 6, and receives the information of the worker M as an obstacle from the obstacle detection unit 42. Therefore, the display image creation unit 28 recognizes the worker M in the alert range 31 of the image data P. The display image creation unit 28 superimposes an obstacle marking display M1 on the periphery of the worker M in order to explicitly display the obstacle. Here, the obstacle marking indication M1 is a circumscribed rectangle formed around the worker M, but may of course not be a rectangle.

The image data P on which the obstacle marking indication M1 is superimposed is shown in FIG. The image data P becomes a display image and is output to the monitor 9. Then, as shown in the figure, the obstacle marking display M1 is superimposed around the worker M as an obstacle, so that the operator looks at the presence of the worker M based on the obstacle marking display M1. Can be recognized. In addition, although the caution range 31 is shown with a virtual line in FIG. 19, it has shown that the detection of the obstruction was performed only in this range. Although the alerting range 31 is basically not displayed on the monitor 9, it may be displayed.

By the way, it is possible to perform peripheral monitoring by performing viewpoint conversion of the camera image acquired by the camera 6 by image processing and converting it into a bird's-eye view image, and a configuration for displaying this peripheral monitoring image on the monitor 9 is known conventionally. There is. That is, as shown in FIG. 29, when the camera 6 is disposed at a predetermined height position and photographed with the optical axis CA of the objective lens directed obliquely downward at an angle α, a through image P is obtained. From the through image P having this angle θ, as shown by the phantom line in the same figure, an image is generated in which the optical axis VA is perpendicular to the through image P from the virtual viewpoint VF. As a result, it is possible to obtain a bird's-eye view image obtained by planarly viewing a camera image from diagonally above with an angle α. (Note that the image data P is a through image, and an image obtained by viewpoint conversion of the data of the through image to be an upper viewpoint is a bird's eye image. Therefore, in order to distinguish it from the bird's eye image, The acquired image data is called through image P.)

In each embodiment described above, the camera image (through image) acquired by the camera 6 is displayed, but it is also possible to perform signal processing from this camera image to form a bird's-eye view image and display it on the monitor 9. That is, as shown in FIG. 30A, viewpoint conversion processing is performed in which the through image P is an upper viewpoint.

Then, as shown in FIG. 30 (b), cameras are installed in the dump truck 1 at four locations, the front position, the left and right both positions, and the rear position, respectively, and the periphery of the dump truck 1 In order to obtain camera images of the camera, perform overhead image processing on these camera images, and display them on the monitor 9, the front side camera, the left and right side Each bird's-eye view image in each image area SPF, SPL, SPR, SPB can be acquired and displayed on the monitor 9. FIG. 30A shows an image area SPF of the front camera. At the time of this display, an icon image Q having the dump truck 1 in a planar shape is disposed at the center of the screen of the monitor 9, and image areas SPF, SPL, SPR, SPB are respectively provided in four directions around the icon image Q. Display the corresponding overhead image.

As described above, by displaying the overhead view image of the surrounding situation by combining around the icon image Q and displaying the bird's-eye view of the surrounding situation, it is possible to obtain the monitoring image displayed so as to look around the entire periphery of the dump truck 1. Moreover, this surveillance image can display the obstacle marking display M1 indicating the worker M in the same manner as the through image P of FIG. 30 (a), and further, the position relative to the dump truck 1 because of the overhead image. The display mode can also be grasped.

Next, the case where a shift occurs in the visual field range 6A of the camera 6 installed in the dump truck 1 will be described. As described above, the vibration of the dump truck 1 may cause a shift in the visual field range 6A of the camera 6. Therefore, it is determined whether or not a shift occurs in the visual field range 6A. However, the deviation of the visual field range 6A has an allowable range, and it is determined that the deviation occurs when the amount of deviation exceeds the allowable range, and it is determined that the deviation does not occur if within the allowable range.

As shown in FIG. 13, the information on the alert range 31 is input from the alert range setting unit 26 to the mask image generation unit 43. The mask image generation unit 43 generates a mask image based on the image data P and the caution area 31 (step S12). In the mask image, as shown in FIG. 20, the caution range 31 of the image data P is formed of white pixels (pixel value 256), and the caution unnecessary range 32 is formed of black pixels (zero pixel value).

The mask image generation unit 43 holds the image data P in the past (for example, one cycle before) and the information of the caution range 31 and holds the mask image in the past. Then, when the latest image data P and the information of the alert range 31 are input, the latest mask image is generated. The mask image generation unit 43 outputs the past mask image and the latest mask image to the deviation determination unit 44.

The shift determination unit 44 calculates the shift amount based on the input past mask image and the latest mask image, and based on the calculated shift amount, whether or not the shift of the visual field range 6A is within the allowable range. It determines (step S13). For this purpose, the shift determination unit 44 calculates the pixel difference between the past mask image and the latest mask image.

The pixel value becomes zero when the difference between pixels in which the caution range 31 overlaps in the past mask image and the latest mask image is calculated. That is, the pixel becomes black. In addition, when the difference between pixels where the unneeded area 32 for warning is overlapped is calculated between the past mask image and the latest mask image, the pixel value becomes zero. That is, the pixel becomes black. On the other hand, when the difference between the pixels in which the critical range 31 and the non-critical range 32 overlap is calculated, the absolute value of the calculation result is 256. That is, the pixel is white. The difference image P2 in FIG. 20 is an image resulting from the calculation of the difference.

When a shift occurs in the visual field range 6A, white pixels are generated in the difference image P2. The number of pixels of this white pixel indicates the amount of deviation. That is, if there is no shift in the visual field range 6A, the past mask image and the latest mask image completely match, and therefore, there are no pixels in which the caution range 31 and the caution unnecessary range 32 overlap. For this reason, white pixels are not present. On the other hand, if a shift occurs in the visual field range 6A, there is a portion where the caution range 31 and the caution unnecessary range 32 overlap, so the pixel of the portion becomes a white pixel.

A predetermined number of pixels is set as a threshold in the deviation determination unit 44, and the number of white pixels of the difference image P2 is compared with the threshold. This threshold indicates the allowable range of deviation of the visual field range 6A. The allowable range can be set arbitrarily. For example, a deviation that affects the visibility of the operator can be set as the allowable range. Then, as a result of comparing the pixel count of the white pixel with the threshold, if the pixel count of the white pixel is less than the threshold, the shift determining unit 44 determines that the view range 6A is not shifted. On the other hand, if the number of white pixels is equal to or greater than the threshold value, it is determined that the viewing range 6A has a shift.

The shift determination unit 44 outputs information on whether or not there is a shift to the display image creation unit 28. The display image generation unit 28 displays the shift information G on the image data P when the information that the shift is generated is input from the shift determination unit 44. FIG. 21 shows an example of the image data P on which the deviation information G is superimposed. The shift information G is displayed at the corner of the image data P, and is a mark for visually recognizing that the camera 6 is shifted.

The deviation information G is a notification means for causing the operator to recognize that the camera 6 has generated a deviation greater than the allowable range, and based on the deviation information G displayed on the monitor 9, the operator Can be recognized. The operator who recognizes this can return the camera 6 to a normal position, and can eliminate the shift of the visual field range 6A. Alternatively, the maintenance personnel etc. can return the camera 6 to the normal position by the operator contacting the maintenance personnel etc.

As described above, even if a shift occurs in the visual field range 6A, the relative positional relationship between the front structure 10 and the caution range 31 in the image data P is constant, so the caution range 31 is always set correctly Be done. In FIG. 22A, the edge extraction unit 23 performs an edge extraction process on the image data P, and shows image data P from which an edge pixel is extracted. Then, the structure edge determination unit 24 extracts a straight line from the image data P using a method such as Hough transform. FIG. 22B shows a state in which a straight line is extracted from the image data P.

In the first embodiment described above, the structure edge determination unit 24 detects straight lines in the X direction and the Y direction because no shift occurs in the visual field range 6A. However, in the present embodiment, since the inclination range is generated in the visual field range 6A, the front structure 10 in the image data P is also inclined. Therefore, the structure edge determination unit 24 of the present embodiment also extracts a straight line that is inclined at a predetermined angle from the X direction and the Y direction. That is, with the predetermined angle as the inclination angle θ, a straight line within the range of the inclination angle θ is extracted among the extracted straight lines.

Then, of the straight lines inclined within the range of the inclination angle θ from the X direction, the longest straight line is determined as the straight line of the front structure 10 in the X direction being inclined. Here, the straight line L5 is the longest straight line. Next, among the straight lines detected at the lower end (the end closest to the dump truck 1 of the image data P) than the straight line L5 extended to both ends of the image data P, the inclination angle from the Y direction Extract a straight line within the range of θ. Here, straight lines L7 and L8 are extracted. Then, straight lines L7 and L8 are extended. Thus, as shown in FIG. 22B, four intersections C5 to C8 of the straight line L5 and the straight lines L7 and L8 are detected. These intersections C5 to C8 become edge line segment data.

FIG. 23 (a) shows a state in which the alert range 31 is set based on edge line segment data. Since a shift in the tilt direction occurs in the visual field range 6A, the caution range 31 also tilts. At the same time, the caution-unwanted range 32 is also inclined. However, the alert range 31 is all included in the image data P. Then, an image obtained by performing the distortion restoration process on the image data P of FIG. 23 (a) becomes the image data P shown in FIG. 23 (b). The image data P shown in FIG. 23B is output to the caution range image generation unit 41, and a caution range image as shown in FIG. 24 is generated. The warning image is output to the obstacle detection unit 42, and the obstacle detection unit 42 detects the worker M as an obstacle.

Therefore, even if a shift occurs in the visual field range 6A due to the vibration of the dump truck 1, it is not necessary to set the caution range 31 again, and the correct caution range 31 is set based on the front structure 10. Thereby, even if a shift occurs in the visual field range 6A, it is possible to detect an obstacle limited to the caution range 31. However, the visibility of the operator is reduced by the occurrence of the deviation of inclination in the visual field range 6A. In such a case, the position of the camera 6 may be corrected to eliminate the inclination shift.

C: Third Embodiment of the Present Invention Next, a third embodiment will be described. In the present embodiment, the alert range 31 is changed according to the traveling speed of the dump truck 1. The range in which the obstacle should be warned also changes depending on whether the dump truck 1 is stopped, traveling at a low speed, or traveling at a high speed. When the dump truck 1 is in the stopped state, it is not necessary to set the warning range 31 to a very wide range, but when traveling at a certain speed, it is necessary to set the wide range to the warning range 31.

Therefore, as shown in FIG. 25, in the present embodiment, a traveling speed detection unit 45 is added to the second embodiment. The dump truck 1 is provided with means for detecting the traveling speed, and outputs the detected traveling speed to the critical alert range setting unit 26. The alert range setting unit 26 changes the size of the alert range 31 based on the input information on the traveling speed.

Here, when the dump truck 1 is stopped and when traveling at a low speed or a high speed, the relative positional relationship between the front structure 10 and the alert range 31 is constant. Therefore, a predetermined range can be set as the caution range 31 based on the traveling speed. Specifically, as shown in FIG. 31, the size of the critical alert range 31 needs to be within the critical alert range 311 from the stop zone, the critical alert range 312 at low speeds, and the critical alert ranges at high speeds It can be changed to be 313. Here, the low speed region is a speed at which the dump truck 1 stops at about that position when the travel stop operation is performed. It should be noted that the above-described alert range in the operation of the dump truck 1 is a range requiring alertness and is not an image display range on the monitor 9.

D: Fourth Embodiment of the Present Invention In the first, second and third embodiments, the dump truck 1 is applied as a self-propelled industrial machine, but in the fourth embodiment, the dump truck 1 is used. A hydraulic shovel is applied as a self-propelled industrial machine. Here, the case where the camera 6 is installed in the hydraulic shovel, the rear structure 50 of the hydraulic shovel is extracted, and the alert range 31 is set based on the rear structure 50 will be described.

FIG. 26 shows image data P subjected to distortion correction processing by the distortion correction unit 21. The rear structure 50 of the hydraulic shovel is often rounded, and the boundary between the rear structure 50 and the background in the image data P has curvilinearity with a predetermined radius of curvature. The aft structure 50 of the hydraulic shovel is colored in a color distinct from the background color. Therefore, the brightness value of the rear structure 50 and the brightness value of the background are largely different. So, in this embodiment, the pixel which has a luminance value more than a predetermined luminance value (threshold value) about all the pixels of image data P is extracted. In addition, the threshold value at this time is set to a luminance value that can extract a color colored in the rear structure 50.

Thereby, the image data P is binarized with the threshold as a boundary, and as shown in FIG. 27, the pixels constituting the rear structure 50 are extracted. However, in addition to the rear structure 50, there may be an edge pixel having a luminance value equal to or greater than a threshold value in the uneven portion R. Therefore, the area of the pixel in contact with the lower end of the screen among the extracted pixels is determined as the rear structure 50. Thereby, the rear structure 50 is specified in the image data P.

FIG. 28 shows an example of a display image displayed on the monitor 9. The relative positional relationship between the rear structure 50 and the alert range 51 is constant, and by specifying the rear structure 50 in the image data P, the alert range 51 indicated by a virtual line is also set. Then, the worker M can be detected as an obstacle by limiting to the alert range 51 required. Note that FIG. 6 shows the case where the field of view range is tilted, so that the entire screen is displayed tilted. Then, the shift information G is superimposed on the image data P.

In the present embodiment, since the rear structure 50 is specified by binarization, each process by the smoothing processing unit 22, the edge extraction unit 23, the structure edge determination unit 24, and the edge line segment data determination unit 25 described above Is unnecessary. However, as long as the rear structure 50 having a predetermined curvilinear property can be detected, the rear structure 50 may be specified by the processing of each of these portions. Further, in the embodiment described above, that is, in the dump truck 1, when the front structure 10 is specified, the method of binarization may be used as in the present embodiment.

In each of the embodiments described above, the image displayed on the monitor 9 may be a camera image itself, or may be displayed as a bird's-eye view image by performing viewpoint conversion processing based on the principle of FIG.

Reference Signs List 1 dump truck 6 camera 6A view range 8 image processing device 9 monitor 10 front structure 21 distortion correction unit 22 smoothing processing unit 23 edge extraction unit 24 structure edge determination unit 25 edge line segment data determination unit 26 caution range setting unit 27 distortion restoration unit 28 display image creation unit 31, 311, 312, 313 vigilance range 32 vigilance unnecessary range 33 dead zone 41 vigilance range image generation unit 42 obstacle detection unit 43 mask image creation unit 44 deviation determination unit 45 traveling speed Detection unit G Misalignment information M Worker M1 Obstacle marking indication P Image data (through image)
P1 Warning range image SPF, SPL, SPR, SPB Image area

Claims (6)

A photographing unit (9) provided in the self-propelled industrial machine (1) and photographing a peripheral area including a part of a structure (10) constituting the self-propelled industrial machine (1) in a field of view;
A structure detection unit (25) that detects the structure (10) in the image data based on the image data captured by the imaging unit (9);
Among the image data, an obstacle (M) located in the peripheral area is detected, and a predetermined range including the obstacle (M) is set as a required alert range (31, 311, 312, 313). An alert range setting unit (26) for setting an alert range (31, 311, 312, 313) in the image data based on the structure (10) detected by the structure detection unit (25);
Image processing device for a self-propelled industrial machine equipped with The obstacle detection unit (42) according to claim 1, further comprising: an obstacle detection unit (42) configured to detect the obstacle (M) only in the alert range (31, 311, 312, 313) in the image data. Image processing device for self-propelled industrial machines. In the image data, a range other than the alert range (31, 311, 312, 313) is taken as the alert unnecessary range (32), and the difference between the pixel values of each pixel in the alert unnecessary range (32) Deviation determination to determine whether or not the visual field range of the imaging unit (9) has deviated beyond the allowable range based on the number of pixels of pixels whose difference in pixel value is equal to or greater than a predetermined threshold value. The image processing apparatus for a self-propelled industrial machine according to claim 2, further comprising: a section (44). The self-propelled industrial machine (1) includes a traveling speed detection unit (45) for detecting a traveling speed, and the alert range (31, 311,...) Based on the traveling speed detected by the traveling speed detection unit (45). The image processing apparatus for a self-propelled industrial machine according to claim 2, wherein The image processing apparatus for a self-propelled industrial machine according to claim 2, wherein the image data is displayed as an overhead image. A photographing unit (9) is provided in the self-propelled industrial machine (1), and a surrounding area including a part of the structure of the self-propelled industrial machine (1) is photographed,
The structure (10) in the image data is detected based on the image data captured by the imaging unit (9),
Detecting an obstacle (M) located in a peripheral area of the image data;
The structure detection unit (25) detects the alert range (31, 311, 312, 313) with the predetermined range including the obstacle (M) as the alert range (31, 311, 312, 313) Setting in the image data based on the structure (10)
The image processing method of the self-propelled industrial machine which performs the process which detects the said obstruction (M) only with respect to the said caution range (31, 311, 312, 313) in the said image data.

Images