WO2019147024A1 - Object detection method using two cameras having different focal distances, and apparatus therefor - Google Patents

Abstract

A front object detection method for an autonomous vehicle is disclosed. The front object detection method for an autonomous vehicle, according to an embodiment of the present invention, comprises the steps of: obtaining a first image from a first camera having a first focal distance; obtaining a second image from a second camera having a second focal distance that is relatively longer than the first focal distance; detecting an object from the first image and the second image; determining corresponding regions between the first image and the second image; removing an image characteristic from the corresponding region of the first image; and detecting a front object on the basis of the object detection result of the second image and the object detection result of the first image from which the image characteristic has been removed.

Description

Method and apparatus for object detection using two cameras having different focal lengths

The present invention relates to an object detection method and apparatus. Specifically, the present invention relates to an object detecting method and apparatus for detecting a small object that is difficult to detect using two cameras having different focal lengths.

Object detection in the fields of computer vision and machine learning is of great importance in a variety of fields, from medicine to robotics. Human vision is very robust to localize and identify objects in current scenarios, but it is very difficult to manipulate in computers. The ultimate goal of object detection is to localize and identify instances of the same or different objects in the image.

Common object detection methods use color and texture information, shape signals, and state features. The advent of deep neural networks revolutionized object detection technology. Complex ensemble methods that combine several low-level functions for object detection have been removed. The accuracy of this method is much lower than in deep networks.

There is a proposed convolution neural network (CNN) for object detection, which proposes a region using selective retrieval and then uses CNN to create a feature representation in a hierarchical manner. The final layer uses support-vector networks (SVMs) to classify objects and whether or not they are bounding boxes. However, matching the bounding box is considered a regression problem.

An improved version of the previously known R-CNN was developed as Fast-RCNN. To improve the time efficiency and accuracy of Fast-RCNN object detection, specifically, we remove duplicate CNN trained for each local proposal and train only a single CNN for a single image. The regional proposal offers an area using trained CNN facilities.

Recently, vehicles are often equipped with front and rear view cameras. It would be very economical to be able to reuse cameras available for vehicle-related applications such as vehicle and pedestrian detection, lane and traffic sign detection. Detecting and recognizing remote objects is currently a tricky issue and is a requirement for high-speed vehicles.

A camera with a longer focal length can capture farther scenes. On the other hand, cameras with short focal lengths have wider field of view and can capture larger information. Thus, using two cameras helps detect objects in a wide area as well as distant objects.

Prior Art Documents:

Ryoji Tanabe and Alex Fukunaga, "Improving the Search Performance of SHADE Using Linear Population Size Reduction ", Proc. IEEE Congress on Evolutionary Computation, 2014.

D. G. Lowe: "Distinctive image features from scale-invariant keypoints", International Journal of Computer Vision, 2004. W. Liu et al., "SSD: Single Shot MultiBox Detector," ArXiv 151202325 Cs, vol. 9905, pp. 21-37, 2016.

Reno, Jimmy and Chen, Xiaohao and Liu, Jianbo and Sun, Wenxiu and Pang, Jiahao and Yan, Qiong and Tai, Yu-Wing and Xu, Li: 'Accurate Single Stage Detector Using Recurrent Rolling Convolution', CVPR,

Ren, Shaoqing and He, Kaiming and Girshick, Ross and Sun, Jian: 'Faster R-CNN: Towards Real-time Object Detection with Region Proposals Networks' Proceedings of the 28th International Conference on Neural Information Processing Systems.

The present invention proposes an object detecting apparatus and method capable of detecting even a small object which is difficult to detect without using a high-cost high-quality camera.

A method of detecting a front object of an autonomous vehicle according to an embodiment of the present invention includes the steps of: obtaining a first image acquired from a first camera having a first focal length; obtaining a second focal length The method comprising the steps of: obtaining a second image obtained from a second camera having a first image and a second image, detecting an object from the first image and the second image, determining a corresponding region between the first image and the second image, The method comprising the steps of: removing an image feature in a corresponding area of a first image; and detecting an object ahead of the object based on the object detection result in the second image and the object detection result in the first image from which the image feature has been removed .

In this case, the step of detecting the forward object may include detecting the object in the first image except for the corresponding area, by removing the image feature in the corresponding area of the first image.

Wherein determining the corresponding region between the first image and the second image includes determining a corresponding region of the first image corresponding to an area of the object detected in the second image based on the parameter can do.

In this case, the parameter may include a pixel of the first image corresponding to a ratio of the focal distance of the first camera to the focal length of the second camera and the origin of the second image.

On the other hand, the viewing angle of the second camera having the second focal length may be narrower than the viewing angle of the first camera having the first focal distance.

On the other hand, the range photographed by the second camera may be a range in which a large number of objects are expected through learning using existing detection data.

Meanwhile, a front object detecting system of an autonomous vehicle according to an embodiment of the present invention includes a first camera having a first focal length, a second camera having a second focal length relatively longer than the first focal length, And a control unit for performing object detection based on the first image acquired from the first camera and the second image acquired from the second camera, wherein the control unit detects the object from the first image and the second image Determining a corresponding region between the first image and the second image, removing an image feature from a corresponding region of the first image, and comparing the object detection result in the second image with the first image, The object in front is detected based on the object detection result of the object.

In this case, the control unit may detect the object in the first image except for the corresponding area by removing the image feature in the corresponding area of the first image.

The control unit may include determining a corresponding region of the first image corresponding to an area of the object detected in the second image based on the parameter.

In this case, the parameter may include a pixel of the first image corresponding to a ratio of the focal distance of the first camera to the focal length of the second camera and the origin of the second image.

And the viewing angle of the second camera having the second focal length may be narrower than the viewing angle of the first camera having the first focal distance.

On the other hand, the range photographed by the second camera may be a range in which a large number of objects are expected through learning using existing detection data.

The present invention can detect a small object which is difficult to detect without using a high-cost camera and a high-performance processor by using a camera having two different focal lengths.

1 is a conceptual diagram of an object recognition apparatus according to an embodiment of the present invention.

Figure 2 illustrates the problem of duplicate detection, which can generally occur when using two input images.

3 shows a flow for calculating the relationship between two input images.

Figure 4 shows a process for object detection using two cameras with different focal lengths.

Figure 5 shows a comparison of the accuracy between the present invention and the conventional method.

6 is a flowchart illustrating an object detection method according to an embodiment of the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. It should be understood, however, that there is no intention to limit the invention to the embodiments described below, and that those skilled in the art, upon reading and understanding the spirit of the invention, It is to be understood that this is also included within the scope of the present invention.

The accompanying drawings are merely exemplary and are not to be construed as limiting the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Further, even if specific parts such as installation positions are different, the same names are given when the functions are the same, so that the convenience of understanding can be improved. When there are a plurality of identical configurations, only one configuration will be described, and the same description will be applied to the other configurations, and a description thereof will be omitted.

The existing object detection method uses one camera, and the problem that arises here is to detect objects of different size, especially small size. Recently, the object detection method seeks to detect objects of small size using all the calculated feature maps of the convolution network. However, the performance is limited and there is a problem that an object at a distance can not be detected.

1 is a conceptual diagram of an object recognition apparatus according to an embodiment of the present invention.

As shown in FIG. 1, an object recognition apparatus according to an embodiment of the present invention has a main camera 1 having a short focal length and a wide field of view, and conversely, A support camera 2 having a field of view is disclosed. A control unit (not shown) for processing an image acquired from the main camera 1 and an image acquired from the support camera 2 to perform object detection may be further included in the object recognition apparatus. Hereinafter, the operation of a system including a control unit or a control unit for processing images acquired from the main camera 1 and the support camera 2 will be described. Here, the control unit may be a processor and may be a processor on which a program configured according to the algorithm described below is executed.

In the present invention, as shown in FIG. 1, an object is detected using two cameras having different focal lengths. The cameras having short focal lengths are referred to as main cameras 1 and Cm, and the cameras having long focal lengths are referred to as support cameras 2 and Cs. The main camera 1 has a wide viewing angle and can photograph a wide scene. In contrast, the support camera 2 can capture a distant scene with a small field of view. The focal lengths of both cameras are arbitrary. Therefore, a sufficiently long focal length can be used to detect a very distant object. This feature is particularly useful for high-speed vehicles that must notify the driver of traffic signals or obstacles as quickly as possible.

However, when using two cameras with different focal lengths, some problems may occur. It is required to make the relationship between the two cameras such that corresponding areas between them can be calculated respectively.

In addition, the same object may appear in both an image captured by the main camera (hereinafter referred to as a 'main image') and an image captured by a support camera (hereinafter, a 'support image'). As a result, duplicate searches can be a problem.

Figure 2 illustrates the problem of duplicate detection, which can generally occur when using two input images.

(a) shows an image captured by a 35 mm focal length camera. (b) shows an image captured by a 12mm focal length camera. (c) shows the result of object detection using (a). (d) shows the result of object detection using (b).

Taking FIG. 2 as an example, a support image is captured with a 35 mm focal length camera, and a main image captured with a 12 mm focal length camera. A known algorithm, Faster-RCNN, is used to independently search for objects in two images. As a result, the same vehicle is detected twice, and therefore both images can not be used as direct inputs. Therefore, to solve this problem, another solution is needed that utilizes two images to avoid duplicate object detection.

An object detection method according to an embodiment of the present invention uses an L-SHADE algorithm (prior art document 1) to optimize a configured energy function.

Like other evolutionary algorithms, L-SHADE is a black-box optimization algorithm. To apply the L-SHADE algorithm to any problem, two main points are needed. First, configure the encoding for the problem you want to consider. At this stage, you need to determine the parameters of the problem you need to optimize and encode it in a format that your computer can process (for example, an array structure). Second, it creates an energy function that generates optimized parameter values at a minimized cost.

3 shows a flow for calculating the relationship between two input images.

Figure 4 shows a process for object detection using two cameras with different focal lengths.

The object detection framework according to an embodiment of the present invention includes two different processes. The first step is to calculate the relationship between the two input images as shown in FIG. 3, and to perform offline. This relationship is determined by the three parameters, the ratio of the two focal lengths, and the pixel (x, y) of the main image corresponding to the origin (0,0) of the support image.

In the present invention, SIFT (Prior Art 2) is used to find the number of corresponding points between two images. Then, the energy function is constructed using the calculated corresponding points. Here we use the L-SHADE algorithm to optimize the energy function. The optimized function produces the best calculated parameters r, x, and y.

The online process detects an object using two images as shown in FIG. First, a support image is used to detect an object using an existing object detection method. The region of the detected object is then mapped to the main image so that the corresponding region in the main image is not repeatedly detected. The object detection method is then used to detect objects in the processed main image. Finally, the objects detected in the two detection steps are combined and the final result for the main image is output.

First, Relation Computation is explained.

In the present invention, SIFT is used to detect the best correspondence point Nr between stereo images. In the present invention, a method for estimating a relationship between a main image and a support image is proposed. The method proposed by the present invention is based on L-SHADE as described above. There are two important points when using L-SHADE, which is to define the structure of the problem and construct the cost function.

1) Structure of problem: In offline mode, three parameters r, x and y are required for estimation. r represents the ratio of the two focal lengths. Therefore, the value may be a value between 1 and fmax. x and y are the pixels of the main image corresponding to the origin (0, 0) of the support image, and the pixel can be located anywhere in the main image. Thus, 0 <x <W, 0 <y <H, where W and H are the width and height of the main image, respectively.

2) Cost function: In the present invention, the error of the cost function constructed using the calculated correspondence between the main image and the support image is evaluated. In the present invention, SIFT is used to calculate an optimal correspondence point between two images.

In the present invention, the distance between the point (xis, yis) of the support image and the origin (0,0) is scaled by r using the optimum parameter values of r, x, y to the origin (x, y) (Xim, yim) of the corresponding main image. By applying L-SHADE to the definition of the problem structure and cost function, parameter values for r, x, and y with minimized errors can be obtained.

Second, we explain the region mapping.

In an online process, when an object is detected from a support image, the detected object area needs to be mapped to the main image to remove the image feature. As a result, objects in the main image are not detected again.

Suppose that the object area of the support image is a rectangular model with upper left coordinates (xs1, ys1) and lower right coordinates (xs2, ys2). Mapping an object region actually maps two points of a rectangle. The corresponding points of (xs1, ys1) and (xs2, ys2) in the main image are calculated as shown in the following equations (1) and (2).

Where k = 1, 2.

When the corresponding object area of the main image is calculated, the image feature is removed. In the present invention, the area is simply filled with zeros.

Third, a determination of support image is described.

The first and second steps described above need to know in advance which of the two input images is the support image (the focal length is larger). Therefore, in the present invention, it is proposed to reuse the SIFT corresponding point to estimate the support image.

Specifically, I1 and I2 are referred to as two input images, and s1 and s2 are referred to as two sets of corresponding points (Nd) for I1 and I2, respectively, and a difference between I1 and I2 and SIFT points It is a way to compare the sum of Euclidean distances. An image with a larger Euclidean distance sum is a support image. D (s1) and D (s2) can be referred to as the sum of Euclidean distances for two sets of SIFT points s1 and s2 and D (s1) and D (s2) have.

Where d (,) denotes the Euclidean distance between two points, and si denotes the i-th point in set s. If D (s1) - D (s2)> 0, I1 is a support image, and in the opposite case I2 is a support image.

Figure 5 shows a comparison of the accuracy between the present invention and the conventional method.

FIG. 5 shows a result of applying the conventional Faster-RCNN and RRC using only one image and the method using two images (DF) to six sequences as one embodiment of the present invention. Since Faster-RCNN and RRC are not designed to detect vehicles in two images as described above, they simply detect overlapping images using different focal length images. Therefore, the performance of Faster-RCNN and RRC is relatively low. On the other hand, DF + Faster-RCNN and DF + RRC, which are proposed in the present invention, recognize the redundant detection problem and can show higher performance than the conventional method.

6 is a flowchart illustrating an object detection method according to an embodiment of the present invention.

The object detection system according to an embodiment of the present invention acquires a main image and a support image (S101). Here, the main image is an image obtained from a camera having a relatively short focal length and a wide photographing range, and the support image is an image obtained from a camera having a relatively long focal distance and having a narrow photographing range.

In a specific embodiment, the object detection system may be installed in an autonomous vehicle and used for object detection in front of the running of the autonomous vehicle. For example, an autonomous vehicle may have two cameras with different focal lengths. And, the first camera may be a camera having a short focal length for photographing a wide range of the entire front surface of the vehicle, and the second camera may be a camera having a long focal distance for photographing a specific narrow range of the front of the vehicle.

Here, as an example, the specific range captured by the second camera may be a portion recognized as a road in the image captured by the first camera. As another example, the specific range captured by the second camera may be a portion where an object is expected to be relatively large through training using existing detection data.

The object detection system detects the object in the main image and detects the object in the support image (S103). Using the existing object detection method, the object detection system detects the object in the main image and detects the object in the support image. Here, the conventional object detection method may be an algorithm such as RCNN.

The object detection system determines a corresponding region by judging a relation between the main image and the support image (S105). The object detection system can determine the relation between the main image and the support image by using a known algorithm. Here, the existing algorithm may be SIFT.

The object detection system removes the image feature in the corresponding region of the main image (S107). Specifically, the object detection system may detect duplication of objects in a portion where the main image and the support image correspond to each other, so that the object detection system removes the image feature in the corresponding region of the main image so that the object is not detected in the corresponding region.

The object detection system performs object detection by integrating the object detection result in the main image and the object detection result in the support image (S109). Specifically, the object detection system performs object detection by summing the object detection result obtained from the support image and the object detection result obtained from the main image excluding the corresponding region.

Therefore, an autonomous vehicle having cameras having different focal lengths can detect an object photographed in a relatively small size by using only a general camera without installing a high-resolution camera.

The present invention described above can be embodied as computer-readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, , And may also be implemented in the form of a carrier wave (e.g., transmission over the Internet).

Accordingly, the above description should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

Claims (12)

Obtaining a first image acquired from a first camera having a first focal distance; Obtaining a second image acquired from a second camera having a second focal length relatively longer than the first focal distance; Detecting an object from the first image and the second image; Determining a corresponding region between the first image and the second image; Removing an image feature in a corresponding region of the first image; And And detecting an object ahead of the object based on the object detection result in the second image and the object detection result in the first image from which the image feature has been removed A method of detecting a forward object of an autonomous vehicle. The method according to claim 1, Wherein the step of detecting the forward object comprises: Removing an image feature in a corresponding region of the first image, and detecting an object in the first image except a corresponding region A method of detecting a forward object of an autonomous vehicle. The method according to claim 1, Wherein determining the corresponding region between the first image and the second image comprises: Determining a corresponding region of the first image corresponding to an area of the object detected in the second image based on the parameter A method of detecting a forward object of an autonomous vehicle. The method of claim 3, The parameter may comprise: Wherein the first image includes a pixel of the first image corresponding to a ratio of a focal distance of the first camera to a focal length of the second camera and an origin of the second image A method of detecting a forward object of an autonomous vehicle. The method according to claim 1, Wherein the viewing angle of the second camera having the second focal length is smaller than the viewing angle of the first camera having the first focal distance A method of detecting a forward object of an autonomous vehicle. The method according to claim 1, The range photographed by the second camera is a range in which objects are expected to be large through learning using existing detection data A method of detecting a forward object of an autonomous vehicle. A first camera having a first focal length; A second camera having a second focal length that is relatively longer than the first focal length; And And a control unit for performing object detection based on a first image acquired from the first camera and a second image acquired from the second camera, Wherein the control unit detects an object from the first image and the second image, determines a corresponding region between the first image and the second image, removes the image feature from the corresponding region of the first image, 2 &lt; / RTI &gt; image and the object detection result in the first image from which the image feature has been removed A front object detection system for an autonomous vehicle. 8. The method of claim 7, Wherein, Removing the image feature in the corresponding region of the first image, and detecting the object in the first image except the corresponding region A front object detection system for an autonomous vehicle. 8. The method of claim 7, Wherein, Determining a corresponding region of the first image corresponding to an area of the object detected in the second image based on the parameter A front object detection system for an autonomous vehicle. 10. The method of claim 9, The parameter may comprise: Wherein the first image includes a pixel of the first image corresponding to a ratio of a focal distance of the first camera to a focal length of the second camera and an origin of the second image A front object detection system for an autonomous vehicle. 8. The method of claim 7, Wherein the viewing angle of the second camera having the second focal length is smaller than the viewing angle of the first camera having the first focal distance A front object detection system for an autonomous vehicle. 8. The method of claim 7, The range photographed by the second camera is a range in which objects are expected to be large through learning using existing detection data A front object detection system for an autonomous vehicle.

Images