KR102296205B1 - Lane departure detecting device and controlling method thereof - Google Patents

Abstract

A lane departure detection apparatus and control method are disclosed. The control method of the lane departure detection apparatus includes the steps of receiving an image including a lane, generating a composite image by applying a random deformation value to the input image, and generating when the absolute value of the random deformation value is greater than a preset threshold value and labeling the synthesized image as a deviation image, and labeling the synthesized image generated as a normal image when it is smaller than a preset threshold.

Description

LANE DEPARTURE DETECTING DEVICE AND CONTROLLING METHOD THEREOF

The present disclosure relates to a lane departure detection apparatus and a control method, and more particularly, to a lane departure detection apparatus and a control method for automatically labeling learning data.

Research on a technology for recognizing an object from a photographed image is in progress. A technology for recognizing an object is widely used in a security device, a vehicle safety device, and the like. In the past, a technology in which a developer sets a standard for recognizing an object, recognizes an object according to the set standard, and judges a situation has been widely used. However, the method in which the developer sets the standard has a problem in that when an unexpected event occurs, an object recognition error or an error in situation determination occurs.

However, as research on artificial intelligence has become active in recent years, advances in technology for learning artificial intelligence by using a large number of data in advance and the learned artificial intelligence recognizing objects and judging situations are being made. The learned artificial intelligence can properly respond to unexpected events by properly recognizing objects or judging situations using existing learning data. AI needs a lot of data to learn to respond appropriately to unexpected events. In the past, developers directly labeled the learning data to learn artificial intelligence. However, there was a problem in that a lot of time and money was generated because the developer had to label tens of thousands of training data directly. Therefore, there is a need for an efficient technique for learning artificial intelligence.

SUMMARY The present disclosure has been made to solve the above problems, and an object of the present disclosure is to provide an apparatus for detecting a lane departure and a control method for automatically labeling image data.

According to an embodiment of the present disclosure for achieving the above object, the control method of the apparatus for detecting lane departure includes receiving an image including a lane, and generating a composite image by applying a random deformation value to the input image and labeling the generated composite image as a deviation image when the absolute value of the random transformation value is greater than a preset threshold value, and labeling the generated composite image as a normal image when it is smaller than the preset threshold value. includes

In addition, the random deformation value may be at least one of a rotational deformation value and a shift deformation value.

Also, the control method of the apparatus for detecting lane departure may further include learning a neural network using the labeled composite image.

In addition, the step of learning the neural network generates a first matrix map by convolving the labeled composite image, and compares the generated first matrix map with 0 to generate a second matrix map in which a large value is returned, A characteristic map may be generated by resizing the generated second matrix map, the generated characteristic map may be connected, and classes of the labeled composite image may be classified based on the connected characteristic map.

Meanwhile, the class may be at least one of normal driving, left lane departure, and right lane departure.

On the other hand, the control method of the lane departure detection apparatus includes the steps of receiving a determination target image including a lane, detecting a lane from the determination target image, and normal driving based on the learned neural network and the detected lane position; The method may further include determining a left lane departure or a right lane departure.

According to an embodiment of the present disclosure for achieving the above object, the lane departure detection device includes an input unit for receiving an image including a lane and an automatic annotation unit for transforming and labeling the input image, and the automatic annotation The unit generates a composite image by applying a random transformation value to the input image, and when the absolute value of the random transformation value is greater than a preset threshold value, labels the generated composite image as a departure image, and the preset threshold value If it is smaller, the generated composite image is labeled as a normal image.

As described above, according to various embodiments of the present disclosure, the apparatus and control method for detecting lane departure may automatically label image data.

In addition, the lane departure detection apparatus and control method can label tens of thousands of image data within minutes.

In addition, the lane departure detection apparatus and control method may more accurately detect whether or not lane departure exists by learning artificial intelligence with more image data in a short time.

1 is a block diagram of an apparatus for detecting lane departure according to an embodiment of the present disclosure.
2 is a block diagram of an apparatus for detecting lane departure according to another exemplary embodiment of the present disclosure.
3 is a diagram illustrating a process of labeling image data according to an embodiment of the present disclosure.
4 is a view for explaining a process of learning lane departure according to an embodiment of the present disclosure.
5 is a diagram illustrating a result of a lane departure network according to an embodiment of the present disclosure.
6 is a flowchart of a method for controlling an apparatus for detecting lane departure according to an embodiment of the present disclosure.

Hereinafter, various embodiments will be described in more detail with reference to the accompanying drawings. The embodiments described herein may be variously modified. Certain embodiments may be depicted in the drawings and described in detail in the detailed description. However, the specific embodiments disclosed in the accompanying drawings are only provided to facilitate understanding of the various embodiments. Accordingly, the technical spirit is not limited by the specific embodiments disclosed in the accompanying drawings, and it should be understood to include all equivalents or substitutes included in the spirit and scope of the invention.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but these elements are not limited by the above-described terms. The above terminology is used only for the purpose of distinguishing one component from another component.

In this specification, terms such as "comprises" or "have" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof. When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

Meanwhile, as used herein, a “module” or “unit” for a component performs at least one function or operation. In addition, a "module" or "unit" may perform a function or operation by hardware, software, or a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “units” other than a “module” or “unit” that must be performed in specific hardware or are executed in at least one processor may be integrated into at least one module. The singular expression includes the plural expression unless the context clearly dictates otherwise.

In addition, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be abbreviated or omitted.

1 is a block diagram of an apparatus for detecting lane departure according to an embodiment of the present disclosure.

Referring to FIG. 1 , the lane departure detecting apparatus 100 includes a first input unit 110 and an auto annotator 120 . The first input unit 110 receives an image including a lane. The image including the lane may be a normal image in which the lane is not deviated. For example, the first input unit 110 may include a communication module connected through a communication interface, an input terminal connected through an input interface, and the like. That is, when the first input unit 110 is implemented as a communication module, the lane departure detecting apparatus 100 may receive an image including a lane using a wired/wireless communication method. Alternatively, when the first input unit 110 is implemented as an input terminal, the lane departure detecting apparatus 100 may receive an image including a lane from an internal or external storage device.

The automatic annotation unit 120 receives the image including the lane input through the first input unit 110 . The automatic annotation unit 120 generates a random transformation value and then applies the random transformation value to the input image to generate a composite image. For example, the arbitrary deformation value may include at least one of a rotational deformation value and a shift deformation value. As an embodiment, the automatic annotation unit 120 may apply an arbitrary rotational deformation value to the input image. When the automatic annotation unit 120 applies an arbitrary rotational transformation value to the input image, the input image is transformed into an image rotated left or right by the rotational transformation value. That is, the automatic annotation unit 120 may generate a composite image to which an arbitrary rotational deformation value is applied from the input image. Alternatively, the automatic annotation unit 120 may apply an arbitrary shift transformation value to the input image. When the automatic annotation unit 120 applies an arbitrary shift transformation value to the input image, the input image is transformed into an image shifted by the shift transformation value to the left or right. That is, the automatic annotation unit 120 may generate a composite image to which an arbitrary shift transformation value is applied from the input image. The automatic annotation unit 120 may generate a composite image to which an arbitrary rotational transformation value and an arbitrary shift transformation value are applied together.

The automatic annotation unit 120 may generate a composite image by applying different arbitrary deformation values to the input image. Accordingly, the automatic annotation unit 120 may generate a plurality of composite images to which different arbitrary deformation values are applied to one input image.

The automatic annotation unit 120 compares the absolute value of the arbitrary deformation value with a preset threshold value. For example, the preset threshold value may be set as an absolute value or a relative value according to an input image. As an embodiment, when the scale or the lane width of the input image is constant, the preset threshold may be set to an absolute value such as 20 degrees or 2 cm. When the scale or the lane width of the input image is varied, the preset threshold value may be set as a relative value such as 20% of the lane width of the input image. When the preset threshold value is set as a relative value according to the input image, the automatic annotation unit 120 may label images including various scales or various lane widths as normal images or lane departure images. .

The automatic annotation unit 120 compares the absolute value of the arbitrary deformation value with a preset threshold value, and labels the generated composite image as a lane departure image when the arbitrary deformation value is greater than the preset threshold value. Then, the automatic annotation unit 120 compares the absolute value of the arbitrary deformation value with a preset threshold value, and labels the generated composite image as a normal image when the arbitrary deformation value is less than the preset threshold value. That is, the automatic annotation unit 120 may generate a virtual lane departure image and label it as a lane departure image.

As described above, the automatic annotation unit 120 may generate a plurality of composite images by applying different arbitrary deformation values to one input image. In addition, the automatic annotation unit 120 may generate a plurality of labeling images for a plurality of composite images. Accordingly, the existing technology can generate only one labeling image for one input image, but the lane departure detecting apparatus 100 according to the present disclosure may also generate a plurality of labeling images for one input image.

Also, the automatic annotation unit 120 may receive a plurality of images and generate a plurality of labeling images. Meanwhile, the lane departure detecting apparatus 100 may further include an additional configuration.

2 is a block diagram of an apparatus for detecting lane departure according to another exemplary embodiment of the present disclosure.

Referring to FIG. 2 , the lane departure detecting apparatus 100a includes a first input unit 110 , an automatic annotation unit 120 , a convolution neural network (CNN) 130 , a second input unit 140 , and an output unit 150 . may include. Since the first input unit 110 and the automatic annotation unit 120 are the same as those of FIG. 1 , descriptions thereof will be omitted.

The CNN 130 learns about lane departure using the labeled composite image. A detailed process in which the CNN 130 learns lane departure will be described later. When the CNN 130 completes learning about lane departure using the composite image, the lane departure detection apparatus 100a may receive an actual image and detect whether lane departure is present.

The second input unit 140 may receive a determination target image including a lane. For example, similarly to the first input unit 110 , the second input unit 140 may include a communication module connected through a communication interface, an input terminal connected through an input interface, and the like. That is, when the first input unit 110 is implemented as a communication module, the lane departure detecting apparatus 100a may receive a determination target image including a lane using a wired/wireless communication method. Alternatively, when the first input unit 110 is implemented as an input terminal, the lane departure detecting apparatus 100a may receive a determination target image including a lane from an internal or external storage device. Also, the second input unit 140 may include a camera. That is, when the second input unit 140 is implemented as a camera, the lane departure detecting apparatus 100a may receive an actual driving image including a lane photographed by the camera. When the second input unit 140 is implemented as a camera, an actual driving image including a lane may be a determination target image.

The CNN 130 may detect a lane from the judgment target image. Then, the CNN 130 may determine whether to depart from the lane based on the detected lane and learned lane departure data. That is, the CNN 130 may determine whether normal driving, left lane departure, or right lane departure from the determination target image.

The output unit 150 may output a lane departure warning based on a lane departure detection result. For example, the output unit 150 may include a buzzer, a speaker, a motor, a haptic module, an LED, a display, and the like. When the output unit 150 is implemented as a buzzer, the output unit 150 may output various types of warning sounds. When the output unit 150 is implemented as a speaker, the output unit 150 may output a voice indicating left lane departure or right lane departure. When the output unit 150 is implemented as a motor or a haptic module, the output unit 150 may output various patterns of vibration. When the output unit 150 is implemented as an LED, the output unit 150 may output various patterns of light. When the output unit 150 is implemented as a display, the output unit 150 may display a message indicating left lane departure or right lane departure. The output unit 150 may be implemented as one unit or may be implemented as a plurality of units. For example, when the output unit 150 is implemented with a buzzer, a motor, and an LED, the output unit 150 may simultaneously output a warning sound, vibration, and light.

Meanwhile, the CNN 130 may not only determine whether the left lane departs or the right lane departs, but also determines the degree of departure. As an embodiment, the CNN 130 may determine that the left lane departure degree is 10%, 20%, or 30%. When the CNN 130 determines the degree of lane departure, the output unit 150 outputs a preset type of warning sound corresponding to the degree of departure, a vibration pattern, a light pattern, a voice output of a preset period, or a message displayed in a preset period. You can also print

As described above, the lane departure detecting apparatus 100a may include a first input unit 110 , an automatic annotation unit 120 , a CNN 130 , a second input unit 140 , and an output unit 150 . Meanwhile, the lane departure detecting device 100a may be implemented as a separate device in which the learning device 100a-1 and the application device 100a-2 are separate devices. That is, the learning apparatus 100a - 1 of the lane departure detecting apparatus 100a may train the CNN 130 by generating a labeling image including only the first input unit 110 and the automatic annotation unit 120 . And, the application device 100a-2 can determine whether the lane departure using the CNN 130 learned about lane departure, including the CNN 130, the second input unit 140 and the output unit 150. have.

3 is a diagram illustrating a process of labeling image data according to an embodiment of the present disclosure.

Referring to FIG. 3 , the structure of the automatic annotation unit is illustrated. First, the automatic annotation unit receives the video image through the input unit (S11). The video image is a normal image that does not deviate from a lane, and may be a video image of one frame. The automatic annotation unit may generate a composite image using an arbitrary transformation value (S12). For example, any transformation value may be any rotational transformation value or any shift transformation value. The automatic annotation unit may generate a plurality of composite images by applying various arbitrary transformation values from one video image.

The automatic annotation unit may compare an absolute value of an arbitrary deformation value with a preset threshold value for each generated composite image (S13). For example, the preset threshold may be a rotation threshold or a shift threshold. The auto-annotator may compare with a rotation threshold when the arbitrary transformation value is an arbitrary rotational transformation value, and compare with a shift threshold when the arbitrary transformation value is an arbitrary shift transformation value.

The automatic annotation unit may compare the absolute value of the arbitrary deformation value with a preset threshold value, and label the normal image when the absolute value of the arbitrary deformation value is less than the preset threshold value (S14-1). And, the automatic annotation unit may label the deviation image when the absolute value of the arbitrary deformation value is greater than a preset threshold value (S14-2). The case where the arbitrary deformation value is equal to the preset threshold value is not shown in FIG. 3 . However, when an arbitrary deformation value is equal to a preset threshold value, the automatic annotation unit may be set to label the image as either a normal image or a deviation image.

As described above, since the lane departure detecting apparatus can automatically generate a plurality of labeling images from one video image, it is possible to easily and quickly generate a labeling image for CNN learning. The lane departure detection apparatus may train a deep neural network using the generated labeling image.

4 is a view for explaining a process of learning lane departure according to an embodiment of the present disclosure.

4, the structure of the CNN 22 is shown. In FIG. 4 , the CNN 22 including three each of a convolutional layer, a Rectified Linear Unit (ReLU) layer, and a pooling layer is illustrated, but the number of each layer may be variously set. The CNN 22 may receive the labeling image 21 as input. For example, the labeling image may have a size of 360×120 and may be a color image including RGB data. And, the labeling image input to the CNN 22 may be an image obtained by subtracting the average of the entire training image before being input to the CNN 22 .

The labeling image input to the CNN 22 may be generated as a first matrix map by a preset filter in the first convolutional layer. The generated first matrix map is transferred to the first ReLU layer using ReLU as an activation function. The activation function refers to a function that implements the process of being activated when an action potential signal transmitted to a human synapse exceeds the minimum stimulus value and transmitted to the next neuron. That is, the ReLU layer compares the generated first matrix map with 0 to generate a second matrix map in which a large value is returned.

The second matrix map processed in the first ReLU layer is transferred to the first pooling layer. As an embodiment, the pooling layer may perform average pooling. The pooling layer serves to reduce the size of an image, and average pooling refers to a method of taking an average of pixels within a pooling window. The map (or image) processed in the first pooling layer may be resized to half the size of the original image. Accordingly, the image processed by the CNN 22 shown in FIG. 4 may be output as a feature map having a size of 45×15.

The CNN 22 shown in FIG. 4 includes two fully connected layers (FC), and may each have 1024 nodes. The number of FCs and the number of nodes described above is an example, and the CNN 22 may include various numbers of FCs and the number of nodes according to an implementation method. FC connects the characteristics of the characteristic map, and the characteristic map to which the characteristics are connected can be classified into one of three classes through the softmax probability processing process. For example, the three classes may include normal driving, left lane departure and right lane departure.

In FIG. 4, an embodiment of classifying lane departure into three classes has been described, but depending on the implementation method, the CNN 22 is 10% lane departure left, 20% lane departure left, 30% lane departure left, 10% lane departure right Lane departure may be classified into classes of departure, right 20% lane departure, and right 30% lane departure.

5 is a diagram illustrating a result of a lane departure network according to an embodiment of the present disclosure.

5, various images determined by CNN are shown. As shown in FIG. 5 , the CNN may classify images into classes of left lane departure, right lane departure, and normal driving through a learning process.

So far, various embodiments of the lane departure detection device have been described. Hereinafter, a flowchart of a method for controlling the lane departure detection device will be described.

6 is a flowchart of a method for controlling an apparatus for detecting lane departure according to an embodiment of the present disclosure.

Referring to FIG. 6 , the lane departure detecting apparatus receives an image including a lane ( S610 ). For example, the image including the lane may be a normal driving image in which the lane is not deviated. The lane departure detecting apparatus generates a composite image by applying an arbitrary deformation value to the input image (S620). The arbitrary transformation value may be a rotation transformation value that rotates the input image or a shift transformation value that shifts the input image left and right. The lane departure detecting apparatus may generate a plurality of composite images by applying various arbitrary deformation values to one input image.

The lane departure detecting apparatus compares the absolute value of the arbitrary deformation value with a preset threshold value (S630). The preset threshold may be a rotation threshold or a shift threshold.

When the absolute value of the arbitrary deformation value is less than a preset threshold value, the lane departure detection apparatus labels the normal image (S640). When the absolute value of the arbitrary deformation value is greater than a preset threshold value, the lane departure detection apparatus labels the departure image as a departure image (S650). The lane departure detection device may train the CNN using the labeling image.

The ~ method according to the above-described various embodiments may be provided as a computer program product. The computer program product may include the S/W program itself or a non-transitory computer readable medium in which the S/W program is stored.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100, 100a: lane departure detection device 110: first input unit
120: automatic annotation 130: CNN
140: second input unit 150: output unit

Claims (7)

receiving an image including a lane;
generating a composite image by applying a random transformation value to the input image; and
Labeling the generated composite image as a deviation image when the absolute value of the random transformation value is greater than a preset threshold, and labeling the generated composite image as a normal image when less than the preset threshold. A method of controlling a lane departure detection system. According to claim 1,
The random transformation value is
A method of controlling a lane departure detecting device that is at least one of a turning deformation value and a shift deformation value. According to claim 1,
Learning a neural network using the labeled composite image; The control method of the lane departure detection device further comprising a. 4. The method of claim 3,
The step of learning the neural network is,
Convolving the labeled composite image to generate a first matrix map, comparing the generated first matrix map with 0 to generate a second matrix map in which a large value is returned, and resizing the generated second matrix map (resizing) to generate a characteristic map, connect the generated characteristic map, and classify a class of the labeled composite image based on the connected characteristic map. 5. The method of claim 4,
The class is
A method of controlling a lane departure detecting device that is at least one of normal driving, left lane departure, and right lane departure. 6. The method of claim 5,
receiving a judgment target image including a lane;
detecting a lane from the judgment target image; and
and determining normal driving, left lane departure, or right lane departure based on the learned neural network and the detected lane position. an input unit for receiving an image including a lane; and
and an automatic annotation unit for labeling and transforming the input image.
The automatic annotation section,
A composite image is generated by applying a random transformation value to the input image, and when the absolute value of the random transformation value is greater than a preset threshold value, the generated composite image is labeled as a departure image, and greater than the preset threshold value. A lane departure detection device that labels the generated composite image as a normal image when it is small.

Images