TWI684920B - Headlight state analysis method, headlight state analysis system, and non-transitory computer readable media - Google Patents

Abstract

A method for analyzing a headlight state is disclosed. The headlight state analysis method includes the following steps: obtaining a plurality of images, wherein the plurality of images are continuous in time; obtaining a plurality of feature vectors of the plurality of images according to a first convolutional neural network operation model; and determining a combination of the headlights corresponding to the plurality of images according to the plurality of feature vectors and a long-term and short-term memory time feature extraction model.

Description

Lamp status analysis method and lamp status analysis system And non-transitory computer readable media

This case is about a lamp status analysis method, a lamp status analysis system, and a non-transitory computer-readable medium, and in particular, the lamp status of a convolutional neural network operation model and a long-term and short-term memory time feature extraction model Analysis methods, vehicle light status analysis systems, and non-transitory computer-readable media.

Existing vehicle light detection and identification technology is easily affected by light and shadow climate environment and is unstable, and its identification success rate is low. In addition, the existing vehicle light detection and identification technology cannot use the same process to handle day and night light identification, and multiple thresholds must be set for the light identification in response to the day and night environment. The design complexity is very high.

One aspect of this case is to provide an analysis method law. The vehicle lamp state analysis method includes the following steps to obtain multiple images, wherein the multiple images are continuous in time; based on the first convolutional neural network operation model to obtain multiple feature vectors of multiple images; and based on multiple feature vectors And a long-term and short-term memory time feature extraction model to determine the combination of vehicle lights corresponding to multiple images.

Another aspect of this case is to provide a vehicle lamp status analysis system. The lamp status analysis system includes a storage device and a processor. The storage device is used to store the first convolutional neural network operation model, the second convolutional neural network operation model, and the long and short-term memory time feature extraction model. The processor is electrically connected to the storage device. The processor is used to obtain multiple regions of interest of multiple images, obtain multiple feature vectors of multiple images according to the first convolutional neural network operation model, and extract feature models based on multiple feature vectors and long and short-term memory time features Determine the combination of lights corresponding to multiple images.

Another aspect of the case is to provide a non-transitory computer readable medium, which includes at least one instruction program, and the processor executes at least one instruction program to implement the vehicle lamp status analysis method. The method for analyzing the state of the light includes: acquiring multiple regions of interest of multiple images; acquiring multiple feature vectors of multiple images according to the first convolutional neural network operation model; and based on multiple feature vectors and long-term and short-term memory time Feature extraction model to determine the combination of vehicle lights corresponding to multiple images.

Therefore, according to the technical aspect of the case, the embodiments of the case utilize a convolutional neural network operation model and long-term and short-term memory by providing a method for analyzing the state of the lamp, a system for analyzing the state of the lamp, and a non-transitory computer-readable medium. Compared with the existing technology, the time feature extraction model does not need to set multiple parameter thresholds. And can adapt to various weather conditions.

100‧‧‧Car light status analysis system

110‧‧‧Storage device

130‧‧‧ processor

DB1 to DB4‧‧‧ models

V‧‧‧Image

C1 to C5‧‧‧Convolutional layer

P1 to P4‧‧‧ Pooling layer

FC1, FC2 ‧‧‧ fully connected network layer

Output‧‧‧ Output feature vector

x _t-1 , x _t , x _t+1 ‧‧‧ feature vector

c _t-1 , c _t , c _t+1 ‧‧‧ memory features

f _t-1 , f _t , f _t+1 ‧‧‧ space state characteristics

h _t-1 , h _t , h _t+1 ‧‧‧ time feature vector

200‧‧‧Analysis method of car lamp status

S210 to S250‧‧‧ steps

In order to make the above and other objects, features, advantages and embodiments of the present invention more obvious and understandable, the drawings are described as follows: FIG. 1 is a vehicle lamp status analysis system according to some embodiments of the case Fig. 2 is a table showing a combination of lamp states according to some embodiments of the case; Fig. 3 is a flowchart of a method for analyzing a lamp state according to some embodiments of the case; Figure 4 is a schematic diagram of a first convolutional neural network operation model according to some embodiments of the present case; and Figure 5 is a schematic diagram of a long-short term memory time feature extraction model according to some embodiments of the present case.

The following disclosure provides many different embodiments or illustrations to implement different features of the present invention. The elements and configurations in the specific illustrations are used to simplify this disclosure in the following discussion. Any illustrations discussed are for illustrative purposes only, and do not limit the scope and meaning of the invention or its illustrations in any way. In addition, the present disclosure may repeatedly refer to numerical symbols and/or letters in different illustrations. These repetitions are for simplicity and explanation, and do not specify the relationship between different embodiments and/or configurations in the following discussion.

Please refer to Figure 1. FIG. 1 is a schematic diagram of a vehicle lamp status analysis system 100 according to some embodiments of the present case. As shown in FIG. 1, the vehicle lamp status analysis system 100 includes a storage device 110 and a processor 130. The processor 130 is electrically connected to the storage device 110. The storage device 110 is used to store the first convolutional neural network operation model DB1, the second convolutional neural network operation model DB2, the long and short-term memory time feature extraction model DB3, and the object tracking model DB4.

In the embodiments of the present invention, the processor 130 may be implemented as an integrated circuit such as a microcontroller, a microprocessor, a digital signal processor, or an application specific integrated circuit integrated circuit (ASIC), logic circuit or other similar elements or a combination of the above elements. The storage device 110 may be implemented as a memory, a hard disk, a flash drive, a memory card, or the like. The detailed operation mode of the vehicle lamp state analysis system 100 will be described below with reference to FIGS. 2 to 5 together.

Please refer to figure 2. Fig. 2 is a table showing a combination of lamp states according to some embodiments of the present case. In some embodiments, the second convolutional neural network operation model DB2 is a model created based on multiple images classified into multiple combinations. For example, as shown in FIG. 2, the lamp shape includes sixteen combinations G1 to G16. Combinations G1 to G16 perform deep learning based on multiple images classified into the group, respectively, to optimize the second convolutional neural network operation model DB2. As shown in FIG. 2, the combination of lamp states G1 to G16 is for illustrative purposes only, and the implementation of the present case is not limited to this.

Please refer to Figure 3. Figure 3 is based on some embodiments of the case A flow chart of a method 300 for analyzing a lamp state is shown. In some embodiments, the vehicle lamp state analysis method 300 includes steps S210 to S250.

In step S210, multiple images are obtained, wherein the multiple images are continuous in time. In some embodiments, step S210 can be obtained by input and output devices (not shown) such as image capture devices and cameras. At this time, n=0. n is a variable used to record the number of images from steps S215 to S245. In some embodiments, the vehicle lamp state analysis method 300 determines which of the combinations G1 to G16 the vehicle lamp state belongs to based on M images. For example, M can be 150, that is, the lamp state analysis method 300 determines which of the combinations G1 to G16 the lamp state belongs to based on 150 images, but this case is not limited to this.

In step S215, the object is obtained from the image. In some embodiments, step S215 may be executed by the processor 130 in FIG. 1. For example, the processor 130 executes an object recognition module (not shown) stored in the storage device 110 to obtain objects in the image. In some embodiments, if the processor 130 can obtain objects from the image, the processor 130 executes step S220. If the processor cannot obtain the object from the image, the processor continues to execute step S215 until the object can be obtained from the image. In detail, if the processor 130 cannot obtain the object from the first image, the processor continues to perform step S215 and determines whether the object can be obtained from the second image. If the processor 130 can obtain the object from the first image, the processor then proceeds to step S220.

In step S220, it is determined whether the object is a detected object. In some embodiments, step S220 may be executed by the processor 130 in FIG. 1. If it is determined that the object is not a detected object, step S222 is executed. if It is determined that the object is a detected object, and step S224 is executed. In some embodiments, the storage device 110 stores variables (not shown) to record whether the object is a detected object, so that the processor 130 can determine whether the object is a detected object. For example, when the variable is true (True), it is determined that the object has been detected. When the variable is False, it is determined that it is not a detected object.

In step S222, the region of interest is obtained according to the image and the second convolutional neural network operation model. In some embodiments, step S222 may be executed by the processor 130 in FIG. 1. The second convolutional neural network operation model DB2 can be YOLOv3, SSD, MobileNetv2, etc., but this case is not limited to the above.

In step S224, the region of interest is obtained based on the image and object tracking model. In some embodiments, step S224 may be executed by the processor 130 in FIG. 1. The object tracking model DB4 can be MDNet, TLD and other models, but this case is not limited to the above.

Since the computational cost of the object tracking model DB4 is lower than the computational cost of the second convolutional neural network operation model DB2, by determining whether it is a detected object and selecting the object tracking model DB4 or the second according to the result of the determination The convolutional neural network operation model DB2 obtains the region of interest of the image to reduce the overall calculation cost.

In step S230, the image feature vector is obtained according to the first convolutional neural network operation model and the region of interest. In some embodiments, step S230 may be executed by the processor 130 in FIG. 1.

Please also refer to Figure 4. FIG. 4 is a schematic diagram of the first convolutional neural network operation model DB1 according to some embodiments of the present case. Such as As shown in FIG. 4, the first convolutional neural network operation model DB1 includes a ten-layer sub-operational model, which is a lightweight convolutional neural network operation model. In detail, the ten-layer sub-operation model of the convolutional neural network operation model DB1 includes the convolution layer C1, C2, the pooling layer P1, the convolution layer C3, the pooling layer P2, the convolution layer C4, the pooling layer P3, Convolutional layer C5, pooling layer P4 and fully connected network layer FC1. The numbers described on each sub-operation model represent the dimensions of each sub-operation model.

In some embodiments, after the image V passes through the ten-layer sub-operation model of the convolutional neural network operation model DB1, a feature vector with a length of 4096 is generated at the fully connected network layer FC1 to represent the state of the image V space energy .

Please refer back to Figure 3. In step S240, a time feature vector is generated according to the feature vector and the long and short-term memory time feature extraction model, and n is incremented by 1. In some embodiments, step S240 may be executed by the processor 130 in FIG. 1.

Please also refer to Figure 5. FIG. 5 is a schematic diagram of a long-short-term memory time feature extraction model DB3 according to some embodiments of the present case. x _t-1 , x _t and x _t+1 are the feature vectors generated by the convolutional neural network operation model DB1. c _t-1 , c _t and c _t+1 are the memory features generated by each image based on the long and short-term memory time feature extraction model DB3. f _t-1 , f _t , and f _t+1 are the spatial state features generated by the long and short-term memory time feature extraction model DB3 based on each image. h _t-1 , h _t , and h _t+1 are long-term and short-term memory time feature extraction models DB3 based on the feature vector of each image, the spatial state characteristics of the previous image of each image, and the memory of the previous image of each image Time feature vector generated by feature.

Please refer back to Figure 3. In step S245, it is determined whether n=M. In some embodiments, step S245 may be executed by the processor 130 in FIG. 1. When n is equal to M, it indicates that there are M images that have performed steps S215 to S245, and step S250 is performed. When n is not equal to M, it means that there are no M images that have performed steps S215 to S245, and step S215 is performed to obtain the object of the next image.

In step S250, a combination of vehicle lights corresponding to a plurality of images is determined. In some embodiments, step S250 may be executed by the processor 130 in FIG. 1.

Please also refer to Figure 5. After all the M images have executed steps S215 to S245, the long- and short-term memory time feature extraction model DB3 generates an output feature vector of length 512, and the long- and short-term memory time feature extraction model DB3 generates a full-length network layer FC2 of length 16. Probability vector.

In some embodiments, in the long- and short-term memory time feature extraction model DB3, the probability vector is normalized to generate a probability corresponding to multiple combinations G1 vs. G16 as shown in FIG. 2. Next, in the long and short-term memory time feature extraction model DB3, it is determined that the most probable among the multiple combinations G1 to G16 is the combination corresponding to multiple images.

According to the aforementioned embodiments, some other embodiments of the present disclosure provide a non-transitory computer readable medium. The non-transitory computer can read the media and store the computer software and use it to perform the aforementioned method 200 for analyzing the state of the vehicle lamp as shown in FIG.

As mentioned above, since the state of the lamp is dynamic, for example The flickering state, it is not possible to determine the state of the lamp from a single static image. In the embodiment of the present invention, a convolutional neural network operation model and a long-term and short-term memory time feature extraction model are used to determine the state of the vehicle light based on the spatial energy features in multiple temporally continuous images. Through the neural network learning and the judgment of the long-term and short-term memory time feature extraction model, compared with the prior art, the embodiment of this case does not need to set multiple parameter thresholds, and can be adapted to various weather conditions.

In addition, the above example includes exemplary steps in order, but the steps need not be performed in the order shown. Performing these steps in different orders is within the scope of this disclosure. Within the spirit and scope of the embodiments of the present disclosure, the order may be added, replaced, changed, and/or omitted as appropriate.

Although this case has been disclosed as above by way of implementation, it is not intended to limit this case. Anyone who is familiar with this skill can make various changes and modifications within the spirit and scope of this case, so the scope of protection of this case should be considered The scope of the attached patent application shall prevail.

200‧‧‧Analysis method of car lamp status

S210 to S250‧‧‧ steps

Claims (7)

A vehicle lamp state analysis method includes: acquiring a plurality of regions of interest in a plurality of images, wherein the images are continuous in time; obtaining a plurality of the regions of interest according to a first convolutional neural network operation model Feature vectors; and a feature extraction model based on the feature vectors and a long-short-term memory time feature to determine a combination of vehicle lights corresponding to the images, wherein the regions of interest to obtain the images further include: Obtain an object from a first image in the image; determine whether the object is a detected object; if it is determined that the object is the detected object, based on the first image and a second convolutional neural network operation model to Obtaining a first region of interest among the regions of interest; if it is determined that the object is not the detected object, the first sense in the regions of interest is obtained based on the first image and an object tracking model Area of interest. The vehicle lamp state analysis method according to claim 1, wherein the first convolutional neural network operation model includes a ten-layer sub-operation model. The vehicle lamp state analysis method according to claim 1, wherein determining the vehicle lamp combination corresponding to the images according to the feature vectors and the long-short-term memory time feature extraction model includes: A probability vector is generated based on the feature vectors and the long-short-term memory time feature extraction model; and a vehicle light combination corresponding to the images is determined based on the probability vector. A lamp state analysis system includes: a storage device for storing a first convolutional neural network operation model, a second convolutional neural network operation model, and a long-short-term memory time feature extraction model; a processor , Electrically connected to the storage device, the processor is used to obtain a plurality of regions of interest of a plurality of images, according to the first convolutional neural network operation model to obtain a plurality of feature vectors of the regions of interest, and Based on the feature vectors and the long-short-term memory time feature extraction model to determine a vehicle light combination corresponding to the images, the processor is further used to obtain an object from a first image in the images, and Determine whether the object is a detected object, and if the object is determined to be the detected object, the processor is further used to obtain the interest based on the first image and a second convolutional neural network operation model A first region of interest in the region, if it is determined that the object is not the detected object, the processor is further used to obtain the first of the regions of interest based on the first image and an object tracking model Area of interest. The vehicle lamp state analysis system according to claim 4, wherein the first convolutional neural network operation model includes a ten-layer sub-operation model. The vehicle lamp state analysis system according to claim 4, wherein the processor is further used to generate a probability vector based on the feature vectors and the long-short-term memory time feature extraction model, and determine and determine the probability based on the probability vector The combination of the lights corresponding to the image. A non-transitory computer readable medium, including at least one instruction program, a processor executes the at least one instruction program to implement a vehicle lamp status analysis method, the vehicle lamp status analysis method includes: acquiring a plurality of a plurality of images Regions of interest; based on a first convolutional neural network operation model to obtain a plurality of feature vectors of the regions of interest; and based on the feature vectors and a long-short-term memory time feature extraction model to determine and the images A corresponding lamp combination, wherein the regions of interest to obtain the images further include: obtaining an object from a first image in the images; determining whether the object is a detected object; if determining the The object is the detected object, based on the first image and a second convolutional neural network operation model to obtain a first region of interest among the regions of interest; if it is determined that the object is not the detected object The object obtains the first region of interest among the regions of interest according to the first image and an object tracking model.

Images