CN111723602A - Driver's behavior recognition method, device, equipment and storage medium - Google Patents

Abstract

The present application discloses a method, device, device and storage medium for identifying a driver's behavior, belonging to the technical field of intelligent transportation. The method includes: acquiring a target image, acquiring a first area image from the target image, where the first area image includes the driver's preset behavior, and performing a measurement on the surrounding area of the first area image according to a first proportional threshold in the target image. Reducing processing to obtain a second area image, and performing expansion processing on the surrounding area of the first area image according to the second proportional threshold to obtain a third area image, based on the first area image, the second area image and the third area image, identify out of the driver's behavior. After the first area image is cut in and expanded in the present application, the obtained second area image and the third area image can include more information about the surrounding objects related to the preset behavior. Therefore, based on the three areas The image can identify the driver's behavior, which can improve the accuracy of the recognition.

Description

Driver's behavior recognition method, device, equipment and storage medium

technical field

The present application relates to the technical field of intelligent transportation, and in particular, to a method, device, device, and storage medium for identifying a driver's behavior.

Background technique

With the rapid development of intelligent transportation technology, people pay more and more attention to how to solve the driving safety problem caused by driver distraction through intelligent means. In some scenarios, the driver's face can be photographed, and the driver's behavior can be analyzed based on the photographed images. When it is determined that there is a violation, an alarm will be issued to remind the driver to drive safely.

At present, the network model after deep learning can be used to detect the driver's behavior. That is to say, the network model to be trained can be deeply trained based on some training samples in advance, and the training samples can include the image samples and the location information and behavior categories of the regions where the behaviors in the image samples are located, so that the trained network model can be based on the image samples. The image recognizes some gestures such as making a phone call, smoking a cigarette, etc.

However, since the driver may have gestures similar to the illegal behavior during driving, for example, touching the ear, touching the chin, covering the mouth, etc., in this case, the driver's behavior recognition method provided above may lead to misjudgment.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a method, device, device, and storage medium for a driver's behavior identification, which can solve the problem that the driver's behavior identification method in the related art may lead to misjudgment. The technical solution is as follows:

In a first aspect, a method for identifying a driver's behavior is provided, the method comprising:

acquiring a target image, the target image including the driver's face;

acquiring a first area image from the target image, where the first area image includes a preset behavior of the driver, and the preset behavior refers to a behavior whose similarity to the illegal behavior is greater than a preset threshold;

In the target image, reducing the surrounding area of the first area image according to the first proportional threshold to obtain the second area image, and expanding the surrounding area of the first area image according to the second proportional threshold processing to obtain a third region image;

Based on the first area image, the second area image and the third area image, the behavior of the driver is identified.

Optionally, before recognizing the behavior of the driver based on the first area image, the second area image and the third area image, the method further includes:

Adjusting the first area image, the second area image and the third area image into area images of the same size;

Correspondingly, the detecting the driver's behavior based on the first area image, the second area image and the third area image includes:

Calling a target network model, the target network model is used to determine the behavior category of the behavior based on a group of regional images corresponding to any behavior;

Based on the resized first area image, second area image and third area image, the behavior of the driver is identified through the target network model.

Optionally, the target network model includes an input layer, an intermediate layer, a splicing layer, a fully connected layer and an output layer;

The behavior of the driver is identified through the target network model based on the size-adjusted first area image, second area image and third area image, including:

Based on the resolution and the number of channels of the resized area image, the input layer performs channel superposition processing on the image data in the resized first area image, second area image and third area image, to obtain the first feature map;

Performing convolution sampling processing on the first feature map by the intermediate layer to obtain a plurality of second feature maps, the plurality of second feature maps having the same size and different channel numbers;

Perform channel stacking processing on the plurality of second feature maps through the splicing layer, and perform feature fusion on the feature maps after channel stacking through the convolution layer in the deep layer of the network to obtain a third feature map;

determining the driver's behavior based on the third feature map through the fully connected layer;

The behavior is output through the output layer.

Optionally, the intermediate layer includes N groups of convolution layers and N groups of sampling layers, and each group of convolution layers corresponds to each group of sampling layers one-to-one;

Performing convolution sampling processing on the first feature map through the intermediate layer to obtain a plurality of second feature maps, including:

Let i=1, determine the first feature map as the target feature map; perform convolution processing on the target feature map through the i-th group of convolution layers, and obtain the i-th group of sampling layers according to two different multiple pairs. The feature map is sampled to obtain a ²ⁱ -fold feature map and a second feature map of the i-th reference size, and the ²ⁱ -fold feature map is obtained as the target feature map, and the reference size is greater than or equal to the 2 ⁱ times the size of the feature map;

When i is less than the N, let i=i+1, return the convolution processing of the target feature map through the i-th group of convolution layers, and obtain through the i-th group of sampling layers according to two different multiple pairs. The feature map is sampled to obtain 2 ⁱ times the feature map and the second feature map of the i th reference size, and the 2 ⁱ times feature map is obtained as the operation of the target feature map;

When i is equal to the N, the target feature map is subjected to convolution processing through the Nth group of convolution layers, and the Nth group of sampling layers is used to perform a sampling process on the obtained feature map to obtain the Nth reference size. The second feature map, end the operation.

Optionally, before invoking the target network model, the method further includes:

Acquiring multiple training samples, the multiple training samples include multiple sets of images and behavior categories in each set of images, and each set of images includes an area image of the behavior and a reduced area determined after reducing the area image an image, and an expanded area image determined after the area image is expanded;

The target network model is obtained after training the network model to be trained based on the plurality of training samples.

Optionally, the obtaining the first region image from the target image includes:

Invoke a target detection model, input the target image into the target detection model, and output a face detection frame and a preset behavior detection frame, and the target detection model is used to detect the face in the image based on any image. and pre-determined behavior;

When the number of the face detection frames is one, the face detection frame is determined as the target face detection frame; when the number of the face detection frames is multiple, it is obtained from multiple face detection frames The face detection frame with the largest area, and the acquired face detection frame is determined as the target face detection frame;

The first region image is determined based on the target face detection frame.

Optionally, the determining the first region image based on the target face detection frame includes:

Filter out the preset behavior detection frame that does not overlap with the target face detection frame in the preset behavior detection frame, or whose distance from the target face detection frame is greater than a preset distance threshold;

The area corresponding to the preset behavior detection frame remaining after filtering is cut out from the target image to obtain the first area image.

Optionally, the acquiring the target image includes:

detecting the working mode of the camera for photographing the driver's face;

When the working mode of the camera is an infrared shooting mode, acquiring a grayscale image obtained by shooting;

Invoke the image pseudo-color conversion model, input the grayscale image into the image pseudo-color conversion model, and output a three-channel color image corresponding to the gray-scale image, and the image pseudo-color conversion model is used to convert any grayscale image. converting the grayscale image into a three-channel color image corresponding to the grayscale image;

The output three-channel color image is acquired as the target image.

Optionally, before the detection of the working mode of the camera for photographing the driver's face, the method further includes:

Get the current speed and light intensity;

When the vehicle speed is greater than the preset vehicle speed threshold and the light intensity is lower than the light intensity threshold, the working mode of the camera is switched to the infrared shooting mode.

Optionally, after the driver's behavior is detected based on the first area image, the second area image and the third area image, the method further includes:

When the driver's behavior is a violation, count the number of violations of the driver within a preset period of time;

When the number of illegal behaviors of the driver reaches a preset number of times threshold within the preset time period, an illegal driving alarm prompt is issued.

In a second aspect, a driver's behavior recognition device is provided, the device comprising:

a first acquisition module, for acquiring a target image, the target image including the driver's face;

The second acquiring module is configured to acquire a first area image from the target image, where the first area image includes a preset behavior of the driver, and the preset behavior refers to the similarity with the illegal behavior Behavior greater than a preset threshold;

An image processing module, configured to perform reduction processing on the surrounding area of the first area image in the target image according to a first proportional threshold to obtain a second area image, and perform a reduction process on the first area according to the second proportional threshold The surrounding area of the image is expanded to obtain a third area image;

An identification module, configured to identify the driver's behavior based on the first area image, the second area image and the third area image.

Optionally, the device further includes:

a size adjustment module, configured to adjust the first area image, the second area image and the third area image to area images of the same size;

The identification module is used to call a target network model, and the target network model is used to determine the behavior category of the behavior based on a group of regional images corresponding to any behavior;

Based on the resized first area image, second area image and third area image, the behavior of the driver is identified through the target network model.

Optionally, the identification module is used for:

The target network model includes an input layer, an intermediate layer, a splicing layer, a fully connected layer and an output layer; through the input layer, based on the resolution of the resized regional image and the number of channels, the resized first The image data in the area image, the second area image and the third area image are subjected to channel superposition processing to obtain a first feature map;

Performing convolution sampling processing on the first feature map by the intermediate layer to obtain a plurality of second feature maps, the plurality of second feature maps having the same size and different channel numbers;

Perform channel stacking processing on the plurality of second feature maps through the splicing layer, and perform feature fusion on the feature maps after channel stacking through the convolution layer in the deep layer of the network to obtain a third feature map;

determining the driver's behavior based on the third feature map through the fully connected layer;

The behavior is output through the output layer.

Optionally, the identification module is used for:

The middle layer includes N groups of convolution layers and N groups of sampling layers, each group of convolution layers corresponds to each group of sampling layers one-to-one; let i=1, determine the first feature map as the target feature map; The i group of convolution layers performs convolution processing on the target feature map, and the i-th group of sampling layers respectively sample the obtained feature maps according to two different multiples to obtain 2 ⁱ times the feature map and the i-th reference size. the second feature map, obtaining the ²ⁱ -fold feature map as the target feature map, and the reference size is greater than or equal to the size of the ²ⁱ -fold feature map;

When i is less than the N, let i=i+1, return the convolution processing of the target feature map through the i-th group of convolution layers, and obtain through the i-th group of sampling layers according to two different multiple pairs. The feature map is sampled to obtain 2 ⁱ times the feature map and the second feature map of the i th reference size, and the 2 ⁱ times feature map is obtained as the operation of the target feature map;

When i is equal to the N, the target feature map is subjected to convolution processing through the Nth group of convolution layers, and the Nth group of sampling layers is used to perform a sampling process on the obtained feature map to obtain the Nth reference size. The second feature map, end the operation.

Optionally, the device further includes a training module, the training module is used for:

Acquiring multiple training samples, the multiple training samples include multiple sets of images and behavior categories in each set of images, and each set of images includes an area image of the behavior and a reduced area determined after reducing the area image an image, and an expanded area image determined after the area image is expanded;

The target network model is obtained after training the network model to be trained based on the plurality of training samples.

Optionally, the second obtaining module is used for:

Invoke a target detection model, input the target image into the target detection model, and output a face detection frame and a preset behavior detection frame, and the target detection model is used to detect the face in the image based on any image. and pre-determined behavior;

When the number of the face detection frames is one, the face detection frame is determined as the target face detection frame; when the number of the face detection frames is multiple, it is obtained from multiple face detection frames The face detection frame with the largest area, and the acquired face detection frame is determined as the target face detection frame;

The first region image is determined based on the target face detection frame.

Optionally, the second obtaining module is used for:

Filter out the preset behavior detection frame that does not overlap with the target face detection frame in the preset behavior detection frame, or whose distance from the target face detection frame is greater than a preset distance threshold;

The area corresponding to the preset behavior detection frame remaining after filtering is cut out from the target image to obtain the first area image.

Optionally, the first acquisition module is used for:

detecting the working mode of the camera for photographing the driver's face;

When the working mode of the camera is an infrared shooting mode, acquiring a grayscale image obtained by shooting;

Invoke the image pseudo-color conversion model, input the grayscale image into the image pseudo-color conversion model, and output a three-channel color image corresponding to the gray-scale image, and the image pseudo-color conversion model is used to convert any grayscale image. converting the grayscale image into a three-channel color image corresponding to the grayscale image;

The output three-channel color image is acquired as the target image.

Optionally, the device further includes:

The third acquisition module is used to acquire the current vehicle speed and light intensity;

A switching module, configured to switch the working mode of the camera to the infrared shooting mode when the vehicle speed is greater than a preset vehicle speed threshold and the illumination intensity is lower than the illumination intensity threshold.

Optionally, the device further includes:

A statistics module, configured to count the number of violations of the driver within a preset time period when the driver's behavior is a violation;

An alarm module, configured to issue an alarm for illegal driving when the number of illegal behaviors of the driver reaches a preset number of thresholds within the preset time period.

A third aspect, a smart device, comprising:

processor;

memory for storing processor-executable instructions;

Wherein, the processor is configured to implement the method for recognizing the behavior of the driver described in the first aspect above.

In a fourth aspect, a computer-readable storage medium is provided, and instructions are stored on the computer-readable storage medium, and when the instructions are executed by a processor, the method for recognizing the driver's behavior described in the first aspect above is implemented.

In a fifth aspect, there is provided a computer program product containing instructions, which, when executed on a computer, cause the computer to execute the method for recognizing the driver's behavior as described in the first aspect above.

The beneficial effects brought by the technical solutions provided in the embodiments of the present application are:

A target image including the driver's face is obtained, and a first area image including the driver's preset behavior is obtained from the target image, and the preset behavior is similar to the violation behavior, that is, the target image is obtained from the The first area image of the preset behavior with similar violation behavior. Afterwards, in the target image, the first area image is cut in and out at different scales to obtain the second area image and the third area image. Since the first area image may only include the preset behavior, but does not include or include a very small part of the surrounding objects related to the preset behavior, it is impossible to accurately determine whether the preset behavior is truly a violation based on the first area image. Therefore, After the first area image is cut in and expanded in the present application, the obtained second area image and the third area image can include more information about the surrounding objects related to the preset behavior. Therefore, based on the three areas The image detects the driver's behavior, which can improve the accuracy of the detection.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a schematic diagram of a captured image of a driver's face according to an exemplary embodiment;

2 is a flowchart of a method for recognizing a driver's behavior according to an exemplary embodiment;

3 is a schematic diagram of a driver's face detection frame according to an exemplary embodiment;

FIG. 4 is a schematic diagram of a region image according to an exemplary embodiment;

FIG. 5 is a schematic diagram of a convolution sampling processing flow of an intermediate layer according to an exemplary embodiment;

6 is a schematic structural diagram of a driver's behavior recognition device according to an exemplary embodiment;

7 is a schematic structural diagram of a device for recognizing a driver's behavior according to another exemplary embodiment;

8 is a schematic structural diagram of a device for recognizing driver behavior according to another exemplary embodiment;

9 is a schematic structural diagram of a driver's behavior recognition device according to another exemplary embodiment;

10 is a schematic structural diagram of a device for recognizing driver behavior according to another exemplary embodiment;

FIG. 11 is a schematic structural diagram of a terminal 1000 according to an exemplary embodiment.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

Before the detailed introduction of the driver's behavior identification method provided by the embodiments of the present application, the application scenarios and implementation environments involved in the embodiments of the present application are briefly introduced.

First, the application scenarios involved in the embodiments of the present application are briefly introduced.

Currently, image recognition is widely used in the field of intelligent transportation, and in some embodiments, can be used to detect driver behavior. During the driving process of the driver, the driver's face part can be photographed, and then the photographed image can be detected and analyzed by the trained network model to detect whether the driver has violated the rules. However, some of the driver's behavior may be similar to the violation. For example, please refer to Figure 1. The driver's face touching gesture is similar to the phone call gesture. In this case, when the trained network model is used for detection and analysis, It is easy to misjudge the act of touching the face as a violation, resulting in false alarms, which will not only affect the passengers, but may also affect the driver's distraction. To this end, an embodiment of the present application provides a method for identifying a driver's behavior, which can accurately determine whether the driver's behavior is a violation. For the specific implementation, please refer to the embodiment shown in FIG. 2 below.

Next, the implementation environment involved in the embodiments of the present application is briefly introduced.

The method for recognizing the driver's behavior provided in the embodiment of the present application may be performed by a smart device, and the smart device may be configured with or connected with a camera, and the camera may be installed in an area such as a center console, a dashboard, or an A-pillar in front of the vehicle to pass the The camera captures part of the driver's face to obtain clear images around the driver in real time. In some embodiments, the smart device may be a terminal such as a mobile phone, a tablet computer, a computer device, or the like, or the smart device may also be a smart camera device, etc., which is not limited in this embodiment of the present application.

After introducing the application scenarios and implementation environments involved in the embodiments of the present application, the method for recognizing the driver's behavior provided by the embodiments of the present application will be described in detail next with reference to the accompanying drawings.

Please refer to FIG. 2. FIG. 2 is a flowchart of a method for recognizing a driver's behavior according to an exemplary embodiment. This embodiment of the present application is described by taking the method executed by the above-mentioned smart device as an example. The behavior recognition method can include the following implementation steps:

Step 201: Acquire a target image, where the target image includes the driver's face.

In a possible implementation manner, the smart device may acquire a video image captured by a camera at intervals of a time threshold to obtain the target image. Alternatively, it can be understood that, during the process of video shooting by the camera, the smart device may acquire a frame of video image every preset number of video image frames to obtain the target image.

The above-mentioned duration threshold may be set by the user according to actual needs, or may be set by default by the smart device, which is not limited in this embodiment of the present application.

In addition, the preset number may be set by the user according to actual needs, or may be set by default by the smart device, which is not limited in this embodiment of the present application. For example, the preset number may be 5, and the target image may be the first video image frame, the sixth video image frame, the eleventh video image frame, and the like.

Further, the target image may be a three-channel color image, and the three channels refer to three channels of R, G and B, respectively representing the red value, the green value and the blue value of the pixel point. In the implementation, the smart device detects the working mode of the camera used for photographing the driver's face, and when the working mode of the camera is the infrared photographing mode, obtains the grayscale image obtained by shooting, and invokes the image pseudo-color conversion model , input the grayscale image into the image pseudo-color conversion model, and output the three-channel color image corresponding to the grayscale image. The image pseudocolor conversion model is used to convert any grayscale image into the corresponding grayscale image. A three-channel color image, and the output three-channel color image is acquired as the target image.

That is to say, the camera can automatically switch the working mode according to the light intensity of day and night, and the working mode can include an infrared shooting mode and a non-infrared shooting mode. In the non-infrared shooting mode, the image captured by the camera is a three-channel color image. At this time, the image captured by the camera can be directly obtained as the target image. However, in the infrared shooting mode, the image captured by the camera is a grayscale image. Since the grayscale image loses color information, it may increase the difficulty of subsequent processing for smart devices. To this end, when it is detected that the camera is in the infrared shooting mode, the grayscale image can be converted into a three-channel color image, that is, the false color information of the grayscale image can be restored, and the three-channel color image obtained after conversion can be obtained as the target. image.

In some embodiments, a grayscale image can be converted to a corresponding three-channel color image by an image pseudocolor conversion model. That is, the grayscale image can be input into a pseudo-color conversion model that outputs a three-channel color image of the same size as the grayscale image.

Further, before calling the image pseudo-color conversion model, a plurality of grayscale image samples and color image samples corresponding to each grayscale image sample can also be obtained, and based on the corresponding grayscale image samples and each grayscale image sample The color image sample of , the image pseudo-color conversion model is obtained after training the network to be trained.

For example, it is possible to obtain more than 500,000 color image samples taken from natural scenes, and convert them into corresponding grayscale image samples through the following formula (1). The image samples are input into the network to be trained for training. Wherein, the formula (1) is:

H=R*0.299+G*0.587+B*0.114 (1)

Among them, H represents the gray value of the pixel point, R represents the red value of the pixel point, G represents the green value of the pixel point, and B represents the blue value of the pixel point.

In a possible implementation manner, the network to be trained can use, but is not limited to, a conditional generative adversarial network. At this time, during the training process, the network includes a generator and a discriminator. The image samples are superimposed with parameters to generate a color image of the same size. The discriminator generates the training loss by judging the difference between the generated color image and the color image sample corresponding to the grayscale image sample, thereby iteratively training the network.

Further, before detecting the working mode of the camera for photographing the driver's face, the smart device obtains the current vehicle speed and light intensity, when the vehicle speed is greater than the preset speed threshold and the light intensity is lower than the light intensity threshold. , switch the working mode of the camera to the infrared shooting mode.

That is to say, the camera can automatically switch to the infrared shooting mode only when the vehicle speed is greater than a certain preset vehicle speed threshold and the light intensity is low; Do video shooting.

The preset vehicle speed threshold may be set by the user according to actual needs, or may be set by default by the smart device, which is not limited in this embodiment of the present application.

In addition, the light intensity threshold may be set by the user according to actual needs, or may be set by default by the smart device, which is not limited in this embodiment of the present application.

Further, the above description is only described by taking the light intensity as one of the conditions to control the camera to automatically switch to the infrared shooting mode as an example. In another embodiment, in order to remind the driver not to drive fatigued, there may be a need to detect whether the driver closes his eyes. In this case, if the driver wears sunglasses, the camera also needs to be switched to the infrared shooting mode. At this time, the user can manually switch to trigger the infrared shooting switching instruction, so that the smart device switches the camera to the infrared shooting mode based on the infrared shooting switching instruction.

It should be noted that the above description only takes the target image as a three-channel color image as an example for description. In another embodiment, the target image may also be a grayscale image, which is not limited in this embodiment of the present application.

It should also be noted that the above description is only based on the case where the vehicle speed is greater than the preset vehicle speed threshold and the light intensity is low, the camera automatically switches to the infrared shooting mode, otherwise, the non-infrared shooting mode is used as an example for description. In another embodiment, when the vehicle speed is lower than the preset threshold, the camera may not be turned on, that is, if the vehicle speed is lower than the preset threshold, the behavior detection of the driver may not be performed. Therefore, before acquiring the target image, you can first determine whether the vehicle speed is greater than the preset vehicle speed threshold, and when the vehicle speed is greater than the preset vehicle speed threshold, the operation of acquiring the target image is performed, otherwise, the camera is not activated, that is, the acquisition of the target image is not performed. operation.

Step 202: Acquire a first area image from the target image, where the first area image includes a preset behavior of the driver, where the preset behavior refers to behaviors whose similarity to the illegal behavior is greater than a preset threshold.

The preset behavior may be set by the user according to actual needs, or may be set by default by the smart device, which is not limited in this embodiment of the present application.

For example, the preset behavior may include making a phone call, smoking a cigarette, touching the chin, covering the mouth, touching the side of the face, touching the ear, and the like. Wherein, the making a phone call here refers to the posture of holding the mobile phone close to the ear.

In addition, the above-mentioned preset threshold may be set by the user according to actual needs, or may be set by default by the computer device, which is not limited in this embodiment of the present application.

That is to say, the smart device determines the area where the driver's preset behavior similar to the illegal behavior is located from the target image, and then acquires the determined area from the target image, so that the preset behavior can be further processed in the future Detailed analysis to determine whether the preset behavior is truly a violation.

In some embodiments, the specific implementation of acquiring the first region image from the target image may include: calling a target detection model, inputting the target image into the target detection model, and outputting a face detection frame and a preset behavior detection frame , the target detection model is used to identify faces and preset behaviors in any image based on the image. When the number of the face detection frame is one, the face detection frame is determined as the target face detection frame; when the number of the face detection frame is multiple, the largest area of the face detection frame is obtained from the multiple face detection frames. The face detection frame, the acquired face detection frame is determined as the target face detection frame, and the first region image is determined based on the target face detection frame.

That is to say, the target detection model can not only detect faces, but also detect areas corresponding to preset behaviors, such as areas corresponding to behaviors such as calling, touching your face, and smoking. Usually, the number of detected preset behavior detection frames is multiple, and the preset behavior corresponding to one of the preset behavior detection frames may not belong to the driver. For passengers, smart devices can filter out pre-set behavior detection boxes that are irrelevant to the driver. In implementation, the driver's face detection frame may be determined, and then a filtering operation is performed based on the driver's face detection frame.

Since the camera shoots at the driver's face, when the number of output face detection frames is one, it can be determined that the face detection frame corresponds to the driver's face. The face detection frame is determined as the target detection frame. In other embodiments, the passenger may be standing near the driver, for example, standing behind the driver's side and facing the camera, at which point the object detection model may detect multiple face detection frames. Since the camera shoots the driver's face, it can be determined that the face detection frame with the largest area corresponds to the driver's face. For example, as shown in Figure 3, the face detection frame with the largest area is Determined as the target face detection frame.

Further, based on the target face detection frame, the specific implementation of determining the first area image may include: the preset behavior detection frame does not overlap with the target face detection frame, or overlaps with the target face detection frame. The preset behavior detection frames whose distances are greater than the preset distance threshold are filtered out, and the area corresponding to the remaining preset behavior detection frames after filtering is cut out from the target image to obtain the first area image.

The smart device can filter the preset behavior detection frame that does not belong to the driver's preset behavior based on the target face detection frame. It is not difficult to understand that if the preset behavior detection frame does not overlap with the target face detection frame, it means that the preset behavior detection frame is far away from the driver's face, so the preset behavior corresponding to the preset behavior detection frame can be determined. Not the driver's behavior. In addition, it can also be determined according to the distance between the preset behavior detection frame and the target face detection frame. When the distance between the preset behavior detection frame and the target face detection frame is greater than the preset distance threshold It is assumed that the behavior detection frame is far away from the driver's face, so that it can be determined that the preset behavior corresponding to the preset behavior detection frame is not the behavior of the driver.

The preset distance threshold may be set by the user according to actual needs, or may be set by default by the smart device, which is not limited in this embodiment of the present application.

After the smart device filters the preset behavior detection frames, it can determine that the remaining preset behavior detection frames are the detection frames corresponding to the driver's preset behavior, and the smart device cuts out the corresponding area from the target image to obtain the first area. image.

It should be noted that, if the currently output preset behavior detection frame does not overlap with the target face detection frame, or if the distance from the target face detection frame is greater than the preset distance threshold, it can be determined that the target image The driver currently has no violations.

Further, when it is detected through the target detection model that the driver is wearing sunglasses, the camera can be automatically controlled to switch to the infrared shooting mode. That is to say, if the target detection model detects that the driver is wearing sunglasses during the day, in order to detect whether the driver has closed eyes, the camera can be switched to infrared shooting mode.

Further, before calling the target detection model, multiple image samples and the face frame and preset behavior frame in each image sample can be obtained, and the multiple image samples and the face frame and preset behavior frame in each image sample can be obtained. Assume that the behavior box is input to the detection model to be trained for training, and the target detection model is obtained.

The face frame and the preset behavior frame in each image sample may be pre-calibrated, that is, based on multiple image samples and the calibrated face frame and preset behavior frame in the multiple image samples, Deep learning and training are performed on the detection model to be trained, so that the obtained target detection model can automatically detect faces and preset behaviors in any image.

In a possible implementation manner, the detection model to be trained may be a YOLO (You Only LookOnce, you only look once) network, an SSD (Single Shot Detector, a one-time detector), etc. This embodiment of the present application does not cover this. Do limit.

Step 203: in the target image, reducing the surrounding area of the first area image according to the first proportional threshold to obtain a second area image, and expanding the surrounding area of the first area image according to the second proportional threshold processing to obtain a third region image.

Since the first area image may only include the preset behavior, but not or only include a small part of the items related to the preset behavior, for example, the item is a mobile phone, a cigarette, etc., that is to say, the mobile phone or the cigarette may not be completely included in the In the image of the first area, or only a small part of it, objects such as mobile phones or smoke may appear in different sizes due to the installation of the camera and its own size. In order to further determine whether the preset behavior is truly a violation, the smart device performs inward cropping and outward expansion of the area corresponding to the first area image according to different scales, that is, in the target image, according to the first proportional threshold The surrounding area of the area image is reduced, and the surrounding area of the first area image is expanded according to the second proportional threshold to obtain the second area image and the third area image.

For example, please refer to FIG. 4 , assuming that the first scale threshold is 0.8 and the second scale threshold is 1.2, after the first region image is inwardly clipped and outwardly expanded from the target image, 41 and 42 in FIG. 4 can be obtained. The second area image and the third area image are shown, wherein 43 in FIG. 4 is the first area image.

It is worth mentioning that the first area image, the second area image and the third area image are obtained from the target image, and the redundant information in the entire target image is eliminated. In this way, it is different from the behavior detection based on the entire target image. In comparison, the subsequent behavior discrimination processing based on the first area image, the second area image and the third area image can improve the detection accuracy.

Wherein, the first proportional threshold may be set by self-definition according to actual requirements, or may be set by default by the smart device, which is not limited in this embodiment of the present application.

Wherein, the second proportional threshold value may be set according to actual needs, and may also be set by default by the smart device, which is not limited in this embodiment of the present application.

Step 204: Adjust the first area image, the second area image and the third area image into area images of the same size.

In this embodiment of the present application, in order to facilitate the subsequent identification of the driver's behavior based on the first area image, the second area image, and the third area image, the smart device adjusts the size of the three area images, namely Region images at different scales are scaled to the same size.

Step 205: Invoke the target network model, where the target network model is used to determine the behavior category of any behavior based on a group of regional images corresponding to the behavior.

Wherein, a group of area images corresponding to any behavior includes an area image, a reduced area image and an expanded area image of the behavior, and the reduced area image is determined from the captured image where the behavior is located around the area where the behavior is located. The area image of , the expanded area image is an area image determined after expanding around the area where the behavior is located from the captured image where the behavior is located. Further, the area image, the reduced area image and the expanded area image of this behavior are of the same size.

Further, before calling the target network model, multiple training samples can be obtained, the multiple training samples include multiple groups of images and behavior categories in each group of images, and each group of images includes an area image of behavior, an image of the area. The reduced region image determined after the reduction process and the expanded region image determined after the expansion process are performed on the region image, and the target network model is obtained after training the network model to be trained based on the plurality of training samples.

That is to say, the target network model can be obtained by training in advance. During the training process, multiple sets of images and the behavior categories in each set of images can be input into the network model to be trained for training. Wherein, each group of images includes an area image, a reduced area image and an expanded area image of the behavior, that is to say, the area image in each group of images may be obtained by intercepting the area where the behavior is located from the captured image where the behavior is located , the reduced area image can be obtained by reducing the surrounding area of the area where the behavior is located from the photographed image where the behavior is located, and the expanded area image can be obtained from the photographed image where the behavior is located. The surrounding area of the area where the behavior is located is expanded obtained later. Further, the sizes of the three area images in each group of images are the same, that is to say, after the reduction processing and expansion processing, the obtained area images and the area images corresponding to the behavior can be resized, so that the behavior The corresponding three area images are of the same size.

In addition, since the information contained in different channels is different, in order to enable the network model to be trained to learn the weight of each channel for the recognition task, a compression-activation module can be added to the network model to be trained, as the network model to be trained. Each channel of the model is weighted, so that the learning of the network model to be trained pays more attention to the region of interest, and the accuracy of the target network model detection is improved.

Step 206: Identify the driver's behavior through the target network model based on the resized first area image, second area image, and third area image.

Further, the target network model includes an input layer, an intermediate layer, a splicing layer, a fully connected layer and an output layer. The specific implementation of recognizing the behavior of the driver may include: using the input layer based on the resolution and the number of channels of the resized area image, to resize the first area image, the second area image and the third area image. The image data is subjected to channel superposition processing to obtain the first feature map. The first feature map is subjected to convolution sampling processing through the intermediate layer to obtain multiple second feature maps. The multiple second feature maps have the same size and different number of channels. The multiple second feature maps are subjected to channel stacking processing through the splicing layer. , and perform feature fusion on the feature map after channel stacking through the convolution layer in the deep layer of the network to obtain the third feature map. The driver's behavior is determined based on the third feature map through the fully connected layer, and the behavior is output through the output layer.

For example, suppose that the resolution of each area image is 256*256, and the number of channels of each area image is 3 channels, in this case, the resized first area image, second area image and third area image are resized. The three regional images are input into the target network model, and the three regional images are superimposed through the output layer, and a 256*256*9 three-dimensional matrix can be obtained, where 9 is the number of channels after channel stacking, and the A three-dimensional matrix is used to indicate the first feature map.

After that, convolution sampling is performed on the first feature map through an intermediate layer in the target network model. In a possible manner, the intermediate layer includes N groups of convolution layers and N groups of sampling layers, and each group of convolution layers The layers correspond to each group of sampling layers one-to-one, where N is an integer greater than 1. Further, each group of convolutional layers may include at least one convolutional layer, and each group of sampling layers may include at least one sampling layer. In this case, a specific implementation of performing convolution sampling processing on the first feature map by the intermediate layer to obtain multiple second feature maps may include: setting i=1, and determining the first feature map as the target feature map; Perform convolution processing on the target feature map through the ith group of convolution layers, and sample the obtained feature maps according to two different multiples through the ith group of sampling layers to obtain 2 ⁱ times the feature map and the ith reference size. The ^{second feature} map of the The target feature map is subjected to convolution processing through the ith group of convolutional layers, and the obtained feature maps are sampled according to two different multiples through the ith group of sampling layers to obtain 2 ⁱ times the feature map and the ith reference. The second feature map of the size is the operation of obtaining the 2 ⁱ times feature map as the target feature map; when i is equal to the N, the target feature map is convolved through the Nth group of convolution layers, and the Nth The group sampling layer performs a sampling process on the obtained feature map to obtain a second feature map of the Nth reference size, and ends the operation.

The above reference size may be set according to actual requirements, or may be set by default by the smart device type, which is not limited in this embodiment of the present application.

Through the convolution processing of the convolution layer, image features that are conducive to identifying behavior categories can be extracted, and multi-scale feature images can be obtained through the sampling processing of the sampling layer. Please refer to Figure 5. During implementation, the first feature map output by the output layer is input to the first set of convolutional layers for convolution processing to obtain multi-channel feature maps. The sampling layer and the 8*8 sampling layer are subjected to down-sampling processing with different multiples, respectively, to obtain a feature map twice as large and a second feature map of the reference size. Then continue to input the 2x feature map to the second set of convolutional layers for convolution processing to obtain multi-channel feature maps, and then pass through the 2*2 sampling layers and 4*4 sampling layers in the second set of sampling layers. Multiple down-sampling process to obtain 4 times the feature map and the second feature map of the reference size. Then continue to perform convolution sampling processing in the above-mentioned manner, until the Nth group of convolution layers performs convolution processing, and then perform 2*2 sampling layer processing through the Nth group of sampling layers to obtain the second feature map of the Nth reference size, End the convolution sampling operation.

Please continue to refer to Figure 5. After obtaining multiple second feature maps, channel stacking is performed through the splicing layer, and then feature fusion is performed through the convolution layer to obtain the third feature map. That is, before the final prediction result, the one-fold, two-fold, and four-fold feature maps of the shallow layer of the network are downsampled by different multiples to obtain feature maps of the same size, and then superimposed on the channel. A convolutional layer performs feature fusion, and then the feature map after feature fusion is input to the fully connected layer.

It is worth mentioning that the fusion of multi-scale feature maps can enable the target network model to learn the feature distribution of objects at different scales, thereby improving the accuracy of behavior detection.

It should be noted that the size of the convolution kernel of each convolution layer can be set in advance according to actual requirements. In addition, the parameters in the convolution kernel of each convolution layer are determined during the training process. In addition, in the implementation process, the image size, feature map size, number of channels, number of downsampling times, network depth and other parameters of target network models with different behaviors may be different.

After the third feature map is obtained, the behavior of the driver is determined based on the third feature map through the fully connected layer, and then the behavior is output through the output layer, for example, whether the behavior is making a phone call or not making a phone call. Further, the behavior can be identified by the behavior identifier. For example, if the output behavior identifier is "1", it means making a phone call, and if the output behavior identifier is "0", it means not making a phone call. In some embodiments, a confidence level judgment can be given to the output result, and the confidence level interval is [0, 1]. When the confidence level is greater than 0.5, it is determined that the behavior category is a violation, otherwise, it is determined that the behavior does not belong to Irregularities.

It should be noted that the above is only described by taking the target network model including an input layer, an intermediate layer, a splicing layer, a fully connected layer and an output layer as an example. In another embodiment, the target network model also includes other layers, To perform other operations on the feature map through other layers, for example, the target network model may further include a BN (Batch Normalization, batch normalization) layer, etc., which is not limited in this embodiment of the present application.

Further, when the driver's behavior is illegal, count the number of illegal behaviors of the driver within a preset time period, and when the number of violations of the driver within the preset time period reaches the preset number of times threshold, carry out Warning of illegal driving.

When it is determined that the driver's behavior is illegal, count the number of consecutive illegal behaviors of the driver within a certain period of time, that is, count the detection of multiple video image frames in all video image frames within a preset period of time. The behavior of the outbound driver is a violation, and if the number of violations reaches a preset number of thresholds, it can be determined that the driver is engaged in dangerous driving behavior. For example, the detection results of 75% of all video image frames within 3 seconds of statistics indicate that the driver's behavior is illegal. At this time, an illegal driving alarm can be prompted.

In some embodiments, a buzzer can be used to give an alarm for illegal driving, or a voice broadcast can also be used to give an alarm for illegal driving, so as to remind the driver to drive safely, and also allow passengers to supervise the driver's behavior. .

In this way, when it is detected that the number of illegal behaviors of the driver reaches a certain preset number of thresholds, an illegal driving alarm is prompted, which can avoid false alarms caused by deviations in the detection of individual video image frames, and improve the accuracy of the alarms.

The preset duration may be set by the user according to actual needs, or may be set by default by the smart device, which is not limited in this embodiment of the present application.

The preset number of times threshold may be set by the user according to actual needs, or may be set by default by the smart device, which is not limited in this embodiment of the present application.

In the embodiment of the present application, a target image including the driver's face is obtained, and a first area image including the driver's preset behavior is obtained from the target image, and the preset behavior is similar to the illegal behavior, that is, A first area image of a preset behavior similar to the violation behavior is acquired from the target image. Afterwards, in the target image, the first area image is cut in and out of different scales to obtain the second area image and the third area image. Since the first area image may only include a small part of the preset behavior, but does not include or include a very small part of the surrounding objects related to the preset behavior, it is impossible to accurately determine whether the preset behavior is actually a violation based on the first area image. Therefore, after the first area image is cut in and expanded in this application, the obtained second area image and third area image can include more information about the surrounding objects related to the preset behavior. Therefore, based on The three area images are used to detect the driver's behavior, which can improve the detection accuracy.

In addition, the channel fusion of the three regional images can make the target network model pay attention to the information within a certain range around the object, so that the target network model can accurately determine the driver's behavior category and improve the robustness of the target network model.

Fig. 6 is a schematic structural diagram of a device for recognizing driver's behavior according to an exemplary embodiment. The device for recognizing driver's behavior can be implemented by software, hardware, or a combination of the two. The driver's behavior recognition device may include:

The first acquisition module 501 is used to acquire a target image, and the target image includes the driver's face;

The second acquiring module 502 is configured to acquire a first area image from the target image, where the first area image includes a preset behavior of the driver, and the preset behavior refers to the similarity with the illegal behavior Behavior with a degree greater than a preset threshold;

The image processing module 503 is configured to, in the target image, perform reduction processing on the surrounding area of the first area image according to a first proportional threshold to obtain a second area image, and perform a reduction process on the first area image according to the second proportional threshold The surrounding area of the area image is expanded to obtain the third area image;

The identification module 504 is configured to identify the behavior of the driver based on the first area image, the second area image and the third area image.

Optionally, referring to FIG. 7 , the apparatus further includes:

a size adjustment module 505, configured to adjust the first area image, the second area image and the third area image into area images of the same size;

The identifying module 504 is used to invoke a target network model, where the target network model is used to determine the behavior category of the behavior based on a group of regional images corresponding to any behavior;

Based on the resized first area image, second area image and third area image, the behavior of the driver is identified through the target network model.

Optionally, the identification module 504 is used to:

The target network model includes an input layer, an intermediate layer, a splicing layer, a fully connected layer and an output layer; through the input layer, based on the resolution of the resized regional image and the number of channels, the resized first The image data in the area image, the second area image and the third area image are subjected to channel superposition processing to obtain a first feature map;

Performing convolution sampling processing on the first feature map by the intermediate layer to obtain a plurality of second feature maps, the plurality of second feature maps having the same size and different channel numbers;

Perform channel stacking processing on the plurality of second feature maps through the splicing layer, and perform feature fusion on the feature maps after channel stacking through the convolution layer in the deep layer of the network to obtain a third feature map;

determining the driver's behavior based on the third feature map through the fully connected layer;

The behavior is output through the output layer.

Optionally, the identification module 504 is used to:

The middle layer includes N groups of convolution layers and N groups of sampling layers, each group of convolution layers corresponds to each group of sampling layers one-to-one; let i=1, determine the first feature map as the target feature map; The i group of convolution layers performs convolution processing on the target feature map, and the i-th group of sampling layers respectively sample the obtained feature maps according to two different multiples to obtain 2 ⁱ times the feature map and the i-th reference size. the second feature map, obtaining the ²ⁱ -fold feature map as the target feature map, and the reference size is greater than or equal to the size of the ²ⁱ -fold feature map;

When i is less than the N, let i=i+1, return the convolution processing of the target feature map through the i-th group of convolution layers, and obtain through the i-th group of sampling layers according to two different multiple pairs. The feature map is sampled to obtain 2 ⁱ times the feature map and the second feature map of the i th reference size, and the 2 ⁱ times feature map is obtained as the operation of the target feature map;

When i is equal to the N, the target feature map is subjected to convolution processing through the Nth group of convolution layers, and the Nth group of sampling layers is used to perform a sampling process on the obtained feature map to obtain the Nth reference size. The second feature map, end the operation.

Optionally, please refer to FIG. 8 , the apparatus further includes a training module 506, and the training module 506 is used for:

Acquiring multiple training samples, the multiple training samples include multiple sets of images and behavior categories in each set of images, and each set of images includes an area image of the behavior and a reduced area determined after reducing the area image an image, and an expanded area image determined after the area image is expanded;

The target network model is obtained after training the network model to be trained based on the plurality of training samples.

Optionally, the second obtaining module 502 is used for:

Invoke a target detection model, input the target image into the target detection model, and output a face detection frame and a preset behavior detection frame, and the target detection model is used to detect the face in the image based on any image. and pre-determined behavior;

When the number of the face detection frames is one, the face detection frame is determined as the target face detection frame; when the number of the face detection frames is multiple, it is obtained from multiple face detection frames The face detection frame with the largest area, and the acquired face detection frame is determined as the target face detection frame;

The first region image is determined based on the target face detection frame.

Optionally, the second obtaining module 502 is used for:

Filter out the preset behavior detection frame that does not overlap with the target face detection frame in the preset behavior detection frame, or whose distance from the target face detection frame is greater than a preset distance threshold;

The area corresponding to the preset behavior detection frame remaining after filtering is cut out from the target image to obtain the first area image.

Optionally, the first obtaining module 501 is used for:

detecting the working mode of the camera for photographing the driver's face;

When the working mode of the camera is an infrared shooting mode, acquiring a grayscale image obtained by shooting;

Invoke the image pseudo-color conversion model, input the grayscale image into the image pseudo-color conversion model, and output a three-channel color image corresponding to the gray-scale image, and the image pseudo-color conversion model is used to convert any grayscale image. converting the grayscale image into a three-channel color image corresponding to the grayscale image;

The output three-channel color image is acquired as the target image.

Optionally, referring to FIG. 9 , the apparatus further includes:

The third obtaining module 507 is used to obtain the current vehicle speed and light intensity;

The switching module 508 is configured to switch the working mode of the camera to the infrared shooting mode when the vehicle speed is greater than a preset vehicle speed threshold and the illumination intensity is lower than the illumination intensity threshold.

Optionally, please refer to FIG. 10, the device further includes:

A statistics module 509, configured to count the number of violations of the driver within a preset time period when the driver's behavior is a violation;

The alarm module 510 is configured to issue a warning of illegal driving when the number of violations of the driver reaches a preset number of thresholds within the preset time period.

In the embodiment of the present application, a target image including the driver's face is obtained, and a first area image including the driver's preset behavior is obtained from the target image, and the preset behavior is similar to the illegal behavior, that is, A first area image of a preset behavior similar to the violation behavior is acquired from the target image. Afterwards, in the target image, the first area image is cut in and out of different scales to obtain the second area image and the third area image. Since the first area image may only include a small part of the preset behavior, but does not include or include a very small part of the surrounding objects related to the preset behavior, it is impossible to accurately determine whether the preset behavior is actually a violation based on the first area image. Therefore, after the first area image is cut in and expanded in this application, the obtained second area image and third area image can include more information about the surrounding objects related to the preset behavior. Therefore, based on The three area images are used to detect the driver's behavior, which can improve the detection accuracy.

It should be noted that: when the driver's behavior recognition device provided in the above embodiment implements the driver's behavior recognition method, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated as required. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the driver's behavior recognition device and the driver's behavior recognition method embodiment provided by the above embodiments belong to the same concept, and the specific implementation process thereof is detailed in the method embodiment, which will not be repeated here.

FIG. 11 shows a structural block diagram of a terminal 1000 provided by an exemplary embodiment of the present application. The terminal 1000 can be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, the standard audio layer 3 of Moving Picture Experts compression), MP4 (Moving Picture Experts Group AudioLayer IV, the standard audio layer 4 of Moving Picture Experts compression) ) player, laptop or desktop computer. Terminal 1000 may also be called user equipment, portable terminal, laptop terminal, desktop terminal, and the like by other names.

Generally, the terminal 1000 includes: a processor 1001 and a memory 1002 .

The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1001 may use at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) accomplish. The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor used to process data in a wake-up state, also called a CPU (Central Processing Unit, central processing unit); A low-power processor for processing data in a standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor 1001 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.

Memory 1002 may include one or more computer-readable storage media, which may be non-transitory. Memory 1002 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction for being executed by the processor 1001 to implement the driver provided by the method embodiments of the present application behavior identification method.

In some embodiments, the terminal 1000 may optionally further include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 may be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1003 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1004 , a touch display screen 1005 , a camera 1006 , an audio circuit 1007 , a positioning component 1008 and a power supply 1009 .

The peripheral device interface 1003 may be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 1001 and the memory 1002 . In some embodiments, processor 1001, memory 1002, and peripherals interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one of processor 1001, memory 1002, and peripherals interface 1003 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 1004 is used for receiving and transmitting RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. The radio frequency circuit 1004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1004 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: World Wide Web, Metropolitan Area Network, Intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), wireless local area network and/or WiFi (Wireless Fidelity, Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may further include a circuit related to NFC (Near Field Communication, short-range wireless communication), which is not limited in this application.

The display screen 1005 is used for displaying UI (User Interface, user interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to acquire touch signals on or above the surface of the display screen 1005 . The touch signal can be input to the processor 1001 as a control signal for processing. At this time, the display screen 1005 may also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards. In some embodiments, there may be one display screen 1005, which is provided on the front panel of the terminal 1000; in other embodiments, there may be at least two display screens 1005, which are respectively arranged on different surfaces of the terminal 1000 or in a folded design; In still other embodiments, the display screen 1005 may be a flexible display screen, which is disposed on a curved surface or a folding surface of the terminal 1000 . Even, the display screen 1005 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light emitting diode).

The camera assembly 1006 is used to capture images or video. Optionally, the camera assembly 1006 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth-of-field camera to realize the background blur function, the main camera It is integrated with the wide-angle camera to achieve panoramic shooting and VR (Virtual Reality, virtual reality) shooting functions or other integrated shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a single color temperature flash or a dual color temperature flash. Dual color temperature flash refers to the combination of warm light flash and cold light flash, which can be used for light compensation under different color temperatures.

Audio circuitry 1007 may include a microphone and speakers. The microphone is used to collect the sound waves of the user and the environment, convert the sound waves into electrical signals, and input them to the processor 1001 for processing, or to the radio frequency circuit 1004 to realize voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively disposed in different parts of the terminal 1000 . The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The loudspeaker can be a traditional thin-film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for distance measurement and other purposes. In some embodiments, the audio circuit 1007 may also include a headphone jack.

The positioning component 1008 is used to locate the current geographic location of the terminal 1000 to implement navigation or LBS (Location Based Service, location-based service). The positioning component 1008 may be a positioning component based on the GPS (Global Positioning System, global positioning system) of the United States, the Beidou system of China or the Galileo system of Russia.

The power supply 1009 is used to power various components in the terminal 1000 . The power source 1009 may be alternating current, direct current, disposable batteries or rechargeable batteries. When the power source 1009 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. Wired rechargeable batteries are batteries that are charged through wired lines, and wireless rechargeable batteries are batteries that are charged through wireless coils. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, the terminal 1000 further includes one or more sensors 1010 . The one or more sensors 1010 include, but are not limited to, an acceleration sensor 1011 , a gyro sensor 1012 , a pressure sensor 1013 , a fingerprint sensor 1014 , an optical sensor 1015 and a proximity sensor 1016 .

The acceleration sensor 1011 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1000 . For example, the acceleration sensor 1011 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 1001 can control the touch display screen 1005 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1011 . The acceleration sensor 1011 can also be used for game or user movement data collection.

The gyroscope sensor 1012 can detect the body direction and rotation angle of the terminal 1000 , and the gyroscope sensor 1012 can cooperate with the acceleration sensor 1011 to collect 3D actions of the user on the terminal 1000 . The processor 1001 can implement the following functions according to the data collected by the gyro sensor 1012: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

The pressure sensor 1013 may be disposed on the side frame of the terminal 1000 and/or the lower layer of the touch display screen 1005 . When the pressure sensor 1013 is disposed on the side frame of the terminal 1000, the user's holding signal of the terminal 1000 can be detected, and the processor 1001 performs left and right hand identification or shortcut operations according to the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is disposed on the lower layer of the touch display screen 1005 , the processor 1001 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1005 . The operability controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

The fingerprint sensor 1014 is used to collect the user's fingerprint, and the processor 1001 identifies the user's identity according to the fingerprint collected by the fingerprint sensor 1014, or the fingerprint sensor 1014 identifies the user's identity according to the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1001 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 1014 may be provided on the front, back or side of the terminal 1000 . When the terminal 1000 is provided with physical buttons or a manufacturer's logo, the fingerprint sensor 1014 may be integrated with the physical buttons or the manufacturer's logo.

The optical sensor 1015 is used to collect ambient light intensity. In one embodiment, the processor 1001 may control the display brightness of the touch display screen 1005 according to the ambient light intensity collected by the optical sensor 1015 . Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1005 is decreased. In another embodiment, the processor 1001 may also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1015 .

A proximity sensor 1016 , also called a distance sensor, is usually disposed on the front panel of the terminal 1000 . The proximity sensor 1016 is used to collect the distance between the user and the front of the terminal 1000 . In one embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 gradually decreases, the processor 1001 controls the touch display screen 1005 to switch from the bright screen state to the off screen state; when the proximity sensor 1016 detects When the distance between the user and the front of the terminal 1000 gradually increases, the processor 1001 controls the touch display screen 1005 to switch from the off-screen state to the bright-screen state.

Those skilled in the art can understand that the structure shown in FIG. 11 does not constitute a limitation on the terminal 1000, and may include more or less components than the one shown, or combine some components, or adopt different component arrangements.

The embodiments of the present application further provide a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal can perform the behavior recognition of the driver provided by the above embodiments method.

The embodiments of the present application also provide a computer program product containing instructions, which, when running on a computer, cause the computer to execute the method for recognizing the driver's behavior provided by the above embodiments.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, etc.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection of the present application. within the range.

Claims (13)

1. A method for identifying a driver's behavior, the method comprising:

acquiring a target image, wherein the target image comprises a face of a driver;

acquiring a first area image from the target image, wherein the first area image comprises a preset behavior of the driver, and the preset behavior refers to a behavior with similarity to an illegal behavior larger than a preset threshold;

in the target image, carrying out reduction processing on the peripheral area of the first area image according to a first proportion threshold value to obtain a second area image, and carrying out expansion processing on the peripheral area of the first area image according to a second proportion threshold value to obtain a third area image;

recognizing the behavior of the driver based on the first area image, the second area image, and the third area image.

2. The method of claim 1, wherein prior to identifying the behavior of the driver based on the first region image, the second region image, and the third region image, further comprising:

adjusting the first area image, the second area image and the third area image to be area images of the same size;

accordingly, the identifying the behavior of the driver based on the first area image, the second area image, and the third area image includes:

calling a target network model, wherein the target network model is used for determining the behavior category of any behavior based on a group of regional images corresponding to the behavior;

and identifying the behavior of the driver through the target network model based on the first area image, the second area image and the third area image after the size adjustment.

3. The method of claim 2, wherein the target network model comprises an input layer, an intermediate layer, a splice layer, a fully-connected layer, and an output layer;

the identifying the behavior of the driver through the target network model based on the resized first area image, second area image, and third area image includes:

performing channel superposition processing on image data in the first area image, the second area image and the third area image after the size adjustment through the input layer based on the resolution and the number of channels of the area image after the size adjustment to obtain a first feature map;

performing convolution sampling processing on the first characteristic diagram through the intermediate layer to obtain a plurality of second characteristic diagrams, wherein the second characteristic diagrams are the same in size and different in channel number;

performing channel superposition processing on the plurality of second feature maps through the splicing layer, and performing feature fusion on the feature maps after channel superposition through the convolution layer in the deep network layer to obtain a third feature map;

determining, by the fully-connected layer, behavior of the driver based on the third feature map;

outputting the behavior through the output layer.

4. The method of claim 3, wherein the intermediate layer comprises N sets of convolutional layers and N sets of sampling layers, each set of convolutional layers corresponding one-to-one to each set of sampling layers;

the obtaining a plurality of second feature maps by performing convolution sampling processing on the first feature map through the intermediate layer includes:

determining the first feature map as a target feature map by taking i as 1; performing convolution processing on the target characteristic diagram through the ith group of convolution layers, and performing sampling processing on the obtained characteristic diagram through the ith group of sampling layers according to two different multiples to obtain 2ⁱMultiple feature map and ith second feature map of reference size, 2ⁱObtaining the target feature map as a multiple feature map, wherein the reference dimension is greater than or equal to 2ⁱThe size of the multiple feature map;

when i is smaller than N, making i equal to i +1, returning to the step of performing convolution processing on the target feature map through the ith group of convolution layers, and performing sampling processing on the obtained feature map according to two different multiples through the ith group of sampling layers to obtain 2ⁱMultiple feature map and ith second feature map of reference size, 2ⁱAcquiring a multiplied feature map as the operation of the target feature map;

and when i is equal to the N, performing convolution processing on the target feature map through the Nth group of convolution layers, performing primary sampling processing on the obtained feature map through the Nth group of sampling layers to obtain a second feature map of the Nth reference size, and ending the operation.

5. The method of claim 2, wherein prior to invoking the target network model, further comprising:

obtaining a plurality of training samples, wherein the plurality of training samples comprise a plurality of groups of images and behavior categories in each group of images, each group of images comprise regional images of behaviors, reduced regional images determined after the regional images are subjected to reduction processing, and expanded regional images determined after the regional images are subjected to expansion processing;

and training the network model to be trained based on the plurality of training samples to obtain the target network model.

6. The method of claim 1, wherein said acquiring a first region image from said target image comprises:

calling a target detection model, inputting the target image into the target detection model, and outputting a face detection frame and a preset behavior detection frame, wherein the target detection model is used for identifying the face and the preset behavior in the image based on any image;

when the number of the face detection frames is one, determining the face detection frames as target face detection frames; when the number of the face detection frames is multiple, acquiring a face detection frame with the largest area from the multiple face detection frames, and determining the acquired face detection frame as a target face detection frame;

and determining the first area image based on the target face detection frame.

7. The method of claim 6, wherein the determining the first region image based on the target face detection box comprises:

filtering out the preset behavior detection frames which are not overlapped with the target face detection frame or have the distance with the target face detection frame larger than a preset distance threshold value;

and cutting out the region corresponding to the filtered residual preset behavior detection frame from the target image to obtain the first region image.

8. The method of claim 1, wherein said acquiring a target image comprises:

detecting a working mode of a camera for shooting the face of the driver;

when the working mode of the camera is an infrared shooting mode, obtaining a gray image obtained by shooting;

calling an image pseudo-color conversion model, inputting the gray level image into the image pseudo-color conversion model, and outputting a three-channel color image corresponding to the gray level image, wherein the image pseudo-color conversion model is used for converting any gray level image into the three-channel color image corresponding to the gray level image;

and acquiring the output three-channel color image as the target image.

9. The method of claim 8, wherein prior to detecting an operating mode of a camera used to capture the driver's face, further comprising:

acquiring the current vehicle speed and the current illumination intensity;

and when the vehicle speed is greater than a preset vehicle speed threshold value and the illumination intensity is lower than an illumination intensity threshold value, switching the working mode of the camera to the infrared shooting mode.

10. The method of claim 1, wherein after identifying the behavior of the driver based on the first region image, the second region image, and the third region image, further comprising:

when the behavior of the driver belongs to the violation behavior, counting the number of the violation behaviors of the driver in a preset time length;

and when the number of illegal behaviors of the driver reaches a preset number threshold within the preset time, giving an alarm for illegal driving.

11. A behavior recognition apparatus for a driver, characterized in that the apparatus comprises:

the device comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a target image, and the target image comprises the face of a driver;

the second acquisition module is used for acquiring a first area image from the target image, wherein the first area image comprises a preset behavior of the driver, and the preset behavior refers to a behavior with similarity between the behavior and an illegal behavior larger than a preset threshold;

the image processing module is used for carrying out reduction processing on the peripheral area of the first area image according to a first proportion threshold value in the target image to obtain a second area image, and carrying out expansion processing on the peripheral area of the first area image according to a second proportion threshold value to obtain a third area image;

an identification module to identify a behavior of the driver based on the first area image, the second area image, and the third area image.

12. A smart device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to implement the steps of any of the methods of claims 1-10.

13. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the steps of any of the methods of claims 1-10.

Images