TW202330327A - Blind spot detection auxiliary system for motorcycle - Google Patents

Abstract

A blind spot detection auxiliary system for a motorcycle has an image capture device, an image processor and an alarming module. The image capturing device is set on the motorcycle for shooting the rear of the motorcycle to generate a video. The image processor is coupled to the image capturing device for determining at least one vehicle behind the motorcycle based on the video; converting image coordinates of a display area of each vehicle into world coordinates to calculate a distance between each vehicle and the motorcycle; obtaining a local optical flow of at least one characteristic point of each vehicle, calculating a relative velocity of each vehicle according to the local optical flow, and calculating a steering direction of the motorcycle according to a global optical flow; and determining whether the driving distance and the speed of each vehicle satisfy predetermined conditions. When the driving distance and speed of each vehicle satisfy the predetermined conditions, the alarming module issues a warning.

Description

Blind spot detection assist system for locomotives

The present invention relates to a blind spot detection auxiliary system for a locomotive, in particular to a blind spot detection auxiliary system for a locomotive that detects vehicles coming from behind through real-time image recognition and judges the steering of the vehicle.

The rear blind spot detection function of the Advanced Driver Assistance Systems (ADAS) carried out by current two-wheeled motor vehicles (referred to as: locomotives) is achieved by using mirrors or left and right dual radar solutions. Among them, the way of the reflector is to install a convex mirror on the top of the motorcycle dashboard, and through the reflection of the convex mirror, the motorcycle rider can see the picture behind, but because the convex mirror will also reflect the figure of the motorcycle rider himself at the same time, resulting in dead corner. As for the left and right dual radar solution, one radar is installed on each side of the locomotive to detect whether there are other vehicles approaching behind the locomotive. However, the radar cannot determine whether the approaching object is really a vehicle, nor can it identify the type of vehicle. make its function limited.

An embodiment of the present invention discloses a blind spot detection auxiliary system for a locomotive, which includes an image capture device, an image processor, and a warning module. The image capture device is installed on the locomotive to photograph the rear of the locomotive to generate video. The image processor is coupled to the image capture device and includes a machine learning object detection module, an image distortion perspective conversion module, an image optical flow vector module, and a security identification module. The machine learning object detection module is used to analyze the video to determine at least one vehicle behind the locomotive. The image distortion perspective conversion module is used to convert the image coordinates of the display area of the at least one vehicle into world coordinates according to the video, so as to calculate the driving distance between the at least one vehicle and the locomotive. The image optical flow vector module is used to obtain the overall optical flow and the regional optical flow (optical flow) of at least one feature point of the at least one vehicle according to the video, and calculate the optical flow of the at least one vehicle according to the regional optical flow. Relative vehicle speed, as well as judging the steering of the ego vehicle through the global optical flow. The safety judgment module is used to judge whether the driving distance and speed meet the preset conditions. The warning module is coupled to the image processor, and is used for issuing a warning when the safety judging module judges that the driving distance and the speed meet the preset conditions.

Please refer to FIG. 1 , which is a functional block diagram of a blind spot detection assisting system 100 according to an embodiment of the present invention. The blind spot detection assistance system 100 includes an image capture device 110 , an image processor 120 and a warning module 130 . The image capture device 110 is installed on the locomotive for shooting the rear of the locomotive to generate video Sv. The image capture device 110 is, for example, a digital camera having a photosensitive element and a lens. The image processor 120 is coupled to the image capture device 110 to receive the video Sv generated by the image capture device 110 . In another embodiment of the present invention, the blind spot detection assisting system 100 can also store the video Sv to achieve the function of video monitoring and recording. The image processor 120 includes a machine learning object detection module 122 , an image distortion and perspective conversion module 124 , an image optical flow vector module 126 and a security identification module 128 . The machine learning object detection module 122 is used to analyze the video Sv to determine at least one vehicle behind the locomotive. More specifically, the machine learning object detection module 122 can be pre-trained through machine learning to establish the ability to recognize vehicles in the video Sv frame. After the machine learning object detection module 122 is trained, it can recognize the objects in the video Sv frame to find the vehicle in the video Sv frame. Take Fig. 2 and Fig. 3 as an example, wherein Fig. 2 is a frame in the video Sv generated by the image capture device 110, and Fig. 3 is the frame of the machine learning object detection module 122 in Fig. 2 On the screen after judging the vehicle behind the locomotive. The machine learning object detection module 122 will mark the determined vehicles C1 , C2 and C3 with boxes in areas A1 , A2 and A3 respectively. In this way, the motorcyclist can know which vehicles are behind him through the areas A1, A2 and A3 displayed on the screen. As for how the machine learning object detection module 122 recognizes the vehicle in the video Sv frame through machine learning, there are many previous technologies that can be referred to, and will not be repeated here. In this embodiment, the vehicles C1 , C2 and C3 are respectively an object for the machine learning object detection module 122 , and the areas A1 , A2 and A3 can be referred to as areas of objects detected by machine learning.

The image distortion perspective conversion module 124 of the image processor 120 is used to transform the image coordinates of the areas A1, A2 and A3 of the vehicles C1, C2 and C3 identified by the machine learning object detection module 122 according to the video Sv into world coordinates. Among them, the technology of converting the image coordinates into the world coordinates is also a known technology, and will not be described here. In this embodiment, in order to effectively convert the image coordinates of each pixel in the frame into the corresponding world coordinates, the image warp perspective conversion module 124 implements it by means of a lookup table. The above-mentioned lookup table records the world coordinates corresponding to the image coordinates of each pixel in the frame. The image distortion perspective conversion module 124 can substitute the image coordinates of each pixel into the lookup table to obtain the corresponding world coordinates. In order for the motorcyclist to easily know the approximate distance of the vehicle behind through the video Sv, the image distortion perspective conversion module 124 can display the distance line and the vanishing point on the video Sv. Please refer to FIG. 4 . FIG. 4 is the screen after the image distortion perspective conversion module 124 marks the distance line 170 and the vanishing point N on the screen in FIG. 3 . Among them, the distance between two adjacent distance lines 170 is equal (for example: 1 meter or other lengths), the horizon line in the picture is marked by symbol 160, and the vanishing point N represents each object displayed on the picture (such as the road on both sides buildings) will eventually disappear in the picture, and it is widely used in the field of image perspective.

The image optical flow vector module 126 of the image processor 120 is used to obtain the optical flow (optical flow) of the feature points in the areas A1, A2 and A3 of the vehicles C1, C2 and C3 according to the video Sv, and obtain the optical flow of the feature points in the regions A1, A2 and A3 of the vehicles C1, C2 and C3, and obtain The optical flow of the feature points of C1, C2 and C3 calculates the relative speed of each vehicle C1, C2 and C3. The number of feature points of each vehicle C1, C2 and C3 can be one or more, and the selection of feature points can be specific image edge corner points of the vehicle, for example: the edge corner points of the headlights, the edge corner points of the license plate or The bottom center points of the areas A1, A2, and A3 belong to. Please refer to FIG. 5 , which is a schematic diagram of the optical flows E1 , E2 and E3 obtained by the image optical flow vector module 126 . The optical flows E1, E2, and E3 correspond to the historical trajectories of the optical flows of the vehicles C1, C2, and C3, respectively, and the image processor 120 can calculate the speeds of the vehicles C1, C2, and C3 respectively according to the optical flows E1, E2, and E3.

Please refer to FIG. 6 . FIG. 6 is used to help explain how the image processor 120 of the blind spot detection assistance system in FIG. 1 calculates the vehicle speed according to the optical flow. in Figure 6 is the horizontal movement vector of the optical flow, is the horizontal offset vector of the center of the optical axis of the lens of the image pickup device 110, X ₀ is the lateral distance between the vehicle and the locomotive, Indicates the relative speed between the vehicle and the ego vehicle (i.e. the locomotive itself), f _x is the focal length of the lens of the image capture device 110, and ψ represents the optical axis between the feature point of the vehicle and the lens of the image capture device 110 in the previous frame The translation angle between (the translation angle between the vehicle and the optical axis of the lens at the previous moment), and θ indicates the change of the angle between the feature point of the vehicle and the optical axis between the front and rear two pictures (the translation angle between the vehicle and the optical axis of the lens at this moment and The amount of change between the translation angle of the vehicle and the optical axis of the lens at the previous moment). Among them, ψ and (θ+ψ) can be obtained through the following trigonometric functions:

Among them, c _x is the horizontal offset vector The absolute value of , u ₀ is the optical flow vector the absolute value of . while the relative velocity Then it can be obtained by the following formula:

Among them, the relative speed The unit is how many meters per frame (m/frame). Therefore, when the number of frames per second of the video Sv is known, the relative speed The unit of can be converted into meters per second (m/sec).

In addition, the safety judging module 128 is used to judge whether the driving distance and speed of the vehicles C1, C2 and C3 meet the preset conditions (for example: whether the distance is too short and/or whether the speed is too fast). When the safety judging module 128 determines that the driving distance and speed of any of the vehicles C1, C2 and C3 meet the preset conditions, the image processor 120 will control the coupled warning module 130 to issue a warning to remind motorcyclists to pay attention. The warning module 130 may include a display 132 and an alarm device 134 , and the way in which the warning module 130 issues warnings may include but not limited to: the display 132 displays a warning sign on a corresponding position of a vehicle requiring attention on the displayed screen. As shown in FIG. 7 , since the driving distance and speed of the vehicles C1 and C2 are determined to meet the above preset conditions, the display 132 will mark the corresponding positions of the vehicles C1 and C2 on the screen with warning signs 180 respectively. In addition, the alarm device 134 may be a device capable of emitting sound and/or light, for example, a device including a speaker and/or a light emitting diode (LED). When the above preset conditions are met, the alarm device 134 can emit sound and/or bright light to remind the motorcyclist.

In another embodiment of the present invention, the machine learning object detection module 122 is also used to identify the vehicle types of the vehicles C1 to C3 according to the video Sv, and the safety identification module 128 can judge whether to send Alert to alert motorcyclists. For example, when the vehicle types identified by the machine learning object detection module 122 are combined vehicles, buses, large trucks, etc., motorcyclists can be reminded to pay attention.

In addition, in addition to the optical flow in the above-mentioned areas A1, A2 and A3 (which can be called: regional optical flow) can be used to calculate the vehicle speed, the optical flow outside the areas A1, A2 and A3 (which can be called: global Optical flow) can be used to calculate the updated position of the vanishing point N in the picture, and the offset between the vanishing point N in the picture and the preset vanishing point can be used to judge whether the locomotive is turning , and judge whether the steering is right or left. Wherein, the above-mentioned preset vanishing point may have the same distance from the left side and the right side of the picture in the picture. The specific way to judge whether the locomotive is turning and turning is to select at least two feature points outside the area of the vehicle in the picture, calculate the moving vector of the optical flow of the selected feature points one by one, and then calculate the moving vector of these optical flow Extended, the extended intersection point can be used as a reference for the vanishing point N. Furthermore, the coordinates of the intersection points of the movement vectors of multiple optical flows can be calculated and averaged, and the averaged coordinates can be used as the coordinates of the vanishing point N. Take the moving vector of two sets of optical flow and For example, if the coordinates of the intersection points are , then the coordinates of the intersection point can be obtained by the following equation . ................(1) ................(2) ................(3)

in, is the optical flow vector The coordinates of the feature points to which they belong in the screen, is the optical flow vector The coordinates of the feature points to which they belong in the screen, the above formula (1) represents the optical flow vector The simultaneous equations in the picture, and the above formula (2) represents the optical flow vector Simultaneous equations in the picture , these two simultaneous equations represent the optical flow vector in the picture and and Two lines that overlap, where and is the unknown, and the unknown can be obtained by the matrix in the above formula (3) and , and then get the optical flow vector and Coordinates of the extended intersection point . To simplify the calculation, the optical flow vectors of two feature points in the picture and The extended intersection point can be directly used as the vanishing point N. For the sake of accuracy, the average value of the coordinates of the intersection points of the optical flow vectors of more feature points can be taken as the coordinates of the vanishing point N. The offset of the vanishing point N in the picture can be used as a basis for judging whether the locomotive is turning. Please refer to Figure 8A and Figure 8B, Figure 8A is a picture of the locomotive before turning, and Figure 8B is a picture of the locomotive when it is making a right turn. It can be seen from the pictures before and after the turn that the position of the vanishing point N in the two pictures has changed, and its horizontal coordinates have shifted to the right, so the image processor 120 can judge that the locomotive is turning right. At this time, if the image processor 120 also determines that the oncoming vehicle C1 on the right is too short or its speed is too fast, the safety judgment module 128 can issue a turning warning through the warning module 130 .

In another embodiment of the present invention, the blind spot detection assisting system 100 may further include a gravity sensor 150 for judging the steering of the locomotive, and the safety judging module 128 is also based on the steering of the locomotive and the positions of the vehicles C1 to C3 behind. The position determines whether to control the warning module 130 to issue a turning warning.

In another embodiment of the present invention, the image processor 120 also judges the inclination angle of the horizon 160 according to the video signal Sv, and judges the steering of the locomotive according to the inclination angle of the horizon 160, and the safety judgment module 128 also judges the steering of the locomotive and The positions of the vehicles C1 to C3 behind determine whether to control the warning module 130 to issue a turning warning.

Please refer to FIG. 9 , which is a flowchart of a method 900 of the blind spot detection assisting system 100 according to an embodiment of the present invention. Method 900 includes the following steps:

Step S910: the image capture device 110 photographs the rear of the locomotive to generate a video Sv;

Step S920: the machine learning object detection module 122 analyzes the video Sv to determine at least one vehicle C1 to C3 behind the locomotive;

Step S930: The image distortion perspective conversion module 124 converts the image coordinates of the display areas A1 to A3 of the vehicles C1 to C3 into world coordinates according to the video Sv, so as to calculate the driving distance between the vehicles C1 to C3 and the locomotive;

Step S940: The image optical flow vector module 126 obtains the optical flows E1 to E3 of at least one feature point of each of the vehicles C1 to C3 according to the video Sv, and calculates the speed of each of the vehicles C1 to C3 according to the optical flows E1 to E3;

Step S950: The image optical flow vector module 126 obtains the optical flow of at least two feature points outside the areas A1, A2 and A3 of the vehicles C1 to C3 according to the video Sv, and calculates the vanishing point N based on the optical flow of the at least two feature points. The offset (that is, the offset between the vanishing point N in the picture and the preset vanishing point), wherein, the image processor 120 can judge whether the locomotive is turning according to the offset of the vanishing point N in the picture and turn;

Step S960: Determine whether the driving distance and vehicle speed of each vehicle C1 to C3 meet the preset conditions; if the conditions are met, proceed to Step S960; otherwise, end the process or return to Step S910; and

Step S970: the warning device issues a warning.

In step S950, the image processor 120 can not only judge whether the locomotive is turning and the steering of the locomotive according to the offset of the vanishing point N in the screen, but also judge according to the signal provided by the gravity sensor 150. Whether the locomotive is making a turn and the locomotive's steering. In addition, the blind spot detection assisting system 100 can repeatedly perform the above steps S910 to S970 to provide real-time blind spot detection and warning functions according to the video Sv.

The present invention uses machine-learning artificial intelligence (AI) image recognition technology to capture and mark the vehicles in the screen, and calculates the relative distance and relative speed of each vehicle through image distortion perspective transformation and optical flow detection technology, which is comparable to the locomotive Information about the left and right turns. Therefore, the present invention can use a single wide-angle lens camera to replace the traditional dual radar rear blind zone warning solution, and can provide visual warning and/or sound warning, as well as video monitoring and recording functions. The above descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

100: Blind spot detection auxiliary system 110: Image capture device 120: Image processor 122: Machine learning object detection module 124: Image distortion perspective conversion module 126: Image optical flow vector module 128: Safety discrimination module 130: Warning module 132: display 134: alarm device 150: gravity sensor 160: horizon 170: distance line 180: warning sign 900: methods S910 to S970: steps A1, A2, A3: area C1 of the object detected by machine learning , C2, C3: vehicles E1, E2, E3: the historical trajectory of the vehicle optical flow direction N: vanishing point Sv: video : The relative speed of the vehicle and the locomotive X ₀ : The lateral distance between the vehicle and the locomotive ψ: The translation angle between the vehicle and the optical axis of the lens at the previous moment θ: The translation angle between the vehicle and the optical axis of the lens at the moment and the translation between the vehicle and the optical axis of the lens at the previous moment The amount of change between the angles : Horizontal movement vector of optical flow : The horizontal offset vector of the optical axis center of the lens

FIG. 1 is a functional block diagram of a blind spot detection assistance system according to an embodiment of the present invention. Fig. 2 is one of the frames of the video generated by the image capture device of the blind spot detection assistance system in Fig. 1. Figure 3 is a picture of the machine learning object detection module of the blind spot detection assistance system in Figure 1 judging the vehicle behind the locomotive on the screen in Figure 2. Figure 4 is the image after the distance line and vanishing point are marked on the screen in Figure 3 by the image distortion perspective conversion module of the blind spot detection assistance system in Figure 1. FIG. 5 is a schematic diagram of the optical flow obtained by the image optical flow vector module of the blind spot detection assistance system in FIG. 1 . Figure 6 is used to help explain how the image processor of the blind spot detection assistance system in Figure 1 calculates the vehicle speed based on the optical flow. Fig. 7 is a schematic diagram of the blind spot detection assistance system in Fig. 1 displaying warning signs on the monitor. Figure 8A and Figure 8B are used to assist in explaining how the blind spot detection auxiliary system in Figure 1 determines the steering of the locomotive. FIG. 9 is a flow chart of the method of the blind spot detection assisting system according to an embodiment of the present invention.

100: Blind Spot Detection Assist System

110: image capture device

120: image processor

122:Machine Learning Object Detection Module

124: Image Distortion Perspective Transformation Module

126:Image optical flow vector module

128:Safety discrimination module

130:Warning module

132: Display

134: Alarm device

150: Gravity sensor

Sv: Video

Claims (8)

A blind spot detection assist system for a locomotive, comprising: An image capture device, installed on a locomotive, is used to photograph the rear of the locomotive to generate a video; An image processor, coupled to the image capture device, includes; a machine learning object detection module for analyzing the video to determine at least one vehicle behind the locomotive; An image distortion perspective conversion module, used to convert the image coordinates of a display area of the at least one vehicle into world coordinates according to the video, so as to calculate the driving distance between the at least one vehicle and the locomotive; An image optical flow vector module, used to obtain the optical flow (optical flow) of at least one feature point of the at least one vehicle according to the video, and calculate the relative speed of the at least one vehicle according to the optical flow; and A safety judging module, used to judge whether the driving distance and the speed meet a preset condition; and A warning module, coupled to the image processor, is used to issue a warning when the safety judging module judges that the driving distance and the vehicle speed meet the preset condition. The blind spot detection assisting system for a locomotive according to claim 1, wherein the machine learning object detection module is also used to identify the type of the at least one vehicle according to the video. The blind spot detection auxiliary system for locomotives as described in claim 1, wherein the image optical flow vector module is also used to obtain the optical flow of at least two feature points outside the area of each vehicle according to the video, and according to the above The optical flow of at least two feature points calculates the offset of a vanishing point; Wherein the image processor judges the steering of the locomotive according to the offset of the vanishing point; and Wherein the safety judging module also controls the warning module to issue a turning warning according to the steering of the locomotive and the position of the at least one vehicle. The blind spot detection auxiliary system for a locomotive as described in claim 1 further includes a gravity sensor for judging the steering of the locomotive, and the safety judgment module is also based on the steering of the locomotive and the at least one The location of the vehicle is used to control the warning module to issue a turning warning. The blind spot detection auxiliary system for a locomotive as described in claim 1, wherein the image processor also judges the inclination angle of a horizon according to the video, and judges the steering of the locomotive according to the inclination angle of the horizon, and the The safety judging module also controls the warning module to issue a turning warning according to the steering of the locomotive and the position of the at least one vehicle. The blind spot detection auxiliary system for locomotives according to claim 1, wherein the image optical flow vector module calculates the speed of the at least one vehicle according to the displacement vector of the optical flow. The blind spot detection auxiliary system for a motorcycle as described in claim 1, wherein the warning module includes a display for displaying the video, and is used for corresponding to the at least one vehicle when the preset condition is met A warning sign is displayed at the location. The blind spot detection auxiliary system for a motorcycle as described in claim 1, wherein the warning module includes an alarm device for emitting sound and/or light when the preset condition is met.

Images