WO2022044453A1 - Image processing device, image processing method, and vehicle onboard electronic control device - Google Patents

Abstract

An image processing device for processing images photographed by a vehicle onboard camera. The image processing device comprises: an image acquisition unit for acquiring an image from the vehicle onboard camera; a structure color specification unit for specifying, on the basis of the type of a structure estimated to be present in the surroundings of the host vehicle, color information pertaining to the structure; an image region specification unit for specifying, on the basis of the type of the structure, a region in the image where the structure is estimated to be present; and a camera abnormality detection unit for detecting an abnormality in the vehicle onboard camera. When the structure is present in the surroundings of the host vehicle, the camera abnormality detection unit compares the color information obtained from the structure color specification unit and the color component of the region in the image where the structure is estimated to be present, and detects the abnormality in the vehicle onboard camera.

Description

Image processing device, image processing method, and in-vehicle electronic control device Capture by reference

This application claims the priority of Japanese Patent Application No. 2020-141436, which is a Japanese application filed on August 25, 2020, and incorporates it into this application by referring to its contents.

The present invention relates to an image processing device, and more particularly to a method for easily detecting an abnormality in an in-vehicle camera.

With the progress of electronic control technology of vehicles such as advanced driving support and automatic driving, image sensors are being installed outside the vehicle interior, and the chances that the front member (for example, lens) of the image sensor becomes dirty are increasing. .. If foreign matter adheres to the front lens of the image sensor, the image in the area cannot be captured, and the recognition accuracy of the object outside the vehicle may decrease.

For example, in Patent Document 1 (Japanese Unexamined Patent Publication No. 2007-318355), the difference image generation unit acquires a plurality of image data captured by a camera at different times from the image storage unit, and among these image data. Generate difference image data. The difference image update unit compares the previous difference image and the current difference image for each pixel, and updates the current difference image with the pixel having the larger difference. When the number of updates of the difference image reaches a predetermined number, the latest difference image is output to the dirt determination unit. The dirt determination unit describes an image pickup device that determines the presence or absence of lens dirt in the camera based on the latest difference image.

Further, in Patent Document 2 (Japanese Unexamined Patent Publication No. 2012-166705), a detection means for detecting a predetermined long object to be detected formed on a predetermined vehicle peripheral region based on an image taken by an in-vehicle camera. A determination means for determining the presence or absence of foreign matter adhering to the lens based on the detection result thereof is provided, and the determination means 8 has a predetermined constantness regarding the length or formation interval of the detected object to be detected. A foreign matter adhesion determination device for an in-vehicle camera lens that makes a determination based on the presence or absence of a foreign object is described.

Further, Patent Document 3 (Japanese Unexamined Patent Publication No. 2019-29940) is a deposit detection device for detecting deposits reflected in a photographed image taken by an image pickup device installed on a moving object, and the deposits are detected from the photographed image. The contour extraction unit that extracts the contour area of the deposit as the contour region, the inner extraction unit that extracts the region inside the contour of the deposit as the inner region from the captured image, and the contour region and the contour of the deposit are compared. The one that matches either the shape or the brightness is detected as the deposit contour region, the inner region is compared with the inside of the contour of the deposit, and the one that matches either the shape or the brightness is the deposit. Adhesion detection that detects as an inner region and detects as a deposit detection region a region of a lens deposit consisting of either the contour of the deposit or the inside of the contour from either the contour region of the deposit or the inner region of the deposit. A deposit detection device including a region determination unit is described.

The camera has a function to detect a hardware failure of the image sensor. Further, although many methods for detecting obstacles due to deposits on the lens provided on the front surface of the camera have been proposed, the processing load is high and the processing load is required to be reduced.

Therefore, an object of the present invention is to detect an abnormality in the camera with a low processing load.

A typical example of the invention disclosed in the present application is as follows. That is, it is an image processing device that processes an image taken by an in-vehicle camera, and is based on an image acquisition unit that acquires an image from the in-vehicle camera and a type of structure presumed to exist around the own vehicle. A structure color specifying part that specifies the color information of the structure, an image area specifying part that specifies a region in the image where the structure is presumed to exist based on the type of the structure, and the in-vehicle camera. The camera abnormality detecting unit is provided with a camera abnormality detecting unit for detecting the abnormality of the above, and when the structure is present around the own vehicle, the camera abnormality detecting unit includes color information obtained from the structure color specifying unit and the image. It is characterized in that an abnormality of the in-vehicle camera is detected by comparing with a color component of a region in which the structure is presumed to be present.

According to one aspect of the present invention, an abnormality of the camera can be detected with a low processing load. Issues, configurations and effects other than those mentioned above will be clarified by the description of the following examples.

It is a block diagram which shows the structure of the image processing apparatus of the Example of this invention. It is a figure which shows the structural example of the structure position table of the Example of this invention. It is a figure which shows the structural example of the structural color component table of the Example of this invention. It is a figure which shows the structural example of the structural color component table of the Example of this invention. It is a figure which shows the structural example of the structural color component table of the Example of this invention. It is a flowchart of the camera abnormality detection processing of the Example of this invention. It is a flowchart of the camera abnormality detection processing of the Example of this invention. It is a flowchart of the camera abnormality detection processing of the Example of this invention.

<Example 1>
FIG. 1 is a block diagram showing a configuration of an image processing device according to an embodiment of the present invention.

The image processing device 1 is mounted on an in-vehicle electronic control device (ECU) or an in-vehicle camera device, and has a memory 10, an arithmetic unit (CPU) 30, a network interface (CAN) 40, an interface (I / F) 50, and a network interface ( It has Eth) 60.

The arithmetic unit 30 executes the program stored in the memory 10. A part of the processing performed by the arithmetic unit 30 by executing the program may be executed by another arithmetic unit (for example, hardware such as FPGA (Field Programable Gate Array) or ASIC (Application Specific Integrated Circuit)).

The memory 10 includes a ROM and a RAM which are non-volatile storage elements. The ROM stores an invariant program (for example, BIOS) and the like. The RAM is a high-speed and volatile storage element such as DRAM (Dynamic Random Access Memory) and a non-volatile storage element (non-temporary storage medium) such as SRAM (Static Random Access Memory). Stores the program executed by and the data used when the program is executed.

Specifically, the memory 10 includes an image acquisition unit 11, a vehicle information acquisition unit 12, a map information acquisition unit 13, an image area identification unit 14, a structure color identification unit 16, a camera abnormality detection unit 18, and an abnormality area notification completed determination. A program for realizing each functional block such as the unit 19 and the abnormality notification unit 21 is stored. Further, the memory 10 stores data referred to when the program is executed, such as the structure position table 15, the structure color component table 17, and the abnormal area notified table 20.

The network interface (CAN) 40 controls communication with other devices mounted on the vehicle via CAN (Control Area Network). The interface (I / F) 50 is a serial or parallel input / output interface, and an in-vehicle camera 51 is connected to the interface (I / F) 50. The network interface (Eth) 60 controls communication with other devices mounted on the vehicle via Ethernet (registered trademark, the same applies hereinafter).

The image acquisition unit 11 acquires an image taken by the vehicle-mounted camera 51 via the interface 50. The vehicle information 41 including the behavior of the vehicle (for example, speed, acceleration, steering angle, occupant's brake and accelerator operation amount, etc.) is acquired via the vehicle information acquisition unit 12 and the network interface (CAN) 40. The map information acquisition unit 13 acquires map information 61 via the network interface (Eth) 60. The map information to be acquired may be detailed map information for automatic driving or map information for navigation.

The image area specifying unit 14 refers to the structure position table 15 and specifies a structure detection area in which the structure is presumed to exist in the image. The structure position table 15 is a table in which information on a region in which a structure estimated from map information may exist in an image is recorded, and the details thereof will be described with reference to FIG.

The structure color specifying unit 16 specifies the color of the structure to be determined with reference to the structure color component table 17. The structure color component table 17 is a table in which color information for each type of structure is recorded, and the details thereof will be described with reference to FIGS. 3, 4, and 5.

The camera abnormality detection unit 18 detects deposits on the front surface of the camera (for example, a lens) based on the color information of the structure existing in the image taken by the in-vehicle camera 51.

The abnormal area notified determination unit 19 refers to the abnormal area already notified table 20 and determines whether the abnormality detected by the camera abnormality detecting unit 18 has been notified. The abnormality area already notified table 20 is a table in which the abnormality notified by the abnormality notification unit 21 is recorded. The abnormality notification unit 21 notifies the occupant and other ECUs of the abnormality determined by the abnormality area notification completion determination unit 19 to be unnotified.

FIG. 2 is a diagram showing a configuration example of the structure position table 15 according to the embodiment of the present invention.

In the structure position table 15, the image area specifying unit 14 is referred to for specifying a structure detection area (that is, a region for determining the color of the structure) in which the structure is presumed to exist, and each type of structure is referred to. Holds the coordinate data in the image of.

For example, in the structure position table 15 shown in FIG. 2, the position of the structure detection area where a traffic light, a road sign, and a pedestrian crossing are presumed to exist in the image is recorded as a structure for determining color. The structure detection area recorded in the structure position table 15 shown in FIG. 2 is defined by the three coordinates of the vertices of the triangle. The shape of the structure detection area recorded in the structure position table 15 may be a quadrangle (trapezoid, square, rectangle) instead of a triangle. Since the image captured by the in-vehicle camera 51 appears small at a distance, if the structure detection area is defined by a triangle or a trapezoid, the area where the structure exists can be accurately represented and the structure detection area can be made small.

Further, a plurality of structure detection areas may be recorded for the same type of structure. For example, since the traffic light is provided on the left side of the road (above the own lane in the left traffic) and the right side (above the oncoming lane in the left traffic), if a plurality of structure detection areas are defined according to the installation pattern of the traffic light. good.

Since the position where the structure can be seen in the image changes depending on the distance from the intersection, the structure detection area acquired from the structure position table 15 may be changed. Further, the structure detection area may be calculated based on the distance from the intersection using the data of one structure detection area as a variable, and the structure detection area may be changed according to the distance from the intersection. By switching the structure detection area according to the position and distance in this way, the scanning range for color detection can be reduced and the processing load can be reduced.

In this embodiment, the structure detection areas of traffic lights, road signs, and pedestrian crossings are recorded because the colors are similar in the same type of structures and the pattern of the installation location is constant, but other structures. Structure detection areas for objects (eg, road signs (road paint), railroad crossings) may be defined.

FIGS. 3, 4, and 5 are diagrams showing a configuration example of the structure color component table 17 according to the embodiment of the present invention, and a plurality of colors are defined according to the combination of the environment and the structure. In the illustrated structure color component table 17, the environment is composed of a plurality of tables divided according to the weather and day and night. FIG. 3 shows the color of the structure in the daytime in fine weather, FIG. 4 shows the color of the structure in the daytime in rainy weather, and FIG. Represents. Colors may be defined in other formats as long as the colors are defined according to the combination of environment and structure.

The color of the structure is represented by the chromaticity of red, green, and blue in the RGB color space, but other color spaces (for example, CMYK) may be used.

In each environment, the typical colors contained in the structure are defined. For example, many stop signs are placed at intersections and are colored white on a red background. Therefore, the red color of the ground color is defined. Further, the blue color of the ground color of the sign indicating the prohibition of traveling outside the designated direction may be defined, or the yellow color of the ground color of the sign indicating the shape of the intersection may be defined.

For traffic lights, it is advisable to define one or both of the color when lit in blue and the color when lit in red. Although the lighting time is short, yellow lighting may be defined. Further, the color (blue, yellow, red) of the signal being turned off may be defined.

For pedestrian crossings, it is good to define the white color of the paint. In the RGB space, if the chromaticity of each color of RGB is equal, it is either white to gray or black depending on the lightness.

FIGS. 6, 7, and 8 are flowcharts of camera abnormality detection processing.

First, the arithmetic unit 30 executes a camera abnormality detection method selection process. Specifically, the image processing device 1 determines whether or not the map information can be acquired (S601). For example, if the vehicle is equipped with an automatic driving function, detailed map information can be acquired. Further, if the vehicle is equipped with a navigation system, map information for navigation can be acquired. Further, if the map information is provided from the map ECU via CAN, the image processing device 1 can acquire the map information via CAN.

If the map information can be obtained, proceed to the camera abnormality detection process (Fig. 7) from the map information. On the other hand, if the map information cannot be acquired, the process proceeds to the camera abnormality detection process (FIG. 8) from the own vehicle information.

In the camera abnormality detection process from the map information, first, the image acquisition unit 11 acquires the image data taken by the in-vehicle camera 51 (S701). The image acquisition unit 11 may acquire an image every frame (for example, 30 fps), but may acquire one image every few frames (for example, every 0.5 seconds). The number of images acquired per unit time may be changed depending on the traveling speed. For example, when the traveling speed is slow, the time interval for acquiring an image is lengthened, and when the traveling speed is high, the time interval for acquiring an image is shortened. By changing the image acquisition interval according to the speed, the structure can be captured by a plurality of still images regardless of the speed, and the color of the structure can be reliably determined.

Then, the image acquisition unit 11 executes the image correction process (S702). Specifically, the shape of the image is shaped. The image acquired by the in-vehicle camera 51 may be distorted, and the image is formed by linear transformation or projective transformation. In addition, the brightness of the image is adjusted by adjusting the exposure to facilitate the detection of colors.

After that, the map information acquisition unit 13 acquires the map information. The map information acquired by the map information acquisition unit 13 includes at least the position of the intersection. Then, the map information acquisition unit 13 extracts information on traffic lights, road signs, pedestrian crossings, and road markings (paints) included in the acquired map information (S703). When detailed map information is acquired, the positions of structures such as traffic lights, road signs, pedestrian crossings, and road markings (paint) can be known. On the other hand, since the navigation map information does not include the position of the structure, the area where the structure exists in the vicinity of the intersection such as a traffic light, a road sign, a pedestrian crossing, and a road marking can be known.

After that, the map information acquisition unit 13 determines from the acquired map information whether or not there is a sign or the like in the vicinity of the own vehicle (S704). When the detailed map information is acquired, the presence or absence of the structure can be determined depending on whether the structure information can be extracted from the detailed map information. On the other hand, when the map information for navigation is acquired, if the distance to the intersection is closer than a predetermined threshold, the intersection is approached, and generally, the structure of a traffic light, a road sign, a pedestrian crossing, a road marking, etc. near the intersection. It can be determined that the object is near the own vehicle.

In this embodiment, the processing is executed in the order of the processing by the image acquisition unit 11 and the processing by the map information acquisition unit 13, but either of them may be executed first or may be executed in parallel.

Then, when it is determined in step S704 that there is no structure in the vicinity of the own vehicle (No in S705), the process returns to step S701 and the next image data is acquired.

On the other hand, when it is determined in step S704 that there is a structure in the vicinity of the own vehicle (Yes in S705), the structure color specifying unit 16 refers to the data corresponding to the current environment of the structure color component table 17 and owns itself. The color of the structure determined to be in the vicinity of the vehicle is specified (S706). Since the lighting color is not known in the traffic light, all the lighting colors (and the extinguishing color if registered) are specified from the structure color component table 17. Further, when the type of the installed sign cannot be specified and the color of the sign to be recognized cannot be specified, all the possible colors may be specified from the structure color component table 17.

After that, the image area specifying unit 14 refers to the structure position table 15 and specifies a structure detection area in which it is presumed that a structure determined to be in the vicinity of the own vehicle exists in the image (S707).

In this embodiment, the structure color specifying process by the structure color specifying unit 16 and the image area specifying process by the image area specifying unit 14 are executed in this order, but whichever is executed first or in parallel is executed. good.

After that, the camera abnormality detection unit 18 determines whether or not the pixel of the color component of the structure determined to be in the vicinity of the own vehicle exists in the structure detection region (S708). Specifically, the pixels in the specified structure detection region are scanned, and it is determined whether or not a predetermined number of pixels satisfying the color condition of the structure specified in step S706 are continuously present.

When determining whether or not there are color components of multiple structures, it is preferable to determine in order of ease of color recognition. For example, a light emitting traffic light can easily recognize a color, so it is preferable to recognize the color first. Next, it is advisable to recognize the white paint on the road (for example, a pedestrian crossing) and then the sign.

Then, when the color of the structure is detected in the structure detection area, the continuous counter is initialized to 0, the abnormal area notified table 20 is cleared (S709), the process returns to step S701, and the next image data is input. get. On the other hand, if the color of the structure is not detected in the structure detection region, the continuous counter is added by 1 (S710), and it is determined whether the number of continuous counters is a predetermined number or more (S711). As a result, if the continuous counter does not exceed a predetermined threshold value, the process returns to step S701 without determining that there is an abnormality, and the next image data is acquired.

On the other hand, if the continuous counter exceeds a predetermined threshold value, the camera abnormality detection unit 18 determines that the camera abnormality is due to the fact that the structure that should exist is not shown in the image. Then, the abnormal area notified determination unit 19 refers to the abnormal area already notified table 20 and determines whether the determined abnormality has been notified (S712).

As a result, if the determined abnormality is registered in the abnormality area already notified table 20, the dirt abnormality of the camera has been notified (Yes in S713). Therefore, the process returns to step S701 without further notifying the abnormality, and the next step is performed. Get the image data of.

On the other hand, if the determined abnormality is not registered in the abnormality area already notified table 20, the dirt abnormality of the in-vehicle camera 51 is not notified (No in S713), so it is determined that the camera dirt abnormality is newly detected. Then, the abnormality notification unit 21 notifies the occupant and other ECUs of the camera dirt abnormality (S714). The occupant sees the camera dirt abnormality warning displayed on the instrument panel and removes the dirt on the front surface of the camera (for example, the lens). Further, when the ECU receives the camera dirt abnormality notification, it determines that the reliability of the image of the vehicle-mounted camera 51 is low, and the vehicle control process is performed without using the image of the vehicle-mounted camera 51 until the dirt is removed. To execute. For example, the ECU may stop the control of the automatic driving (AD) using the image of the vehicle-mounted camera 51 and notify the occupant of the stop of the automatic driving. Further, the ECU stops the system including the vehicle-mounted camera 51 in which the dirt is detected when the camera dirt abnormality is detected, and stops the control of the automatic driving due to the decrease in the redundancy, so that the advanced driver assistance (ADAS) You may switch to.

In the above description, it is determined whether or not the color component of the structure exists by using the image as one area. However, the image is divided into a plurality of areas (for example, 16 pieces of 4 vertical × 4 horizontal), and in each area. It may be determined whether the color component of the structure is present. In this case, a continuous counter and an abnormal region notified table 20 are provided for each region, and in step S708, it is determined whether or not the pixel of the color component of the structure exists in the structure detection region for each divided region. In steps S709 and S710, the continuous counter and the abnormal area notified table 20 are updated for each area. By determining whether or not the color component of the structure exists in each region of the image divided in this way, it is possible to identify not only the foreign matter adhered to the front surface of the vehicle-mounted camera 51 but also the region to which the foreign matter adheres. ..

Next, the camera abnormality detection process from the vehicle information, which is executed when the map information cannot be acquired, will be described. In the camera abnormality detection process (FIG. 8) from the own vehicle information, the same processing steps as the camera abnormality detection process (FIG. 7) from the map information described above are designated by the same reference numerals, and detailed explanations of those steps are omitted. do.

In the camera abnormality detection process from the own vehicle information, first, the image acquisition unit 11 acquires the image data taken by the in-vehicle camera 51 (S701) and executes the image correction process (S702).

After that, the own vehicle information acquisition unit 12 acquires the own vehicle information 41 (S803). The vehicle information 41 acquired by the vehicle information acquisition unit 12 includes speed, acceleration, steering angle, occupant operation amount (brake operation amount, accelerator operation amount), and the like as vehicle behavior information.

Then, the own vehicle information acquisition unit 12 estimates whether there is a vehicle in front traveling in front of the own vehicle or a signal in the vicinity of the own vehicle based on the acquired own vehicle information 41 (S804). Specifically, when low-speed driving and stopping are repeated, it can be estimated that there is a traffic jam and there is a vehicle in front. In addition, when the vehicle decelerates from constant speed and stops, it can be estimated that the signal is approaching an intersection.

After that, the structure color specifying unit 16 specifies the color of the structure determined to be in the vicinity of the own vehicle by referring to the data corresponding to the current environment of the structure color component table 17 (S706). After that, the image region specifying unit 14 refers to the structure position table 15 and identifies a structure detection region in which it is presumed that a structure determined to be in the vicinity of the own vehicle exists in the image (S707). For example, when it is estimated that there is a vehicle in front, the area of the vehicle in front is set as a structure detection area, and the red color of the tail lamp in the structure detection area is detected. Further, when it is estimated that the vehicle is approaching an intersection, the color of each structure is detected in the structure detection area of the traffic light, the road sign, and the pedestrian crossing in the same manner as described above.

After that, the camera abnormality detection unit 18 determines whether or not the pixel of the color component of the structure determined to be in the vicinity of the own vehicle exists in the structure detection region (S708). When the color of the structure is detected in the structure detection area, the continuous counter is initialized to 0, the abnormal area notified table 20 is cleared (S709), the process returns to step S701, and the next image data is acquired. .. On the other hand, if the color of the structure is not detected in the structure detection region, the continuous counter is added by 1 (S710), and it is determined whether the number of continuous counters is a predetermined number or more (S711). As a result, if the continuous counter does not exceed a predetermined threshold value, the process returns to step S701 without determining that there is an abnormality, and the next image data is acquired.

On the other hand, if the continuous counter exceeds a predetermined threshold value, the predetermined color component cannot be continuously detected. Therefore, the abnormal area notified determination unit 19 refers to the abnormal area notified table 20 and determines the abnormality. Is determined (S712).

As a result, if the detected color component is registered in the abnormality area already notified table 20, the stain abnormality of the camera has been notified (Yes in S713), so the process returns to step S701 without further notifying the abnormality. Acquire the next image data.

On the other hand, if the detected color component is not registered in the abnormal region notified table 20, the abnormal stain on the camera is not notified (No in S713), so that the abnormal region notified determination unit 19 newly determines that the camera is dirty. After determining that an abnormality has been detected, the abnormality notification unit 21 notifies the occupant and other ECUs of the camera dirt abnormality (S714).

As described above, the image may be divided into a plurality of regions (for example, 16 vertical × 4 horizontal regions), and it may be determined whether or not the color component of the structure is present in each region.

As described above, the image processing device 1 of the embodiment of the present invention is based on the image acquisition unit 11 that acquires an image from the in-vehicle camera 51 and the type of the structure that is presumed to exist around the own vehicle. , The structure color specifying unit 16 that specifies the color information of the structure, the image area specifying unit 14 that specifies the area in which the structure is presumed to exist in the image based on the type of the structure, and the in-vehicle camera. A camera abnormality detecting unit 18 for detecting an abnormality of 51 is provided, and when a structure exists around the own vehicle, the camera abnormality detecting unit 18 includes color information obtained from the structure color specifying unit 16 and an image. Since the abnormality of the vehicle-mounted camera 51 is detected by comparing with the color component of the region where the structure is presumed to exist, the abnormality of the vehicle-mounted camera 51 can be detected with a low processing load. That is, since the presence or absence of the color component of the structure is determined, the processing load can be reduced and the detection can be facilitated by reducing the amount of information used for the determination. In addition, it is possible to easily implement the detection of deposits on the front surface (lens) of the camera, which causes performance deterioration, even in a system that does not require high processing performance. Further, by using a structure having a common color such as a signal or a sign, it becomes easy to set the color condition of the structure based on the type of the structure.

Further, since the image area specifying unit 14 acquires the information of the structure presumed to exist around the own vehicle from the map information, the image is based on the information of the traffic light, the road sign, and the pedestrian crossing included in the detailed map. The area where the structure is presumed to exist can be accurately identified. In addition, based on the position of the intersection included in the navigation map, it is possible to accurately identify the area where structures such as traffic lights, road signs, and pedestrian crossings are presumed to exist in the image. Therefore, it can be easily determined whether or not the color of the structure to be reflected in the image exists.

Further, since the own vehicle information acquisition unit 12 estimates the information of the structure estimated to exist around the own vehicle from the state of the own vehicle, the preceding vehicle is used as the structure and the vehicle is mounted on the vehicle with a low processing load. The abnormality of the camera 51 can be detected. Moreover, even if the map information cannot be obtained, the approach to the intersection can be detected.

Further, since the own vehicle information acquisition unit 12 estimates the state of the vehicle from the vehicle speed, it can detect the presence or absence of a vehicle in front and the approach to an intersection even when the map information cannot be acquired.

Further, since the structure position table 15 holds a region in which the structure is estimated to exist corresponding to the type of the structure, it is estimated that the image area specifying unit 14 has a structure corresponding to the type. Area can be easily obtained.

Further, the structure color specifying unit 16 is a vehicle-mounted camera when the color information obtained from the structure color specifying unit 16 cannot be detected in a region where the structure is presumed to exist in the image for a predetermined number of times in succession. Detects 51 anomalies. That is, since it is determined that the lens in the region where a predetermined color cannot be continuously detected has stains that hinder the shooting of the outside world, temporary non-detection can be eliminated and the abnormality of the in-vehicle camera 51 can be reliably detected.

Further, the abnormality area notification determination unit 19 notifies the occupant and / or another ECU when the camera abnormality detected by the camera abnormality detection unit 18 is a viewing notification, so that the occupant is informed that the front surface (lens) of the camera is dirty. Can be promoted to be removed. The ECU can execute the vehicle control process without using the image of the camera to ensure the safety.

Further, when a plurality of structures exist, by having a plurality of continuous counters associated with each structure and the structure detection area, it is possible to determine abnormality detection in a larger number of areas at the same time.

It should be noted that the present invention is not limited to the above-mentioned embodiment, but includes various modifications and equivalent configurations within the scope of the attached claims. For example, the above-described examples have been described in detail in order to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to those having all the described configurations. Further, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Further, the configuration of another embodiment may be added to the configuration of one embodiment. In addition, other configurations may be added / deleted / replaced with respect to a part of the configurations of each embodiment.

Further, each configuration, function, processing unit, processing means, etc. described above may be realized by hardware by designing a part or all of them by, for example, an integrated circuit, and the processor realizes each function. It may be realized by software by interpreting and executing the program to be executed.

Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

Also, the control lines and information lines show what is considered necessary for explanation, and do not necessarily show all the control lines and information lines necessary for mounting. In practice, it can be considered that almost all configurations are interconnected.

Claims (11)

An image processing device that processes images taken by an in-vehicle camera.
An image acquisition unit that acquires an image from the in-vehicle camera,
A structure color specifying part that specifies color information of the structure based on the type of structure that is presumed to exist around the own vehicle.
An image area specifying portion that specifies a region in which the structure is presumed to exist in the image based on the type of the structure.
It is equipped with a camera abnormality detection unit that detects abnormalities in the in-vehicle camera.
When the structure is present around the own vehicle, the camera abnormality detecting unit includes color information obtained from the structure color specifying unit and a region in which the structure is presumed to exist in the image. An image processing device characterized by detecting an abnormality in the in-vehicle camera by comparing it with a color component. The image processing apparatus according to claim 1.
The image area specifying unit is an image processing device characterized in that information on a structure presumed to exist around the own vehicle is acquired from map information. The image processing apparatus according to claim 1.
An image processing device having an own vehicle information acquisition unit that estimates information on a structure presumed to exist around the own vehicle from the state of the own vehicle. The image processing apparatus according to claim 3.
The own vehicle information acquisition unit is an image processing device characterized in that the state of the vehicle is estimated from the vehicle speed. The image processing apparatus according to claim 1.
Corresponding to the type of the structure, the structure position information indicating the region where the structure is presumed to exist is retained, and the structure position information is retained.
The image region specifying unit is an image processing apparatus that refers to the structure position information and identifies a region in which the structure is presumed to exist in the image. It is an image processing method in which an electronic control device processes an image taken by an in-vehicle camera.
The electronic control device includes an arithmetic unit that executes a predetermined arithmetic processing and a storage device that stores data necessary for the arithmetic processing.
The image processing method is
An image acquisition procedure in which the arithmetic unit acquires an image from the in-vehicle camera,
A structure color specifying procedure for specifying the color information of the structure based on the type of the structure presumed to exist around the own vehicle by the arithmetic unit.
An image area specifying procedure in which the arithmetic unit identifies a region in the image where the structure is presumed to exist based on the type of the structure.
The arithmetic unit includes a camera abnormality detection procedure for detecting an abnormality in the vehicle-mounted camera.
In the camera abnormality detection procedure, when the arithmetic device has the structure around the own vehicle, the color information obtained in the structure color specifying procedure and the structure are present in the image. An image processing method characterized by detecting an abnormality in the in-vehicle camera by comparing it with a color component in an estimated region. The image processing method according to claim 6.
In the image area specifying procedure, the image processing method is characterized in that the arithmetic unit acquires information on a structure presumed to exist around the own vehicle from map information. The image processing method according to claim 6.
An image processing method, wherein the arithmetic unit includes a vehicle information acquisition procedure for estimating information on a structure presumed to exist around the vehicle from the state of the vehicle. The image processing method according to claim 8.
In the own vehicle information acquisition procedure, the image processing method is characterized in that the arithmetic unit estimates the state of the vehicle from the vehicle speed. The image processing method according to claim 6.
The storage device holds structure position information indicating a region where the structure is presumed to exist, corresponding to the type of the structure.
The image processing method, characterized in that, in the image region specifying procedure, the arithmetic unit refers to the structure position information to specify a region in the image where the structure is presumed to exist. It is an in-vehicle electronic control device.
An arithmetic unit that executes predetermined arithmetic processing and
It is equipped with a storage device for storing data necessary for the arithmetic processing.
An image acquisition unit in which the arithmetic unit acquires an image from the in-vehicle camera, and
The arithmetic unit has a structure color specifying unit that specifies color information of the structure based on the type of the structure that is presumed to exist around the own vehicle.
An image area specifying unit that specifies a region in which the structure is presumed to exist in the image based on the type of the structure by the arithmetic unit.
The arithmetic unit has a camera abnormality detecting unit that detects an abnormality of the vehicle-mounted camera.
When the structure is present around the own vehicle, the camera abnormality detecting unit includes color information obtained from the structure color specifying unit and a region in which the structure is presumed to exist in the image. An in-vehicle electronic control device characterized by detecting an abnormality in the in-vehicle camera by comparing it with a color component.

Images