JP6926976B2 - Parking assistance device and computer program - Google Patents

Description

The present invention relates to a parking assist device and a computer program that assist in parking a vehicle.

In recent years, various technologies related to parking support that automatically performs a part or all of parking operations in a parking lot have been proposed. In such a technology related to parking assistance, for example, it is possible to automatically perform only the steering operation for parking, and it is also possible to automatically perform the braking operation and the accelerator operation. Especially in recent years, as the technology related to automatic driving of vehicles has improved, in order to further reduce the burden on the user, in addition to driving on public roads, driving for warehousing or leaving in the parking lot is also performed by automatic driving. Has been proposed. In such automatic driving, for example, traveling to an empty parking space, entering an empty parking space, exiting from a parking space, traveling toward a parking lot exit, and the like can be automatically performed.

When implementing the various parking assistance described above, it is important for the vehicle to accurately detect the parking space that is vacant in the parking lot. Here, as one of the means for detecting the vacant parking space on the vehicle side, there is a method of detecting based on the captured image captured by an imaging device such as a camera installed in the vehicle. For example, in Japanese Patent Application Laid-Open No. 2015-170137, an image captured in a parking lot is processed by a camera mounted on a vehicle, the edge direction of an edge pixel included in the image is detected, and the edge direction is within the determination region. There has been proposed a technique for determining a parking space as an empty state when the number of edge pixels in a predetermined direction is less than the threshold value.

Japanese Unexamined Patent Publication No. 2015-170137 (pages 7-8, FIG. 6)

However, in the technique of Patent Document 1, the determination region for counting the number of edge pixels is set near the tip of the parking space as disclosed in FIGS. 6 and 15 of Patent Document 1. Therefore, for example, when a small vehicle is parked in the back side of the parking space, the number of edge pixels detected in the determination area is reduced, and there is a risk that the vehicle may be erroneously determined to be in an empty state.

Further, in the technique of Patent Document 1, since the determination is made by the point on the two-dimensional image, when there is an obstacle such as a beam above the parking space, or when the vehicle is parked on the left and right of the parking space. When a part of the vehicle extends beyond the boundary line, it is determined that the vehicle is not in an empty state because the number of edge pixels detected in the determination area increases even though there is sufficient space for the vehicle to park. There is also a problem that it ends up.

The present invention has been made to solve the above-mentioned conventional problems, and provides a parking support device and a computer program capable of accurately detecting an empty parking space around the current position of a vehicle. The purpose is.

In order to achieve the above object, the parking support device according to the present invention includes an image acquisition means for continuously capturing images captured by an image pickup device installed in the vehicle around the vehicle and the continuously captured images. Based on this, for a plurality of feature points around the vehicle, the three-dimensional position coordinates relative to the initial position of the imaging device when the continuous imaging of the continuous captured image is started for each of the plurality of feature points is set in a three-dimensional space. The feature point identifying means specified above and the position coordinates of the plurality of feature points specified by the feature point identifying means are corrected to three-dimensional position coordinates indicating the position of the parking lot where the vehicle is currently located with respect to the map. A parking space that detects an empty parking space around the current position of the vehicle based on the coordinate correction means and the arrangement mode of the plurality of feature points on the map of the parking lot after the position coordinates are corrected. It has a detection means.

Further, the computer program according to the present invention is a computer program for assisting parking of a vehicle in a parking lot. Specifically, the computer is located in the vicinity of the vehicle based on an image acquisition means for continuously capturing an image captured by an imaging device installed in the vehicle around the vehicle as a continuously captured image and the continuously captured image. For a plurality of feature points, feature point identification that specifies, in the three-dimensional space, three-dimensional position coordinates relative to the initial position of the imaging device when the continuous imaging of the continuous captured image is started for each of the plurality of feature points. Means, coordinate correction means for correcting the position coordinates of the plurality of feature points specified by the feature point specifying means to three-dimensional position coordinates indicating the position of the parking lot where the vehicle is currently located with respect to the map, and the position. After correcting the coordinates, it functions as a parking space detecting means for detecting an empty parking space around the current position of the vehicle based on the arrangement mode of the plurality of feature points on the map of the parking lot. Let me.

According to the parking support device and the computer program according to the present invention having the above configuration, a three-dimensional point cloud of relative feature points specified based on images continuously captured by an image pickup device installed in a vehicle is obtained. Since the vacant parking space is detected by associating it with the map of the actual parking lot, it is possible to accurately detect the vacant parking space around the current position of the vehicle. In particular, since it is possible to determine the vacancy state of the parking space three-dimensionally, when there is an obstacle only in a part of the parking space, or when there is an obstacle above the parking space, etc. Compared to the above, it is possible to accurately determine whether or not the parking space allows the vehicle to be parked.

It is a block diagram which showed the navigation device which concerns on this embodiment. It is a figure which showed an example of the structure of a parking lot. It is a flowchart of the parking support processing program which concerns on this embodiment. It is a figure which showed the feature point included in the captured image. It is a figure which showed the coordinate system of the 3D space in which the 3D point cloud generated by vSLAM is arranged. It is a figure which showed an example of the 3D point cloud generated by vSLAM. It is a figure which showed the range where the end of the boundary of a parking space is located. It is a figure which showed an example of the final 3D point cloud after correction. It is a figure which showed the arrangement mode of the final 3D point cloud after correction.

Hereinafter, an embodiment in which the parking support device according to the present invention is embodied in the navigation device 1 will be described in detail with reference to the drawings. First, the schematic configuration of the navigation device 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a navigation device 1 according to the present embodiment.

As shown in FIG. 1, the navigation device 1 according to the present embodiment includes a current position detection unit 11 that detects the current position of a vehicle on which the navigation device 1 is mounted, a data recording unit 12 in which various data are recorded, and a data recording unit 12. A navigation ECU 13 that performs various arithmetic processes based on the input information, an operation unit 14 that receives operations from the user, and a liquid crystal display that displays a map around the vehicle, a guidance screen for parking support, etc. to the user. 15, a speaker 16 that outputs voice guidance regarding route guidance, guidance regarding parking support, etc., a DVD drive 17 that reads a DVD as a storage medium, a probe center, a VICS (registered trademark: Vehicle Information and Communication System) center, etc. It has a communication module 18 that communicates with the information center of the above. Further, the navigation device 1 is connected to an external camera 19 and various sensors installed for the vehicle on which the navigation device 1 is mounted via an in-vehicle network such as CAN. Further, it is also connected to the vehicle control ECU 20 that performs various controls on the vehicle on which the navigation device 1 is mounted so as to be capable of bidirectional communication. In addition, various operation buttons 21 mounted on the vehicle such as the parking support start button are also connected.

Hereinafter, each component of the navigation device 1 will be described in order.
The current position detection unit 11 includes a GPS 22, a vehicle speed sensor 23, a steering sensor 24, a gyro sensor 25, and the like, and can detect the current position, direction, vehicle running speed, current time, and the like of the vehicle. .. Here, in particular, the vehicle speed sensor 23 is a sensor for detecting the moving distance and the vehicle speed of the vehicle, generates a pulse according to the rotation of the driving wheel of the vehicle, and outputs the pulse signal to the navigation ECU 13. Then, the navigation ECU 13 calculates the rotation speed and the moving distance of the drive wheels by counting the generated pulses. It is not necessary for the navigation device 1 to include all of the above four types of sensors, and the navigation device 1 may include only one or a plurality of of these sensors.

Further, the data recording unit 12 reads a hard disk (not shown) as an external storage device and a recording medium, a map information DB 31 and a parking lot map DB 32 recorded on the hard disk, a predetermined program, and the like, and also reads predetermined data on the hard disk. It is equipped with a recording head (not shown) that is a driver for writing data. The data recording unit 12 may have a flash memory, a memory card, or an optical disk such as a CD or DVD instead of the hard disk. Further, the map information DB 31 and the parking lot map DB 32 may be stored in an external server and acquired by the navigation device 1 by communication.

Here, the map information DB 31 is, for example, link data related to a road (link), node data related to a node point, search data used for processing related to route search or change, facility data related to a facility, and a map for displaying a map. It is a storage means that stores display data, intersection data for each intersection, search data for searching a point, and the like.

On the other hand, the parking lot map DB 32 is a storage means for storing information on maps of each parking lot in the whole country. The parking lot map stored in the parking lot map DB 32 is basically a two-dimensional map, but a three-dimensional map may be used. The map of the parking lot includes information on passages and parking spaces arranged in the parking lot (for example, position, shape, slope, etc.) in addition to the overall shape of the parking lot. In particular, the information for specifying the parking space includes information for specifying the position and shape of the boundary (for example, a white line or a wall) that divides the parking space, and the width of each parking space. However, the width of the parking space may not be included in the parking lot map DB 32 and may be calculated from the position of the boundary of the parking space.

In addition, the parking lot map DB 32 also includes information for identifying the parking space determination area 35 in the parking lot. Here, the parking space determination area 35 is an area in the parking lot where the vehicle is particularly targeted for detecting an empty parking space, and as shown in FIG. 2, on the passage through which the vehicle can pass in the parking lot 36. However, the area in which the parking operation for entering any of the parking spaces 37 can be started is applicable. For example, the passage in contact with the entrance of the parking space 37 is the parking space determination area 35. Then, the navigation device 1 continuously detects the vacant parking space while the vehicle is located in the parking space determination area 35 as described later.

Each position of the parking lot area, the passage in the parking lot, the parking space determination area 35, the parking space, and the boundary is specified by the latitude and longitude and stored in the parking lot map DB 32. Further, the navigation device 1 may store the map of all the parking lots in the parking lot map DB 32 in advance, or obtains the map of the parking lot entering from an external server when the vehicle enters the parking lot by communication. However, it may be configured to be stored in the parking lot map DB 32.

On the other hand, the navigation ECU (electronic control unit) 13 is an electronic control unit that controls the entire navigation device 1, and is a computing device, a CPU 41 as a control device, and a working memory when the CPU 41 performs various arithmetic processes. In addition to being used as a RAM 42 for storing route data and the like when a route is searched, a control program, and a ROM 43 and a ROM 43 in which a parking support processing program (see FIG. 3) described later is recorded are read. It is provided with an internal storage device such as a flash memory 44 for storing the program. The navigation ECU 13 has various means as a processing algorithm. For example, the image acquisition means acquires captured images continuously captured by an external camera 19 installed in the vehicle as a continuous captured image. Based on the continuously captured images, the feature point identifying means is relative to the initial position of the external camera 19 when the continuous imaging of the plurality of feature points around the vehicle is started for each of the plurality of feature points. Specify the position coordinates of the dimension in the three-dimensional space. The coordinate correction means corrects the position coordinates of the plurality of feature points specified by the feature point specifying means into three-dimensional position coordinates indicating the position of the parking lot where the vehicle is currently located with respect to the map. After correcting the position coordinates, the parking space detecting means detects an empty parking space around the current position of the vehicle based on the arrangement mode of the plurality of feature points on the map of the parking lot.

The operation unit 14 is operated when inputting a starting point as a traveling start point and a destination as a traveling end point, and is composed of a plurality of operation switches (not shown) such as various keys and buttons. Then, the navigation ECU 13 controls to execute various corresponding operations based on the switch signals output by pressing each switch or the like. The operation unit 14 may have a touch panel provided on the front surface of the liquid crystal display 15. It may also have a microphone and a voice recognition device.

In addition, the liquid crystal display 15 displays a map image including roads, traffic information, operation guidance, operation menus, key guidance, guidance information along the guidance route, news, weather forecast, time, mail, TV program, and the like. NS. Further, in the present embodiment, when the parking support described later is performed, the guidance regarding the parking support is also displayed. A HUD or HMD may be used instead of the liquid crystal display 15.

Further, the speaker 16 outputs voice guidance for guiding the traveling along the guidance route and traffic information guidance based on the instruction from the navigation ECU 13. Further, in the present embodiment, when the parking support described later is performed, the voice guidance regarding the parking support is also output.

Here, in the present embodiment, as parking assistance when the vehicle is parked, the vehicle automatically parks in the parking space along the route set on the navigation device 1 side regardless of the driving operation of the user. Automatic parking is possible. It should be noted that the user can select whether to park manually or by the above-mentioned automatic parking. Therefore, in the present embodiment, while the vehicle is traveling in the parking space determination area 35 (see FIG. 2) in the parking lot described above, the navigation device 1 is in addition to the current position of the vehicle and is an empty parking space around the own vehicle. Is detected at any time. Then, when a vacant parking space is detected and the user selects automatic parking, an approach route to the vacant parking space is set, and the vacant parking space is set along the set route. Vehicle control such as steering, drive source, and brake is automatically performed so as to enter the vehicle. It should be noted that only the steering may be controlled, and the accelerator, shift position, and brake may be operated by the user. In addition, only the guidance of parking operation for parking in an empty parking space (for example, steering operation position, operation angle, stop, start) is performed by the navigation device 1, and all operations related to the vehicle (including steering operation) are performed on the user side. It is also possible to let it be done at.

Further, the vehicle may be driven by automatic driving not only during parking but also during normal driving. In that case, if the destination parking lot is set for the navigation device 1, it is possible to automatically drive from the start of traveling at the departure point to the parking in the parking space in the destination parking lot. Is.

The DVD drive 17 is a drive capable of reading data recorded on a recording medium such as a DVD or a CD. Then, based on the read data, music and video are reproduced, the map information DB 31 is updated, and the like. A card slot for reading and writing a memory card may be provided instead of the DVD drive 17.

Further, the communication module 18 is a communication device for receiving traffic information, probe information, weather information, etc. transmitted from a traffic information center, for example, a VICS center, a probe center, etc., and corresponds to, for example, a mobile phone or a DCM. .. It also includes a vehicle-to-vehicle communication device that communicates between vehicles and a road-to-vehicle communication device that communicates with a roadside unit.

Further, the vehicle exterior camera 19 is composed of a camera using a solid-state image sensor such as a CCD, and is mounted above the front bumper of the vehicle or behind the rearview mirror, and is installed with the optical axis direction directed downward by a predetermined angle from the horizontal. Will be done. Then, the outside camera 19 images the front of the vehicle in the traveling direction when the vehicle travels in the parking lot. The external camera 19 may be a still camera that captures a still image, or may be a video camera that captures a moving image. However, in the case of a still camera, images are continuously captured at predetermined intervals. Further, the navigation device 1 detects an empty parking space around the vehicle as described later by performing image processing on the captured images (including moving images) continuously captured by the external camera 19. do. The external camera 19 may be arranged so as to be arranged behind or to the side of the vehicle in addition to the front of the vehicle.

Further, the vehicle control ECU 20 is an electronic control unit that controls the vehicle on which the navigation device 1 is mounted. Further, the vehicle control ECU 20 is connected to each drive unit of the vehicle such as steering, brake, accelerator, etc., and in the present embodiment, the above-mentioned is described by controlling each drive unit particularly after the parking support is started in the vehicle. Carry out automatic parking of vehicles.

Here, the navigation ECU 13 transmits an instruction signal regarding parking support to the vehicle control ECU 20 via the CAN after the parking support is started. Then, the vehicle control ECU 20 automatically parks in response to the received instruction signal. The content of the instruction signal is information for instructing the current position of the own vehicle, the position of the parking space to be parked, the approach route to the parking space, the start, stop, change, etc. of the parking support. The vehicle control ECU 20 may be configured to set the approach route instead of the navigation ECU 13. In that case, the vehicle control ECU 20 is configured to acquire information necessary for setting the approach route from the navigation device 1.

Subsequently, a parking support processing program executed by the CPU 41 in the navigation device 1 according to the present embodiment having the above configuration will be described with reference to FIG. FIG. 3 is a flowchart of the parking support processing program according to the present embodiment. Here, the parking support processing program is a program that is executed after the ACC power supply (accessory power supply) of the vehicle is turned on to support the vehicle when parking in the parking lot. The program shown in the flowchart in FIG. 3 below is stored in the RAM 42 or ROM 43 included in the navigation device 1 and is executed by the CPU 41.

First, in the driving support processing program, in step 1 (hereinafter abbreviated as S) 1, the CPU 41 acquires the current position of the vehicle detected by the current position detection unit 11. In detecting the current position of the vehicle, it is desirable to use various sensors such as GPS 22, the vehicle speed sensor 23, and the gyro sensor 25 in combination to detect the current position of the vehicle with higher accuracy. Further, the position of the vehicle acquired in S1 is the latitude / longitude (absolute coordinates).

Next, in S2, the CPU 41 compares the current position of the vehicle acquired in S1 with the map of the parking lot stored in the parking lot map DB 32, and determines whether or not the vehicle is located in any of the parking lots. judge. The parking lot map DB 32 stores information on latitude and longitude (for example, latitude and longitude of the four corners of the parking lot) that specify the area of each parking lot nationwide.

Then, when it is determined that the vehicle is located in any of the parking lots (S2: YES), the process proceeds to S3. On the other hand, when it is determined that the vehicle is not located in the parking lot (S2: NO), the parking support processing program is terminated without providing parking support for the vehicle.

In S3, the CPU 41 describes "(A) the position of the feature point around the vehicle" and "(B) the continuously captured image" based on the captured images (hereinafter referred to as "continuously captured images") continuously captured by the external camera 19. The positions of the external cameras 19 during the imaging of the image are specified, and the points indicating the specified positions are plotted (generated) in the same three-dimensional space. When the camera outside the vehicle 19 is a video camera, the continuously captured images do not necessarily have to be continuous frames, and may be images at regular frame intervals. The set of points generated in S3 in the three-dimensional space is hereinafter referred to as a three-dimensional point cloud. The processing of S3 will be described in more detail below.

Here, the "feature point" is a point on the image that can be detected conspicuously, and specifically, the corner of the luminance edge corresponds to the point. For example, as shown in FIG. 4, the captured image 51 captured by the outside camera 19 in the parking lot includes a large number of feature points 53. Corners of each part (bumper, license plate, side mirrors, tires, wheels, etc.) that make up the vehicle, corners of the bollard, edges of lines and letters drawn on the road surface, steps on the road surface, dirt on the road surface, etc. Corresponds to feature point 53. Further, in the present embodiment, by using a known method called vSLAM (visual SLAM), the feature point 53 around the vehicle is specified by three-dimensional position coordinates from the continuously captured images. However, the position coordinates specified by vSLAM are relative position coordinates, and specifically, they are specified by three-dimensional position coordinates relative to the initial position of the external camera 19 when the continuous imaging image is started to be captured. .. For example, as shown in FIG. 5, it is specified by a three-dimensional coordinate system based on the initial position of the external camera 19 when the continuous imaging image is started to be captured. Then, the CPU 41 plots (generates) the feature points 53 at the specified position coordinates. The unit of coordinates is dot. However, if the feature point 53 around the vehicle can be specified three-dimensionally, a method other than vSLAM may be used.

Further, in S3, in parallel with the specification of the position coordinates of the feature point 53, the position coordinates and the orientation (optical axis direction) of the external camera 19 during the continuous acquisition of the captured image using vSLAM are also specified. .. The position coordinates and orientation of the external camera 19 during the continuous acquisition of the captured image are specified by the three-dimensional position coordinates and angles relative to the initial position and initial orientation of the external camera 19 when the continuous imaging image is started to be captured. Will be done. In particular, the position coordinates of the outside camera 19 are specified by a three-dimensional coordinate system based on the initial position of the outside camera 19 when the continuous acquisition of the captured image is started as shown in FIG. 5 as in the feature point 53. .. Then, the position of the outside camera 19 is plotted (generated) on the specified position coordinates on the same three-dimensional space (three-dimensional coordinate system) as the feature point 53. The unit of coordinates is dot. The processing of S3 is continuously performed while the vehicle is located in the parking lot.

FIG. 6 is a diagram showing an example of the three-dimensional point cloud finally generated in S3. The light gray dots indicate the feature points 53 around the vehicle, and the black dots indicate the position 54 of the outside camera 19 while capturing the continuously captured images. In the example shown in FIG. 6, the feature point 53 within the imaging range of the outside camera 19 is specified along the α direction by moving the outside camera 19 (that is, the vehicle) in the α direction. The three-dimensional point cloud is arranged in the three-dimensional space, and the position coordinates are specified by the three-dimensional coordinate system based on the initial position of the external camera 19 when the continuous imaging image is started to be captured as described above. There is.

Subsequently, in S4, the CPU 41 compares the current position of the vehicle acquired in S1 with the map of the parking lot stored in the parking lot map DB 32, and the vehicle is located in the parking lot, particularly in the parking space determination area 35. Judge whether or not. As described above, the parking space determination area 35 corresponds to an area where a parking operation for entering any parking space in the parking lot can be started (see FIG. 2). The parking lot map DB 32 stores information on the latitude and longitude (for example, the latitude and longitude of the four corners of the parking space determination area 35) that specifies the parking space determination area 35 for each parking lot nationwide.

Then, when it is determined that the vehicle is located in the parking space determination area 35 in particular (S4: YES), the process proceeds to S5. On the other hand, when it is determined that the vehicle is located outside the parking space determination area 35 (S4: NO), the parking support processing program is terminated without providing parking assistance to the vehicle.

In S5, the CPU 41 calculates the current position of the external camera 19. Specifically, based on the latitude and longitude of the latest vehicle current position acquired in S1 and the information for identifying the installation position of the vehicle exterior camera 19 with respect to the vehicle, the current vehicle exterior camera 19 has three-dimensional absolute coordinates. Calculate the position. Specifically, (latitude, longitude, height (elevation)) is calculated. Information for specifying the installation position of the external camera 19 with respect to the vehicle is stored in the data recording unit 12 in advance. Further, in S5, the absolute orientation of the current external camera 19 is also calculated in the same manner.

Subsequently, in S6, the CPU 41 first reads out the parking space (hereinafter, referred to as a proximity parking space) closest to the current position of the external camera 19 calculated in S5 within the imaging range from the parking lot map DB 32. Identify based on the parking map. For example, when the vehicle 55 travels in front of the parking space as shown in FIG. 7, the parking space 56 closest to the position of the external camera 19 within the imaging range is the proximity parking space. Next, the CPU 41 has an arrangement interval between the boundary of one side (hereinafter referred to as the first boundary) and the boundary of the other side (hereinafter referred to as the second boundary) in the width direction of the specified proximity parking space, that is, "proximity parking space". Width ”is also obtained from the parking lot map DB32. The first boundary and the second boundary may be lines drawn on the road surface such as white lines, or may be structures such as walls and pillars. The parking lot map DB 32 stores information regarding the arrangement of parking spaces and the width of parking spaces in each parking lot and each parking space in advance. However, the width of the parking space may be calculated from the positions of the first boundary and the second boundary stored in the parking lot map DB 32.

After that, in S7, the CPU 41 specifies the range where the ends of the first boundary and the second boundary of the proximity parking space are located based on the parking lot map read from the parking lot map DB 32. Specifically, a range of a predetermined distance (for example, 30 cm) square centered on the ends of the first boundary and the second boundary is specified. For example, in the example shown in FIG. 7, the range 58 in which the end of the first boundary 57 on the left side in the width direction of the parking space 56, which is a close parking space, is located, and the end of the second boundary 59 on the right side are located. A range of 60 is specified, respectively. In S7, as in the case of the three-dimensional point cloud generated in S3, the corresponding range is specified by the three-dimensional position coordinates relative to the initial position of the external camera 19 when the continuous imaging image is started to be captured. It is desirable to do.

Next, in S8, the CPU 41 has a point cloud of feature points indicating the end of the first boundary of the proximity parking space and a feature point of the feature points of the second boundary of the three-dimensional point cloud generated in S3. Each point cloud is extracted. Specifically, a point cloud of feature points located within the range specified in S7 is extracted.

Subsequently, in S9, the CPU 41 first samples one feature point from the point cloud of the feature points indicating the end of the first boundary of the proximity parking space extracted in S8. Similarly, one feature point is sampled from the point cloud of the feature points indicating the end of the second boundary of the proximity parking space extracted in S8. Then, the distance between the position coordinates of the two sampled feature points (hereinafter referred to as the feature point coordinate distance) is calculated. As described above, the unit of the position coordinates of the feature points generated in S3 is dot, so the feature point coordinate intervals are also calculated by dot. Next, the CPU 41 compares the arrangement interval (width of the parking space) between the first boundary and the second boundary acquired in S6 with the coordinate interval of the feature points, and three-dimensionally arranges the three-dimensional point cloud. The scaling coefficient K [m / dot] for making the scale of space (more specifically, the scale of the coordinate system) correspond to the scale of the map of the parking lot where the vehicle is located is calculated. Specifically, assuming that the arrangement interval (width of the parking space) between the first boundary and the second boundary acquired in S6 is W [m] and the feature point coordinate interval is L [dot], the following equation (1) ) Calculates the scaling coefficient K [m / dot].
K = W / L ... (1)

After that, in S10, the CPU 41 parks at the position of the last specified (most recently specified) external camera 19 in the three-dimensional point cloud generated in S3 and the position closest to the position within the imaging range. The point cloud of the feature points indicating the end of the boundary of the space is scaled using the scaling coefficient K calculated in S9. Specifically, each value of the three-dimensional coordinates of each point cloud is multiplied by the scaling coefficient K, and the value after the multiplication is used as the three-dimensional coordinates of each scaled point cloud. The three-dimensional coordinates after scaling are the position coordinates in the three-dimensional space (that is, the three-dimensional coordinate system) of the scale corresponding to the scale of the map of the parking lot where the vehicle is located. The unit of the coordinates after scaling is meters [m], which is the actual distance corresponding to the scale of the map of the parking lot where the vehicle is located from the dot.

Further, since the position of the last specified (most recently specified) out-of-vehicle camera 19 in the three-dimensional point cloud generated in S3 corresponds to the current position of the out-of-vehicle camera 19 calculated in S5, The point cloud of the feature points indicating the end of the boundary of the parking space closest to the position within the imaging range is the same as the point cloud extracted in S8.

After that, in S11, the CPU 41 compares the position of the external camera 19 after being scaled in S10 with each coordinate value of the feature point indicating the end of the boundary of the parking space also after being scaled in S10. Calculate the difference in the height direction (vertical direction with respect to the road surface). Since the unit of the coordinates after scaling is meters [m], the unit of the calculated difference is also meters [m]. Here, the difference in the height direction of the calculated coordinate values corresponds to the height from the ground surface to the position of the external camera 19 calculated based on the scaled three-dimensional point cloud.

Next, in S12, the CPU 41 reads out a specified value (design value) [m] of the installation height of the external camera 19 from the ground surface from the data recording unit 12. After that, the height from the ground surface calculated in S11 to the position of the external camera 19 is compared with the specified value read from the data recording unit 12, and it is determined whether or not they match. In addition to the case where both values are completely the same, it is desirable to consider the case where the difference between the two values is small as a match in consideration of the error.

Then, when it is determined that the height from the ground surface calculated in S11 to the position of the external camera 19 and the specified value read from the data recording unit 12 match (S12: YES), the scaling calculated in S9 is performed. It is estimated that the coefficient K is correct, and the process proceeds to S14. On the other hand, when it is determined that the height from the ground surface to the position of the external camera 19 calculated in S11 and the specified value read from the data recording unit 12 do not match (S12: NO), it is calculated in S9. It is estimated that the scaling coefficient K is not correct, and the process proceeds to S13.

In S13, the CPU 41 changes the two feature points sampled from the two point clouds in S9 to other feature points in the same point cloud. After that, the process returns to S9, and the sampling coefficient is calculated based on the newly sampled feature points. Then, the processes of S9 to S13 are repeated until the height from the ground surface calculated in S11 to the position of the external camera 19 and the specified value read from the data recording unit 12 match.

On the other hand, in S14, the CPU 41 scales all the three-dimensional point clouds generated in S3 by using the scaling coefficient K calculated in S9. Specifically, each value of the 3D coordinates of all the 3D point clouds is multiplied by the scaling coefficient K, and the value after the multiplication is used as the 3D coordinates of the scaled 3D point cloud. The three-dimensional coordinates after scaling are the position coordinates in the three-dimensional space (that is, the three-dimensional coordinate system) of the scale corresponding to the scale of the map of the parking lot where the vehicle is located. The unit of the coordinates after scaling is meters [m], which is the actual distance corresponding to the scale of the map of the parking lot where the vehicle is located from the dot.

Subsequently, in S15, the CPU 41 determines that the position of the last specified (most recently specified) outside camera 19 in the three-dimensional point cloud after being scaled in S14 is the position of the outside camera 19 calculated in S5. The scaled 3D point cloud (including the camera position and feature points) is moved so as to match the current position. Further, the three-dimensional point cloud (camera position is also characterized) after scaling so that the orientation of the vehicle exterior camera 19 also last specified (most recently identified) matches the current orientation of the vehicle exterior camera 19 calculated in S5. (Including points) is rotated.

The movement and rotation of the three-dimensional point cloud are performed, for example, by converting each coordinate of the three-dimensional point cloud with a coordinate conversion matrix, and the converted coordinates are used as the coordinates of the three-dimensional point cloud after the movement and rotation. Here, the coordinate transformation matrix is derived based on each movement amount and rotation amount in the xyz direction. That is, in order for the CPU 41 to match the position of the last specified (most recently specified) out-of-vehicle camera 19 in the scaled three-dimensional point cloud with the current position of the out-of-vehicle camera 19 calculated in S5. The amount of movement in the xyz direction required for the above and the amount of rotation required to match the current orientation of the external camera 19 are calculated, and a coordinate transformation matrix is derived from each of the calculated values. The movement and rotation of the three-dimensional point cloud may be performed by a method other than using the coordinate transformation matrix.

The final position coordinates of the three-dimensional point cloud after being corrected by the processing of S5 to S15 described above are absolute three-dimensional positions indicating the position of the three-dimensional point cloud with respect to the map of the parking lot where the vehicle is currently located. It becomes the coordinates.

FIG. 8 is a diagram showing an example of the final three-dimensional point cloud after being modified by the processing of S5 to S15 described above. In particular, FIG. 8 shows a state in which the vehicle is combined with the map 61 of the parking lot where the vehicle is located.
The light gray dots indicate the feature points 53 around the vehicle, and the black dots indicate the position 54 of the outside camera 19 while capturing the continuously captured images. The position coordinates of the three-dimensional point cloud after being finally modified by the above-mentioned processes of S5 to S15 are absolute three-dimensional positions indicating the position of the three-dimensional point cloud with respect to the map of the parking lot where the vehicle is currently located. It becomes the coordinates. Therefore, as shown in FIG. 8, the feature point 53 of the three-dimensional point cloud is located at the actual point where the feature point is located on the map 61 of the parking lot, and the outside camera 19 is located during imaging on the map 61 of the parking lot. The position 54 of the vehicle exterior camera 19 of the three-dimensional point cloud is located at the actual point.

Subsequently, in S16, the CPU 41 performs the parking space (for example, from the current position of the external camera 19) to be detected in the vacant state in the final three-dimensional point cloud after being corrected by the processing of S5 to S15 described above. The number of point clouds of feature points included in the parking space closest to the imaging range) is counted. Specifically, the area of the parking space to be detected is specified by a two-dimensional coordinate system (xy coordinates) parallel to the road surface, and the number of feature points including the xy coordinates in the area is counted. However, when counting the number of feature points, it is desirable to target only the feature points whose height from the road surface is a predetermined distance (for example, 10 cm) or more. This is to eliminate the characteristic points indicating dirt and steps on the road surface. Further, in an indoor parking lot, it is desirable to target only feature points whose height from the road surface is a predetermined distance (for example, 2 m) or less. This is to eliminate the feature points that indicate the beams and lighting of the ceiling.

After that, in S17, the CPU 41 determines whether or not the number of point clouds of the feature points counted in S16 is less than the threshold value. The threshold value used as the determination criterion in S17 can be changed as appropriate, but is set to, for example, 10.

Then, when it is determined that the number of the point cloud of the feature points counted in S16 is less than the threshold value (S17: YES), the process proceeds to S18. On the other hand, when it is determined that the number of point clouds of the feature points counted in S16 is equal to or greater than the threshold value (S17: NO), the process proceeds to S20.

In S18, the CPU 41 determines that the parking space to be detected is in a vacant state (parkable state) without the presence of other vehicles or other obstacles.

After that, in S19, the CPU 41 provides parking support for the parking space determined to be vacant. For example, when the user selects automatic parking, an approach route to a parking space determined to be vacant from the current position of the vehicle is set. Further, by transmitting an instruction signal regarding parking assistance to the vehicle control ECU 20 via the CAN, vehicle control such as steering, drive source, and brake is performed so as to enter an empty parking space along a set route. Is done automatically. Further, for a vehicle that performs automatic driving, it is possible to start automatic parking without a condition of a user's instruction.

It should be noted that only the steering may be controlled, and the accelerator, shift position, and brake may be operated by the user. In addition, only guidance on parking operations (for example, steering operation position, operation angle, stop, start) for parking in a parking space determined to be vacant is performed, and all operations related to the vehicle (including steering operation) are performed by the user. It may be done on the side.

On the other hand, in S20, the CPU 41 determines that the parking space to be detected is in a non-vacant state (parking impossible state) due to the existence of other vehicles and other obstacles. Then, the parking support processing program is terminated without providing parking support. After that, as the vehicle continues to run in the parking lot, the detection range of the feature points and the parking space for which the vacant state is detected will gradually change. Then, when a vacant parking space is detected, parking support is provided for the parking space (S19).

Here, FIG. 9 is a diagram showing an arrangement mode when the final three-dimensional point cloud after being modified by the above-mentioned processing of S5 to S15 is visually recognized from the height direction (z-axis direction). In particular, FIG. 9 shows a state in which the vehicle is combined with the map 61 of the parking lot where the vehicle is located.
The light gray dots indicate the feature points 53 around the vehicle, and the black dots indicate the position 54 of the outside camera 19 while capturing the continuously captured images. In the example shown in FIG. 9, a large number of feature points 53 are arranged in the parking space 65 and the parking space 67. On the other hand, the feature points 53 are hardly arranged in the parking space 66. Therefore, the parking space 65 and the parking space 67 are determined to be "not vacant", while the parking space 66 is determined to be "vacant". Then, parking support is provided for the parking space 66 determined to be in the vacant state.

As described in detail above, in the navigation device 1 and the computer program executed by the navigation device 1 according to the present embodiment, the captured images of the periphery of the vehicle continuously captured by the vehicle exterior camera 19 are continuously captured by the vehicle exterior camera 19 based on the continuously captured images of the vehicle. For a plurality of feature points in the vicinity, the three-dimensional position coordinates relative to the initial position of the external camera 19 when the continuous imaging of the continuous captured image is started for each of the plurality of feature points are specified in the three-dimensional space (S3). ), The position coordinates of the specified plurality of feature points are corrected to three-dimensional position coordinates indicating the position of the parking lot where the vehicle is currently located with respect to the map (S5 to S15). Since the vacant parking space around the current position of the vehicle is detected (S16 to S18) based on the arrangement mode of the plurality of feature points on the map, the vacant parking space around the current position of the vehicle is accurately detected. It becomes possible to detect. In particular, since it is possible to determine the vacancy state of the parking space three-dimensionally, when there is an obstacle only in a part of the parking space, or when there is an obstacle above the parking space, etc. Compared to the above, it is possible to accurately determine whether or not the parking space allows the vehicle to be parked.

It should be noted that the present invention is not limited to the above-described embodiment, and it goes without saying that various improvements and modifications can be made without departing from the gist of the present invention.
For example, in the present embodiment, it is assumed that the application is applied to a vehicle having a function related to automatic parking in which a part or all of the vehicle operation related to parking is automatically performed, but the present embodiment is applied to a vehicle not having the function to perform automatic parking. It is also possible. In that case, for example, the vacancy state of the parking space determined in S18 and S20 is guided to the user.

Further, in the present embodiment, when calculating the scaling coefficient K, the arrangement interval of the boundary of the parking space is particularly used, but the arrangement interval of the feature other than the boundary of the parking space may be used. However, the arrangement interval is stored in the parking lot map DB 32, or is a feature that can be calculated from the parking lot map DB 32. For example, it may be the arrangement interval of the bollard.

Further, in the present embodiment, the parking support processing program (see FIG. 3) is executed by the navigation device 1 mounted on the vehicle, but an on-board unit other than the navigation device 1 or the vehicle control ECU 20 may execute the program. .. Further, the external server may perform a part or all of the processing of each step shown in FIG. 3, and the navigation device 1 may acquire only the processing result (empty state of the parking space) from the external server.

Further, although the embodiment in which the parking support device according to the present invention is embodied has been described above, the parking support device can also have the following configurations, and in that case, the following effects are obtained.

For example, the first configuration is as follows.
An image acquisition means (41) that continuously acquires an image captured by an imaging device (19) installed in the vehicle around the vehicle (55) as a continuous image, and the area around the vehicle based on the continuously captured image. For a plurality of feature points (53), the three-dimensional position coordinates relative to the initial position of the imaging device when the imaging of the continuous captured image is started for each of the plurality of feature points is specified in the three-dimensional space. The three-dimensional position coordinates indicating the positions of the feature point identifying means (41) and the plurality of feature points specified by the feature point identifying means with respect to the map (61) of the parking lot where the vehicle is currently located. After correcting the position coordinates, the vacant parking around the current position of the vehicle is based on the arrangement mode of the plurality of feature points on the map of the parking lot. According to the parking support device having the above configuration having the parking space detecting means (41) for detecting the space, the relative image identified based on the captured images continuously captured by the imaging device installed in the vehicle. Since the vacant parking space is detected by associating the three-dimensional point group of the feature points with the map of the actual parking lot, it is possible to accurately detect the vacant parking space around the current position of the vehicle. It will be possible. In particular, since it is possible to determine the vacancy state of the parking space three-dimensionally, when there is an obstacle only in a part of the parking space, or when there is an obstacle above the parking space, etc. Compared to the above, it is possible to accurately determine whether or not the parking space allows the vehicle to be parked.

The second configuration is as follows.
In the same three-dimensional space in which the feature point (53) is specified, the position of the image pickup device (19) during the acquisition of the continuously captured image is three-dimensional relative to the initial position of the image pickup device. It has an imaging position specifying means (41) specified by position coordinates and an absolute position acquiring means (41) for acquiring position coordinates indicating the current position of the imaging device with respect to the map (61) of the parking lot. The coordinate correction means (41) is the image pickup device specified by the image pickup position specifying means so that the position of the image pickup device last specified by the image pickup position specifying means coincides with the current position of the image pickup device. By integrally moving the position and the position of the feature point specified by the feature point specifying means, the position coordinates of the plurality of feature points are corrected.
According to the parking support device having the above configuration, the position coordinates of the feature points specified by the relative positional relationship with the image pickup device are referred to the current position of the image pickup device for which the absolute position coordinates are obtained. By converting to the coordinates, it is possible to correct the coordinates to the absolute position coordinates corresponding to the map of the actual parking lot.

The third configuration is as follows.
An imaging orientation specifying means (41) that specifies the orientation of the imaging apparatus (19) during imaging of the continuously captured image by an angle relative to the orientation at the initial position of the imaging apparatus, and the imaging apparatus. It has an absolute azimuth acquisition means (41) for acquiring the current azimuth by an absolute azimuth, and in the coordinate correction means, the azimuth of the imaging device last specified by the imaging azimuth specifying means is the current azimuth of the imaging device. By integrally rotating the position of the imaging device specified by the imaging position specifying means and the position of the feature point specified by the feature point specifying means so as to coincide with, the plurality of feature points Correct the position coordinates.
According to the parking support device having the above configuration, the position coordinates of the feature points specified by the relative positional relationship with the image pickup device, the current position of the image pickup device in which the absolute position coordinates and the orientation are obtained, and the current position of the image pickup device and the orientation are obtained. By converting the coordinates to the coordinates based on the orientation, it is possible to correct the coordinates to the absolute position coordinates corresponding to the map of the actual parking lot.

The fourth configuration is as follows.
The information acquisition means (41) for acquiring information on the arrangement interval between the first feature (57) and the second feature (59) around the vehicle, and the feature point identification means (41) identified by the information acquisition means (41). Of the plurality of feature points (53), the coordinate interval acquisition means (41) for acquiring the distance between the position coordinates of the feature point indicating the first feature and the feature point indicating the second feature, and the coordinates. The correction means (41) compares the arrangement interval acquired by the information acquisition means with the interval of the position coordinates acquired by the coordinate interval acquisition means, and maps the parking lot where the vehicle is located on the scale of the three-dimensional space (41). By making it correspond to the scale of 61), the position coordinates of the plurality of feature points are corrected.
According to the parking support device having the above configuration, it is specified by the relative positional relationship with the imaging device by comparing the arrangement interval of the feature with the arrangement interval of the corresponding feature points, and it is different from the actual scale. Can modify the position coordinates of feature points defined on different scales to the position coordinates based on the scale corresponding to the map of the actual parking lot.

The fifth configuration is as follows.
The first feature is the first boundary (57) indicating the boundary of one side in the width direction of the parking space, and the second feature is the boundary of the other side in the width direction of the parking space. In the boundary (59) of 2, the information acquisition means (41) acquires the width of the parking space as the arrangement interval between the first boundary and the second boundary, and the coordinate interval acquisition means (41). ) Acquires the distance between the position coordinates of the feature point indicating the end of the first boundary and the feature point indicating the end of the second boundary.
According to the parking support device having the above configuration, it is specified by the relative positional relationship with the image pickup device by comparing the arrangement interval of the boundary of the parking space and the arrangement interval of the corresponding feature points as a feature. Therefore, it is possible to correct the position coordinates of the feature points defined on a scale different from the actual scale to the position coordinates based on the scale corresponding to the map of the actual parking lot.

The sixth configuration is as follows.
The coordinate correction means (41) has a comparison means (41) for comparing the height from the road surface of the position of the image pickup device (19) specified by the image pickup position specifying means (41) with a specified value. The scale of the three-dimensional space is a scale in which the height of the position of the captured image from the road surface matches the specified value.
According to the parking support device having the above configuration, the scale is adjusted by using the position of the image pickup device, which can accurately acquire the height value from the road surface as a specified value, so that it corresponds to the map of the actual parking lot. It is possible to accurately correct the position coordinates based on the scale.

The seventh configuration is as follows.
The parking space detecting means (41) detects an empty parking space around the current position of the vehicle based on the arrangement mode of the feature points (53) whose height from the road surface is at least a predetermined distance.
According to the parking support device having the above configuration, it is possible to eliminate the feature points indicating dirt and steps on the road surface, and it is possible to accurately determine whether or not there is an obstacle in the parking space that hinders the parking of the vehicle. It becomes possible.

1 Navigation device 13 Navigation ECU
19 Outside camera 32 Parking lot map DB
35 Parking space judgment area 41 CPU
42 RAM
43 ROM
53 Feature points 54 Position of camera outside the vehicle 55 Vehicle 61 Map of parking lot

Claims (8)

An image acquisition means for acquiring an image taken continuously by an image pickup device installed in the vehicle around the vehicle as a continuous image, and an image acquisition means.
Based on the continuously captured images, with respect to the plurality of feature points around the vehicle, the three-dimensional positions relative to the initial position of the imaging device when the imaging of the continuously captured images is started for each of the plurality of feature points. A feature point identification means for specifying coordinates in a three-dimensional space,
Coordinate correction means for correcting the position coordinates of the plurality of feature points specified by the feature point specifying means to three-dimensional position coordinates indicating the position of the parking lot where the vehicle is currently located with respect to the map.
After correcting the position coordinates, a parking space detecting means for detecting an empty parking space around the current position of the vehicle based on the arrangement mode of the plurality of feature points on the map of the parking lot. Parking support device to have. Imaging in which the position of the imaging device during the imaging of the continuously captured image is specified by the three-dimensional position coordinates relative to the initial position of the imaging device in the same three-dimensional space in which the feature points are specified. Positioning means and
It has an absolute position acquisition means for acquiring position coordinates indicating the current position of the image pickup device with respect to the map of the parking lot.
The coordinate correction means
The position of the image pickup device and the feature point identification means specified by the image pickup position specifying means so that the position of the image pickup device finally specified by the image pickup position specifying means coincides with the current position of the image pickup device. The parking support device according to claim 1, wherein the position coordinates of the plurality of feature points are corrected by integrally moving the positions of the specified feature points. An imaging orientation specifying means for specifying the orientation of the imaging device during the acquisition of the continuously captured images by an angle relative to the orientation at the initial position of the imaging device .
It has an absolute azimuth acquisition means for acquiring the current azimuth of the image pickup apparatus by an absolute azimuth.
The coordinate correction means
The position of the imaging device and the feature point identifying means specified by the imaging position specifying means so that the orientation of the imaging device finally identified by the imaging orientation identifying means matches the current orientation of the imaging device. The parking support device according to claim 2, wherein the position coordinates of the plurality of feature points are corrected by integrally rotating the positions of the specified feature points. An information acquisition means for acquiring information on the arrangement interval between the first feature and the second feature around the vehicle, and
Coordinate interval acquisition means for acquiring the distance between the position coordinates of the feature point indicating the first feature and the feature point indicating the second feature among the plurality of feature points specified by the feature point specifying means. When,
The coordinate correction means compares the arrangement interval acquired by the information acquisition means with the interval of the position coordinates acquired by the coordinate interval acquisition means, and scales the three-dimensional space on the scale of the map of the parking lot where the vehicle is located. The parking support device according to claim 2 or 3, wherein the position coordinates of the plurality of feature points are corrected by corresponding to the above. The first feature is a first boundary indicating the boundary of one side in the width direction of the parking space.
The second feature is a second boundary indicating the boundary of the other side in the width direction of the parking space.
The information acquisition means acquires the width of the parking space as an arrangement interval between the first boundary and the second boundary.
The parking support device according to claim 4, wherein the coordinate interval acquisition means acquires the distance between the position coordinates of the feature point indicating the end of the first boundary and the position coordinates of the feature point indicating the end of the second boundary. It has a comparison means for comparing the height of the position of the imaging device specified by the imaging position specifying means from the road surface with a specified value.
The parking support device according to claim 4 or 5, wherein the coordinate correction means sets the scale of the three-dimensional space to a scale at which the height of the position of the captured image from the road surface matches the specified value. Any of claims 1 to 6, wherein the parking space detecting means detects an empty parking space around the current position of the vehicle based on an arrangement mode of feature points whose height from the road surface is a predetermined distance or more. Parking support device described in Crab. Computer,
An image acquisition means for acquiring an image taken continuously by an image pickup device installed in the vehicle around the vehicle as a continuous image, and an image acquisition means.
Based on the continuously captured images, with respect to the plurality of feature points around the vehicle, the three-dimensional positions relative to the initial position of the imaging device when the imaging of the continuously captured images is started for each of the plurality of feature points. A feature point identification means for specifying coordinates in a three-dimensional space,
Coordinate correction means for correcting the position coordinates of the plurality of feature points specified by the feature point specifying means to three-dimensional position coordinates indicating the position of the parking lot where the vehicle is currently located with respect to the map.
A parking space detecting means for detecting an empty parking space around the current position of the vehicle based on the arrangement mode of the plurality of feature points on the map of the parking lot after the correction of the position coordinates.
A computer program to make it work.

Images