JP5904917B2 - Terminal position and direction detection device of mobile terminal - Google Patents

Description

  The present invention relates to a terminal mounted in a moving body in a terminal mounted on a moving body that is moved by a human being, for example, a car, and that can be freely changed. The present invention relates to a technology for detecting the location and the orientation of the terminal with the orientation of the camera as the orientation of the terminal.

  Research and development of applications using AR technology (Augmented Reality, Augmented Reality) that superimposes information and images using a computer on video (actual environment) taken with a camera is actively conducted. Non-patent document 1 describes general AR technology. Furthermore, a method as disclosed in Patent Document 1 is disclosed as a method of knowing the distance and position from the camera that is shooting to the object being shot.

  Non-Patent Document 1 describes general AR technology. In AR technology, there are a marker method and a sensor method. In the marker method, a marker is required in a camera image photographed by a camera, but a desired image can be superimposed on a camera image photographed by a camera installed in a car as a moving body. On the other hand, in the sensor system, the direction and position of a camera installed on the ground can be obtained using a geomagnetic sensor or GPS (Global Positioning System), and images and images taken by a camera on the ground can be obtained. Other images can be superimposed.

  Patent Document 1 discloses a method for measuring the position and posture of an object as an assembly part from a robot hand when a product is assembled by a robot. In this method, the distance and posture from the object to the camera are measured by processing an image taken by the camera. That is, a geometric feature in the 3D shape model is calculated from the 3D shape model of the object held in advance in the terminal where the camera is installed, and the 2D image of the object captured by the camera The geometric features are made coincident with each other while correcting the difference between the feature points and line segments of the object, and the distance to the object in the image is obtained.

JP 2012-2761 A

Information Processing Society of Japan, “Special Issue Augmented Reality (AR)”, Information Processing 4, [2010] vol.51 No.4, Volume 542, pp. 367-434, published on April 15, 2010

Conventionally, in an AR technology sensor system, if a geomagnetic sensor or a GPS function is used, it is possible to superimpose another image on a video shot by a camera on the ground. However, in an environment where the moving body such as a car is moving and the direction and inclination of the space changes every moment, another image is superimposed on the video captured by the camera installed in the moving body. In this case, it is necessary to accurately grasp the “position and orientation in the moving body” of the camera, and to obtain an accurate positional relationship between the object in the moving body and the camera in the moving body. However, conventionally, there are devices that detect the position of the camera in a moving body such as the inside of the vehicle, for example, the coordinate value where the camera is placed on the coordinate axis with the center point in the vehicle as the base point, and the orientation of the camera. There wasn't.
The present invention has been made to solve the above-described problems, and an object thereof is to obtain the position of a camera mounted on a terminal within a moving body at high speed.

The terminal position and direction detection device of the mobile-equipped terminal in this invention,
A device that is placed on a moving body and detects the position and direction of the terminal in which the camera is installed in the moving body,
A terminal orientation detection unit that detects a relative orientation of the terminal in the moving body;
In-vehicle 3D model holding unit that holds a 3D model in the moving body that models the shape in the moving body in three dimensions,
A 2D model based on the orientation of the terminal in the moving body from the 3D model in the moving body held in the in-vehicle 3D model holding section based on the information on the relative direction in the moving body of the terminal received from the terminal orientation detection unit 2D conversion unit that cuts out
A camera image acquisition unit that acquires images of the moving body captured by the camera mounted on the terminal;
A camera image received from the camera image acquisition unit compares the 2D model received from the 2D conversion unit, an image comparing unit camera image to detect which parts of the two-dimensional model,
A comparison result is received from the image comparison unit, a distance between the camera position and an object photographed by the camera is acquired, and a distance determination unit from the object that acquires the placement position of the terminal in the moving body is provided.

  According to the terminal position and direction detection device for a mobile body-mounted terminal of the present invention, the terminal detects the angle between the direction of the mobile body and the horizontal direction of the mobile body from the mobile body on which the terminal is mounted. It is possible to obtain information, and from the information on the direction of the terminal and the inclination information of the terminal, the relative direction of the terminal in the mobile body and the calculation amount with a small amount of direction and inclination angle can be calculated, and the movement created in advance Cut out the 2D model based on the orientation of the terminal from the 3D model in the body and extract which part of the 2D model the image captured with the terminal camera corresponds to. The distance between the object and the camera can be calculated, and the position of the terminal in the moving body can be uniquely determined from the distance between the camera and the object and the direction of the camera. Thus, the effect that the position where the terminal is placed in the moving body can be detected with a small amount of calculation can be obtained.

It is a block diagram of the terminal position and direction detection apparatus of the mobile body mounting terminal in Embodiment 1 of this invention. It is a conceptual diagram of a 2D model created from a 3D model by a 2D conversion unit. It is a block diagram of the terminal position and direction detection apparatus of the mobile body mounting terminal in Embodiment 2 of this invention.

Embodiment 1 FIG.
FIG. 1 is a configuration diagram of a terminal position and direction detection device of a mobile unit-equipped terminal according to Embodiment 1 in which the present invention is applied to a car, and the flow of data processing is indicated by arrows.
In the figure, a terminal position and direction detection device of a mobile unit-equipped terminal is composed of a mobile unit on which a terminal is mounted, in this embodiment, a car 100 and a terminal 200, each having the following functional means.
The car 100 includes a vehicle information acquisition unit 101 that acquires information on the vehicle. The vehicle information acquisition unit 101 acquires vehicle information representing a vehicle state from each sensor that detects an accelerator or a brake, a gear, a steering wheel angle, a vehicle inclination, and the like in the vehicle.

The terminal 200 is equipped with a camera. The terminal 200 includes a sensor provided in the terminal 200, in this embodiment, a sensor information acquisition unit 201 that acquires information from an acceleration sensor, a gyro sensor, and a magnetic sensor;
A direction / tilt detection unit 202 of the terminal that detects a direction in which the camera mounted on the terminal 200 is facing or a tilt of the camera with respect to the ground based on information from the sensor information acquisition unit 201;
A vehicle direction confirmation unit 203 that obtains vehicle information (accelerator / brake, gear, steering wheel angle, inclination) from the vehicle information acquisition unit 101 and confirms forward (rear) advance and right / left turn of the vehicle;
A vehicle both-direction detection unit 204 is provided for detecting a vehicle traveling direction confirmed by the vehicle direction confirmation unit 203, that is, a forward (rear) direction of the vehicle and a traveling direction of the vehicle, that is, a right / left turn of the vehicle from the operation of the vehicle.

Further, the terminal 200 receives a direction in which a camera (hereinafter referred to as a terminal unless otherwise specified) facing the terminal 200 from the terminal direction / tilt detection unit 202, and receives the vehicle from the vehicle both-angle detection unit 204. Detecting the direction in which the vehicle is facing and detecting the difference in the horizontal direction of the direction in which the vehicle and the terminal are facing, that is, detecting the difference in the direction in which the terminal and the vehicle are facing, Part 205;
Information on the inclination of the vehicle with respect to the ground is received from the vehicle information acquisition unit 101, the angle of inclination with respect to the vertical direction of the terminal is received from the direction / inclination detection unit 202 of the terminal, and the vertical relative between the vehicle and the terminal is received. An angle detection unit 206 in the vertical direction with respect to the vehicle that detects a specific inclination, that is, a difference in inclination angle from each ground is provided.

In addition, the terminal 200 receives a relative angle difference between the vehicle and the terminal in the horizontal direction from the angle detection unit 205 in the horizontal direction with the vehicle, and receives the vehicle and the terminal from the angle detection unit 206 in the vertical direction with respect to the vehicle. And a terminal orientation detection unit 207 that detects the relative vertical and horizontal orientations of the vehicle with respect to the terminal.
In the present invention, the orientation of the camera mounted as described above and the orientation of the display screen are the same, and the orientation of the camera is the orientation of the terminal.
Further, in this embodiment, the accelerator or brake, gear, and handle angle sensors of the vehicle information acquisition unit 101, the acceleration sensor and the vehicle both angle detection unit 204 provided in the terminal 200, and the direction described in the claims A detection part is formed.

Further, the terminal 200 includes an in-vehicle 3D model holding unit 209 that holds an in-vehicle 3D model, which is a 3D model obtained by modeling a three-dimensional (3D) shape of the interior of the vehicle in a state in which in-vehicle equipment such as a handle and a seat is attached. Based on the orientation information of the terminal 200 received from the orientation detection unit 207 and based on the orientation of the terminal 200 in the vehicle from the three-dimensional (3D) model held in the in-vehicle 3D model holding unit 209. 2D conversion unit 208 that cuts out a two-dimensional (2D) model,
A camera image acquisition unit 211 that acquires images in the vehicle captured by a camera mounted on the terminal 200;
An image comparison unit 210 that compares the camera image received from the camera image acquisition unit 211 with the 2D model received from the 2D conversion unit 208 and detects which part of the 2D model the camera image is;
The comparison result is received from the image comparison unit 210, the distance between the previous object and the camera position is received from the previous detection position holding unit 213, and the relative distance between the current camera position and the object taken by the camera is calculated. A distance determination unit 212 from an object to be acquired;
The previous detection position holding which receives the relative distance between the current camera position and the object to be photographed from the object distance determination unit 212 and holds it as information as the previous detection position when the same object is photographed next time. Part 213.

Next, the operation of the terminal position and direction detection device of the mobile unit mounted terminal according to the first embodiment of the present invention will be described. The terminal position and direction detection device of the mobile terminal is operated along the flow of arrows in FIG. The specific process flow is described below.
Process 1. The vehicle information acquisition unit 101 of the car 100 acquires the following information of each state by each sensor and transmits the information to the vehicle direction confirmation unit 203 of the terminal 200 and the angle detection unit 206 in the vertical direction of the vehicle.
・ Accelerator / brake ・ Handle angle ・ Gear position ・ Tilt of the vehicle The sensor information acquisition unit 201 of the terminal 200 acquires information from the following sensors, and transmits the information to the direction / inclination detection unit 202 of the terminal.
・ Acceleration sensor ・ Gyro sensor ・ Magnetic sensor

Process 3. The direction / tilt detection unit 202 of the terminal 200 detects the direction in which the terminal 200 is facing or the tilt with respect to the ground from the information received from the sensor information acquisition unit 201.
Among the information acquired by the sensor information acquisition unit 201, the direction in the horizontal direction that the camera of the terminal faces is known from the information of the magnetic sensor, and the ground of the camera mounted on the terminal is determined from the information of the acceleration sensor and the gyro sensor. You can see the inclination in the vertical direction.

Process 4. The vehicle direction confirmation unit 203 of the terminal 200 confirms the traveling direction of the vehicle.
Based on the accelerator / brake / gear / handle angle information acquired from the vehicle information acquisition unit 101, the vehicle of the terminal 200 determines whether the vehicle 100 is moving forward, moving backward, turning right, or turning left. It is detected by the direction confirmation unit 203. At that time, it is detected which direction is the front of the car 100 with respect to the orientation of the terminal.
From the information of the acceleration sensor, the direction opposite to the direction in which acceleration is applied in the horizontal direction with respect to the terminal 200 is the traveling direction of the vehicle 100 (the relationship perpendicular to the direction of gravity acceleration is the horizontal direction). For example, when the vehicle is moving forward under the condition that the accelerator is stepped on and the gear is moving forward, the direction opposite to the direction in which acceleration is applied can be determined as the traveling direction. If the vehicle is moving backward, the direction of acceleration is the traveling direction. Also, for example, when the accelerator is stepped on, the gear is in the forward direction, the handle angle is tilted to the right, and is turning to the right, the terminal 200 is pulled by pulling the handle angle from the direction in which the acceleration is applied. On the other hand, the direction in which the car 100 is facing can be determined. With these determinations, the front portion of the car 100 can be detected with respect to the orientation of the terminal 200.

Process 5. The direction in which the car 100 is facing is detected by the car both-angle detection unit 204 of the terminal 200.
By comparing the direction in which the terminal 200 is found in the process 3 and the traveling direction of the car 100 with respect to the direction of the terminal 200 found in the process 4 (which is ahead), the direction in which the car 100 is facing Can be detected.
The reason for determining the direction of the car 100 is that when the car 100 is running at a constant speed or when it is stopped, acceleration is not applied and the front of the car is not known. This is because the direction in which the car 100 is facing with respect to the direction in which it is facing is obtained.

Process 6 A direction (horizontal direction) in which the terminal 200 faces in the vehicle 100 is detected by an angle detection unit 205 in the horizontal direction of the terminal 200 with respect to the vehicle.
The direction (horizontal direction) in which the terminal 200 is facing in the vehicle interior 100 is determined by taking the difference between the direction in which the terminal 200 is directed in the process 3 and the direction in which the car 100 is traveling in the process 5. It can be detected.
Process 7 The angle detection unit 206 in the vertical direction of the terminal 200 with respect to the vehicle detects the relative inclination angle of the terminal 200 in the vertical direction with respect to the vehicle 100.
By taking the difference between the information on the inclination of the vehicle 100 with respect to the ground from the vehicle information acquisition unit 101 and the gravitational acceleration direction of the terminal 200 from the direction / tilt detection unit 202 of the terminal 200, that is, the inclination angle in the vertical direction with respect to the ground. The relative angle of the terminal 200 in the car 100 with respect to the car 100 can be detected.

Process 8 The terminal orientation detection unit 207 of the terminal 200 detects the orientation of the terminal in the vehicle 100 in three dimensions.
The orientation of the terminal 200 in the car 100 can be detected in three dimensions from the horizontal orientation and the vertical inclination angle of the terminal 200 with respect to the inside of the car 100 detected in the processes 6 and 7.
Process 9. The 2D conversion unit 208 creates a 2D model from the 3D model.
The 2D conversion unit 208 converts the 3D model acquired from the in-vehicle 3D model holding unit 209 into a 2D model viewed from the direction of the terminal 200 based on the orientation information of the terminal 200 in the vehicle 100 received from the terminal orientation detection unit 207. Create a 2D model.
The 3D model is a model that represents the shape in the vehicle 100 in three dimensions. Feature points such as handles and seats are gathered and represented in three dimensions. Based on the orientation of the terminal 200 received from the terminal orientation detection unit 207, a plane visible from the orientation is cut out from the 3D model, and a 2D model represented by a set of two-dimensional points is created. Here, a conceptual diagram of the 2D model is shown in FIG. The 2D model has information b in the height direction corresponding to the wide angle of the camera that can map all of the 3D model from the viewpoint of the camera installed in the terminal 200 when creating the 2D model from the 3D model. . Furthermore, depth information from the plane of the 2D model to each object is provided. That is, the information on the distance e from the plane to the position where the object such as a handle or a seat actually exists is included with respect to the plane of the 2D model obtained from the 3D model.

Processing 10. The image comparison unit 210 compares the camera image received from the camera image acquisition unit 211 with the 2D model received from the 2D conversion unit 208, and detects which part of the 2D model the camera image corresponds to.
A characteristic point of the camera image is extracted, and the extracted point is collated with the point of the 2D model to detect the location of the camera image in the 2D model.
Process 11. A distance determination unit 212 from the object calculates a distance from the object shown at the center of the camera image from the camera image acquisition unit 211 to the viewpoint of the camera.
When the 2D model is created, the distance b from the center of the 2D model to the viewpoint position in the 2D model is known, so the distance from the camera image to the camera viewpoint is determined from the ratio of the sides of the camera image to the 2D model. d can be calculated. Furthermore, by adding the distance d from the camera viewpoint to the camera image and the depth information e to the object included in the 2D model, the actual image from the center of the camera image to the object reflected in the center is obtained. The distance is revealed.
Note that the previous detection position holding unit 213 cannot capture the feature of the object because the distance between the camera and the object to be imaged is too short, and the image comparison unit 210 cannot compare with the 2D model from the 2D conversion unit 208. The previous detected position held is the relative distance between the current camera position and the object to be imaged.

  As described above, using the information from the car 100, the terminal 200 detects the direction in front of the car, and by taking the difference from the direction in which the terminal 200 faces, Detect orientation. Furthermore, the difference in the inclination angle in the vertical direction is detected by taking the difference in the inclination angle with respect to the gravitational acceleration direction based on the inclination information of the vehicle 100 with respect to the ground and the gyro sensor information from the sensor information acquisition unit 201 of the terminal 200. From the above, the three-dimensional orientation of the terminal 200 in the car 100 can be detected.

A 2D model from the orientation of the terminal 200 is created from the 3D model representing the inside of the car 100 based on the information on the orientation of the terminal 200 obtained above, and a camera image taken by a camera placed in the car 100 And the 2D model, the position and size of the camera image on the 2D model can be grasped, the ratio of the side length of the 2D model to the side length of the camera image, and the viewpoint in the 2D model Can be calculated from the camera image to the camera viewpoint. By adding depth information up to the object reflected in the center of the 2D model, the actual distance from the object reflected in the center of the camera image to the camera viewpoint can be detected.
Here, the distance to the viewpoint in the 2D model gives an orientation when creating the 2D model, but when giving the orientation, a pseudo viewpoint is placed, and the distance from that viewpoint to the plane of the 2D model That means.
From the above, since the orientation and distance of the terminal 200 in the car 100 can be known, the position of the terminal 200 in the car 100 can be uniquely determined. For example, the value of the coordinate on which the terminal 200 is placed can be determined on the coordinate axis with the center point in the vehicle as a base point, and the calculation is performed using the information of the vehicle 100, so that an effect that the calculation amount is small can be obtained.

Embodiment 2. FIG.
In the first embodiment described above, the terminal 200 obtains the front direction of the car 100 from information such as the accelerator / brake of the car 100. If the car 100 has a magnetic sensor, the terminal 200 It is only necessary to receive direction information from 100. Therefore, the processing of the first embodiment 4. 5. The operation of is different.

  The configuration of the second embodiment is shown in FIG. The first embodiment differs from the first embodiment in that the information acquired by the vehicle information acquisition unit 101 of the car 100 is different from the need for the vehicle direction confirmation unit 203 and the vehicle direction detection unit 204 of the terminal 200. In the magnetic sensor fixed to the car 100, the positional relationship between the magnetic sensor and the front portion of the car 100 is determined when the magnetic sensor is fixedly installed on the car 100. In addition, this magnetic sensor corresponds to the direction detection part as described in a claim.

As operations, the changes from the first embodiment are the following two points.
(1) A magnetic sensor is mounted on the vehicle 100, and information acquired by the vehicle information acquisition unit 101 is the following two pieces of information.
・ Magnetic sensor ・ Tilt
(2) Since information acquired from the vehicle information acquisition unit 101 at the terminal 200 is information by the magnetic sensor, there is no need for calculation at the terminal 200 for detecting the direction of the vehicle at the terminal 200. Therefore, the processing 4 and the processing 5 shown in the first embodiment are not necessary, and the information of the magnetic sensor is received by the angle detection unit 205 in the horizontal direction with respect to the vehicle. The operation of the angle detection unit 205 in the horizontal direction with respect to the vehicle after reception is the same as that in the process 6.

  As described above, if the direction of the car 100 is known, there is no need to calculate the direction of the car 100 in the terminal 200, and an effect of further reducing the amount of calculation in the terminal 200 can be obtained.

  In the above embodiments, a terminal mounted on a car and provided with a camera has been described. However, the present invention can be applied not only to a car but also to a mobile body such as a leisure ship or a train. It can also be applied to electronic manuals of objects (parts) installed in the.

  This invention is an electronic manual that applies AR technology, for example, when an explanation of an object installed in a vehicle is desired, when a user photographs an object that the user wants to explain with a camera provided in the terminal, the image is displayed on the terminal screen. The present invention can be applied to an electronic manual or the like that superimposes and displays a description of the object prepared in advance on a captured image and conveys it to the user in an easily understandable manner. The user here includes not only the owner of the car but also a mechanic who uses the repair method to be displayed in a superimposed manner on the part where the part has failed.

  DESCRIPTION OF SYMBOLS 100; Car, 101; Vehicle information acquisition part, 200; Terminal, 201; Sensor information acquisition part, 202; Terminal direction and inclination detection part, 203; Vehicle direction confirmation part, 204; Angle detection unit 206; vertical angle detection unit 207; terminal orientation detection unit 208; 2D conversion unit 209; in-car 3D model holding unit 210; image comparison unit 211; camera image acquisition unit 212; Distance determination unit with object, 213; previous detection position holding unit.

Claims (3)

A device that is mounted on a moving body and detects the position and direction of the terminal on which the camera is installed in the moving body,
A terminal orientation detector for detecting a relative orientation of the terminal in the moving body;
An in-vehicle 3D model holding unit that holds a 3D model of the moving body obtained by modeling the shape of the moving body in three dimensions,
Based on the information of the relative orientation of the mobile of the terminal received from the terminal orientation detection unit, the terminal in the mobile unit from said moving body 3D models stored in the vehicle 3D model holding unit A 2D conversion unit that cuts out a 2D model based on the orientation of
A camera image acquisition unit for acquiring an image of the moving body captured by the camera mounted on the terminal;
Compare the camera image received from the camera video acquisition unit and the two-dimensional model received from the 2D conversion unit, an image comparison unit that detects which part of the two-dimensional model the camera image,
Receiving a comparison result from the image comparison unit, obtaining a distance between the position of the camera and an object photographed by the camera, and a distance determination unit from the object for obtaining a placement position of the terminal in the movable body; A terminal position and direction detection device of a mobile unit equipped terminal. An orientation detection unit that is provided in the mobile body and detects a direction of the mobile body;
A sensor information acquisition unit for acquiring information of an acceleration sensor, a gyro sensor, and a magnetic sensor provided in the terminal ;
A direction / tilt detection unit that detects a direction and a tilt of the terminal based on the information acquired by the sensor information acquisition unit;
The orientation detection unit detects the orientation of the mobile body from the mobile body information of the accelerator or brake, gear, and handle angle of the mobile body and the direction and inclination of the terminal detected by the direction / inclination detection unit. ,
The terminal direction detection unit is configured to detect the direction of the moving body detected by the direction detection unit and the direction and inclination of the terminal detected by the direction / inclination detection unit. The relative position is detected. The terminal position and direction detection device for a mobile unit-equipped terminal according to claim 1.   3. The terminal position of the mobile unit-equipped terminal according to claim 2, wherein the orientation detection unit includes a magnetic sensor that determines a relationship between a direction measured when the mobile unit is installed and a direction of the mobile unit. Direction detection device.

Images