WO2016068092A1 - Smart glasses with self-imaging function - Google Patents

Abstract

Provided are smart glasses with a self-imaging function by which it is possible for a user, while wearing the smart glasses, to image, from the front, herself from the shoulders and above, the smart glasses enabling a user and another user on a video call or the like to be face-to-face and make eye contact. The smart glasses (see-through head-mounted display) with a self-imaging function are used for video calls or the like, and are characterized by being provided with a camera having a fisheye lens or a wide angle lens and with a fixing means such as an arm that connectively fixes the camera to the smart glasses so that the camera is able to be fixed in a position from which the shoulders and above of a user wearing the smart glasses can be imaged from the front.

Description

Smart glasses with self-photographing function

The present invention is a smart glass (see-through type head-mounted display) provided with a camera, and in a videophone or the like, by photographing the top of the user's own shoulder from the front while wearing the smart glass. The present invention relates to a smart glass with a self-photographing function that allows the user and the other user to face each other and adjust their line of sight.

Briefly explain the importance of facing each other in communication.
First of all, it is said that facing each other and looking at the other person's eyes is the basic manner of human beings. Next, from the psychological point of view, a person expresses emotions most prominently, as if there is a habit of saying “the eyes say as much as the mouth”. It is said that you understand. From a physiological point of view, it is said that the line of sight moves in the direction of the brain in use. In other words, from a psychological and physiological point of view, it is considered that a person can communicate his / her feelings with his / her eyes and can read his / her heart by looking at the other person's eyes.
For example, it is possible to confirm whether or not one's intention has been communicated to the other party by checking the reaction of the other party during the conversation. In addition, matching the line of sight is a favorable intention display, and it is possible to communicate more closely by informing the other party of interest.
Therefore, facing each other and matching the line of sight is extremely important in communication.

In recent years, videophones have become widespread as a means of communication, and are widely used in communication with family and friends in remote places, various conferences, and the like. With regard to video conferencing, for example, Patent Document 1 describes a technique for making it possible to better recognize the gesture and facial expression of a subject's face.

In a conventional videophone system, at least the user's own face, the face of the other user, and a camera for photographing the user are often configured to exist in the user's field of view. In this case, only when the user's line of sight is facing the camera, the partner user can match the line of sight with the user. However, since the camera is disposed outside the display, the user is not in line of sight with the other user, and the user is in a strange situation such as talking with the camera.

On the other hand, various kinds of wearable devices are being developed and attracting attention. Smart glasses equipped with a camera can be said to be a representative example. Various smart glasses including their names have been proposed. For example, there are Patent Document 2 and Patent Document 3, and the cameras photograph the front of the user. Although it is not a smart glass (see-through type head mounted display), there is a head mounted camera described in Patent Document 4 as a device for photographing the user himself. As is apparent from the fact that the CCD camera and half mirror of this head mounted camera are formed in a dark box, the see-through type is not used, and the method of fixing to the head is the hair band type or the helmet type. As is clear from the above, it was not an eyeglass type and there was no part corresponding to eyeglasses (glass).

In the present invention, the top of the user's shoulder is photographed from the front using a camera fixed to the smart glass. Conventionally, various techniques using a fisheye camera have been proposed as techniques for photographing a wide range from a short distance. For example, although it is not possible to shoot from the front of the user's shoulder from the front as in the present invention, Japanese Patent Application Laid-Open No. H10-28400 discloses a facial expression in which a fisheye camera is provided in a frame and the face of the user wearing the frame is photographed from a predetermined direction. An output device is described. In the present invention, a camera equipped with a fisheye lens is called a fisheye camera, and a camera equipped with a wide-angle lens is called a wide-angle camera.

A technique for correcting distortion of an image taken using a fisheye camera is not an essential part of the present invention and need not be limited to a specific method, and thus will not be described in detail. For example, as a known technique, Patent Document 5 describes a method of obtaining a flattened image by mapping an image obtained by a lens other than the central projection method onto a virtual plane, and Patent Document 6 describes distortion captured by a fisheye camera. A method for converting an image into an output image corrected by a series of perspective methods is described. Patent Document 7 describes an image conversion apparatus that performs processing for cutting out a part of a distorted circular image obtained by photographing using a fisheye lens and converting it into a planar regular image. Similarly, Patent Document 8 describes an image data conversion apparatus that uses a subject image including spherical distortion photographed using a fisheye lens as an input image and converts the image so as to remove the spherical distortion. In addition, various software for correcting distortion of images taken with a camera equipped with a fisheye lens or a wide-angle lens, and products in which this is implemented in hardware have already been sold.

The present invention also relates to a technique for removing and interpolating a smart glass region from an image of a user wearing a smart glass. Research has already been conducted on a technique for removing and interpolating a region of glasses from an image obtained by photographing a person wearing glasses. For example, in Patent Document 9, the presence or absence of glasses is identified from a photographed image of a person, and an eyeglass pattern is determined. An image processing apparatus that generates a removed face image is described.

Japanese Patent No. 4569196 US Pat. No. D710928S specification Japanese Patent No. 5482711 Japanese Patent No. 4089071 International Publication No. 2013/077076 Japanese Patent No. 3051173 Japanese Patent No. 4629131 Japanese Patent No. 5029645 JP 2009-294955 A

However, the prior art has the following problems.

In the conventional videophone system, the user cannot interact with the other user while keeping his line of sight. This is because, in order for the user and the other user to face each other and adjust the line of sight, it is necessary to match the optical axis of the camera that shoots the user with the line of sight of the user. Because it is on the outside, the optical axis of the camera and the line of sight of the user viewing the other user's eyes do not match, and it is impossible for the user and the other user to face each other.
In this regard, although various ideas have been made in Patent Document 1, since the camera is positioned adjacent to the screen and located outside the screen, it is not possible for the user to face the other user and match his line of sight. could not.

In addition, in the state where the smart glasses described in Patent Document 2 and Patent Document 3 are attached, the distance between the camera and the user is short, or the camera is usually directed forward, etc. Therefore, it was difficult to photograph the user himself with the camera. Therefore, it has been difficult to use the conventional smart glass as it is for a video phone or the like that needs to photograph the user.

Even if the camera provided in the smart glass can be directed to the user himself, if the shooting distance is too close, it is difficult to shoot the entire face, not just the user's shoulder. In videophones and the like, taking a close-up of the face not only gives the other person a feeling of oppression and discomfort, but the user himself may be embarrassed. On the other hand, if the face is too small and the facial expression is not well understood, it is not desirable as a communication means. Therefore, videophones and the like often shoot from the shoulder up. For this reason, it is conceivable that the camera is provided at a position away from the user, but this time the apparatus becomes larger, resulting in a loss of operability and mobility, which are the advantages of wearables.

For example, although it is not a smart glass (see-through type head-mounted display), the head-mounted camera described in Patent Document 4 includes a fixing unit that fixes the imaging unit to the head, thereby increasing the distance between the camera and the user. It was. However, instead of using a fish-eye camera or the like, it is necessary to extend the fixing means considerably in front of the user in order to photograph almost the entire face of the user. Moreover, since not only the imaging means but also the display means are fixed to the head by the same fixing means, and the large casing extends forward, there is a possibility that the operability and mobility, which are the advantages of wearable, may be impaired. Further, as is apparent from the fact that the display means for allowing the user to visually recognize the image information is disposed so as not to be imaged by the imaging means, the imaging means is not capable of imaging the entire face of the user from the front, In videophones and the like, it was difficult for the user and the other user to face each other naturally from the front.
In addition, in the above-mentioned patent document 4, since the camera is directed to the user's face, it is pointed out that there is a large psychological burden due to being conscious of the camera. In this case, the user concentrates on the video of the other user, so there is little awareness of the camera behind it. Therefore, this problem does not occur even when the camera faces the user's face from the front as in the present invention.

Therefore, it is conceivable to apply a technique capable of photographing a wide range from a short distance as described in Patent Document 6 to Patent Document 8.
In this regard, Patent Document 5 describes a method of photographing a face from a fish-eye camera provided on a frame, but the distance between the camera and the user is too close to photograph only a predetermined part. For this reason, complicated processing is required to generate an image of the entire face. For example, the camera in Patent Document 5 has an angle of view that can photograph the user's eyes, eyebrows, and mouth. A checkerboard is attached to a predetermined part of the face, and the user with the checkerboard attached is viewed from the front. It was necessary to photograph in advance with a digital camera and obtain a homography matrix from the correspondence of the checkerboard lattice points. Then, it is necessary to generate front images from images of each part such as eyes, eyebrows, and mouth using this homography matrix, and then synthesize these front images with a three-dimensional model of a face stored in advance. In addition, a process using the upper, lower, left, and right ends of the corners of the eyes, the eyes, and the iris of the eyes is also described. The reason why the above complicated processing is necessary is that the distance between the camera and the user is too close to capture the entire face at once. Certainly, in Patent Document 5, there is a description that a camera may be provided in a headset in which headphones and a microphone are integrated, but this camera is provided at the tip of the same arm as the microphone. As is clear from the above, the user is photographed from around the mouth, and complicated processing is necessary for the same reason as described above. Moreover, when the invention described in Patent Document 5 is used for a videophone or the like, the other user sees the right eye, the left eye, and the mouth portion in the face image of the user stored in advance. Since the combined image is replaced with each developed part, the combined user may not be able to see the face of the user as it is because the combined boundary part may be conspicuous. In addition, since the optical axis of the camera in Patent Document 5 does not match the user's line of sight, it is difficult for the other user to face the user and match the line of sight.

Therefore, a method is conceivable in which a camera capable of wireless LAN connection with a smart glass is installed on a desk or the like without being fixed to the smart glass. However, in this method, it is necessary to install the camera so that it is oriented toward the user. Also, it cannot be used when there is no place to install a camera, such as a park bench.

In addition, by removing the area of the smart glasses from the image and performing interpolation, the relative positional relationship and shooting conditions of the camera and the smart glasses are constant when trying to generate an image of a user who does not wear the smart glasses. Therefore, it is difficult to extract the area of the smart glass. For example, in Patent Document 9, the glasses removal processing procedure is divided into five parts: “pre-processing unit”, “glasses model generation unit”, “glasses detection unit”, “region extraction unit”, and “interpolation unit”. Therefore, complicated processing is required until the spectacle region is extracted.
Further, when linear interpolation or smoothing is used for interpolation of the missing area after removal, there are problems that the texture cannot be expressed or the boundary becomes discontinuous.

Therefore, the present invention solves the problems of the prior art and includes a camera that can take a picture directly from the shoulder of the user while wearing the smart glasses, It is an object of the present invention to provide smart glasses with a self-photographing function and the like that allow the other user to face each other and adjust the line of sight. In addition, the present invention does not require complicated processing, and zooms in the direction of the user's line of sight, or generates an image of the top or front from the user's shoulder without wearing smart glasses. It is also an object to provide a smart glass with a self-photographing function that can capture a user's stereo image.

In order to solve the above-mentioned problem, the smart glass with a self-photographing function according to claim 1 is a smart glass (see-through type head mounted display), a camera having a fish-eye lens or a wide-angle lens, and the camera as the smart glass. A fixing means such as an arm for fixing the connection, and a videophone or the like can be performed by fixing the camera in a position where the top of the user wearing the smart glass can be photographed from the front. And As a result, the user and the other user can face each other and adjust their line of sight.

Next, a smart glass with a self-photographing function according to a second aspect is characterized in that in the smart glass with a self-photographing function according to the first aspect, the fixing means includes an extendable arm. As a result, the arm can be retracted while not making a videophone call.

Next, the smart glass with self-photographing function according to claim 3 is the smart glass with self-photographing function according to claim 1, wherein the fixing means comprises an arm, and the arm can be attached to and detached from the smart glass. It is characterized by that. As a result, the detached arm can be stored in a spectacle case or the like while a videophone or the like is not being performed.

Next, the smart glass with self-photographing function according to claim 4 is the smart glass with self-photographing function according to claim 1, wherein the fixing means includes a pivot shaft at a joint portion with the smart glass, It is possible to turn around the turning axis. This makes it possible to store the fixing means on the temple of the smart glass while not making a videophone or the like.

Next, the smart glass with self-photographing function according to claim 5 is the smart glass with self-photographing function according to claim 1, in which the optical axis of the camera and the user viewing the other user's eyes displayed on the smart glass are displayed. The camera is fixed to the smart glass so that the line of sight can be adjusted, or the position for displaying the image of the other user is fixed, and the user faces the other user and adjusts the line of sight. It is characterized by being able to make telephone calls. As a result, the user and the other user can face each other and adjust their line of sight.

Next, a smart glass with a self-photographing function according to claim 6 is the smart glass with a self-photographing function according to claim 1, comprising a front-facing camera for photographing the front of the user and the fisheye lens or the wide-angle lens. Line-of-sight detecting means for detecting the line of sight of the user based on an image photographed by the camera, and the zooming of the direction of the line of sight can be performed with the forward photographing camera. As a result, it is possible to accurately photograph the object that the user is gazing at. In addition, in a videophone or the like, the other user can view not only the user's video but also the video that the user is watching at the same time, so that the user can share the impression while watching the same beautiful scene, for example. .

Next, a smart glass with a self-photographing function according to claim 7 is the smart glass with a self-photographing function according to claim 1, from an image photographed by the camera or an image after correcting distortion of the image, Smart glass area removing means for automatically removing the smart glass area; and area interpolation means for automatically interpolating the defective area after removing the smart glass area; It is possible to generate an image of the user's face in a non-existing state. Thereby, for example, a user who does not usually wear glasses can show his / her face not wearing smart glasses to the other user.

Next, the smart glass with self-photographing function according to claim 8 is photographed with the front-photographing camera for photographing the front of the user and the front-photographing camera in the smart glass with self-photographing function according to claim 1. A region removing unit that automatically removes the region captured by the self-photographing camera and the fixing unit from the captured image, and a region interpolating unit that automatically interpolates the missing region after removing the region. The front image of the user without wearing the smart glasses can be generated. As a result, it is possible to photograph a front scene without a shielding part by the self-photographing camera and the fixing means.

Next, the smart glass with self-photographing function according to claim 9 is the smart glass with self-photographing function according to claim 1, wherein the stereo image photographing camera having a fish-eye lens or a wide-angle lens and the stereo image photographing camera are Fixing means such as an arm for connecting and fixing to a smart glass, and fixing the stereo image capturing camera at a position away from the camera according to claim 1, whereby a stereo image of the user can be captured. It is characterized by doing so. Accordingly, the other user can view the user's 3D video using the captured stereo image, and a videophone or the like having a more realistic feeling can be realized.

Next, the fixing means according to claim 10 is used for the smart glass with a self-photographing function according to claim 1 and can be attached to and detached from the smart glass. As a result, while the videophone or the like is not being performed, the removed fixing means can be stored in a spectacle case or the like.

The user can shoot from the front of the user's own shoulders with the smart glasses on. In addition, a videophone using a smart glass, a video conference, and the like can be performed, and a user can talk while facing the other user while keeping a line of sight. Furthermore, the user can zoom in on the direction of the user's line of sight, and can generate a user's face and a front image without wearing smart glasses. When two self-photographing cameras are provided, a stereo image of the user's face can be taken.

FIG. 1 is a perspective view of a smart glass with a self-photographing function according to an embodiment of the present invention (when the fixing means is an extendable arm). FIG. 2 is a plan view of a smart glass with a self-photographing function according to an embodiment of the present invention (when the fixing means is an extendable arm). FIG. 3 is an explanatory diagram of an image photographed with a standard lens, and an explanatory diagram of an image photographed with a fisheye lens (Circular Fisheye Lens 180 °). FIG. 4 is an explanatory diagram of an image before distortion correction and an image after distortion correction. FIG. 5 is a perspective view of a smart glass with a self-photographing function (in the case of a round connector connection) according to an embodiment of the present invention. FIG. 6 is a perspective view of smart glasses with a self-photographing function (in the case of flat connector connection) according to an embodiment of the present invention. FIG. 7 is a perspective view of a smart glass with a self-photographing function according to an embodiment of the present invention (when a USB connector is connected). FIG. 8 is a plan view of smart glasses with a self-photographing function (in the case of connector connection) according to the embodiment of the present invention. FIG. 9 is a perspective view of a smart glass with a self-photographing function (in the case of wireless LAN connection) according to an embodiment of the present invention. FIG. 10 is a perspective view of a smart glass with a self-photographing function according to an embodiment of the present invention (when the fixing means is an arm having a pivot axis). FIG. 11 is a plan view of a smart glass with a self-photographing function according to an embodiment of the present invention (when the fixing means is an arm having a pivot axis). FIG. 12 is an explanatory diagram relating to the optical axis of the camera and the line of sight of the user viewing the other user's eyes. FIG. 13 is an explanatory diagram of a scene in front of the user. FIG. 14 is an explanatory diagram of the gaze direction zoom. FIG. 15 is an explanatory diagram of smart glass region removal and interpolation in a self-photographed image. FIG. 16 is an explanatory diagram of camera area removal and interpolation for self-photographing in a front-captured image. FIG. 17 is a plan view of a smart glass with a self-photographing function (for stereo image photographing) according to an embodiment of the present invention.

Examples of smart glasses with a self-photographing function according to the best mode of the present invention will be described below with reference to the drawings. In addition, as much as possible, the same code | symbol is attached | subjected to the same part and the overlapping description is abbreviate | omitted. Further, the description overlapping with the description in the means for solving the problem is omitted as much as possible.

As shown in FIGS. 1 and 2, the smart glass with a self-photographing function according to the embodiment of the present invention is a smart glass 1, a fisheye camera 3 having a fisheye lens 2, and a fisheye camera 3. The fisheye camera 3 is fixed at a position where the upper part of the user wearing the smart glass 1 can be photographed from the front. Note that the one-dot chain line in the figure is the optical axis of the fisheye camera 3. It is desirable that the extendable arm 4 can be expanded and contracted with one button each time the arm extension switch button 5 is pressed. The extendable arm 4 can be automatically extended when calling the other user or answering the call from the other user, and can be automatically retracted when the videophone call is terminated. In this case, the arm extendable switch button 5 is not necessary. Further, it is desirable that the extendable arm 4 can be stored in the spectacle frame of the smart glass 1. The front shooting camera 6 is a camera capable of shooting a scene in front of the user.

As for the video projection unit of the smart glass 1, various methods and forms such as those using a liquid crystal display, those using a projector, those using a prism, and those using a contact lens are conceivable. In the essential part of the present invention, Since it is not shown, it is illustrated by a dotted line in FIGS. 1 and 2 and FIGS. In the present invention, it is a general principle that the smart glass 1 has a binocular display because the user and the other user face each other in a videophone or the like so as to adjust the line of sight. However, when the present invention is applied to a smart glass of a monocular display exceptionally, the fisheye camera 3 is fixed at a position where the top of the right eye or the left eye wearing the monocular display can be photographed from the user's shoulder, The user may be able to match the line of sight with the other user with one eye.

The fisheye camera 3 includes all cameras capable of photographing the top from the user's shoulder from a position close to the user's face, specifically, a position about the length of the temple of the smart glass 1. . That is, any device that can capture a range normally required for a videophone or the like may be used.

The fixing means such as an arm for connecting and fixing the self-photographing camera such as the fisheye camera 3 to the smart glass is not extendable and retractable like the arm 7 shown in FIGS. 5 to 9, and can be attached to and detached from the smart glass 1. It may be. Moreover, there is no restriction | limiting in the material, the contact with the smart glass 1, etc. However, it is desirable to make it as thin or transparent as possible while maintaining strength so as not to obstruct the user's field of view as much as possible. Further, as a common requirement for wearable devices, it is desirable to be lightweight.

The fisheye camera 3 is provided with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), and its output signal is an image signal input section of the smart glass 1 by a cable passing through a fixing means such as an arm. Is transmitted.

Also, the output signal of the fisheye camera 3 can be transmitted to the smart glass 1 by wireless LAN. In this case, since the fisheye camera 3 can be externally attached to the smart glass 1 and the application software can be downloaded and used, it is possible to support various types of smart glasses.

By fixing the fisheye camera 3 in a position where the top of the user's shoulder wearing the smart glass 1 can be photographed from the front, the user and the other user face each other while looking at each other's face in a video phone. To be able to interact with each other.

At this time, it is naturally possible to simultaneously shoot the scene that the user is viewing using the front-facing camera 6, so the other user sees what scene the user sees and what expression Can be seen at the same time.

In order to correct distortion of an image photographed by the fisheye camera 3 or the like, a conventional fisheye image projection conversion technique can be used. As described in the background art, conversion formulas and the like are known and are not an essential part of the present invention, and therefore will not be described again here. The left side of FIG. 3 is an explanatory view of an image taken with a standard lens, and the right side is an explanatory view of an image taken with a fisheye lens (Circular Fischer Lens 180 °). When a user wearing the smart glass 1 is photographed from the same position as the fisheye camera 3 shown in FIGS. 1 and 2 with a camera equipped with a standard lens (angle of view 50 °), an image as shown on the left side of FIG. And only a partial image of the face can be obtained. On the other hand, when the same image is taken with a fisheye lens (angle of view 180 °), an image as shown on the right side of FIG. 3 is obtained, and the image taken from the front of the user's shoulder is corrected by correcting the distortion. Is obtained. The left side of FIG. 4 shows an image before distortion correction, and the right side shows an image after distortion correction / cutout / enlargement processing.

For transmission / reception of image data and audio data between users, the same technology as a conventional videophone using a Web camera and a PC can be applied.

As shown in FIGS. 5 to 9, the smart glass with a self-photographing function according to the embodiment of the present invention can attach and detach the arm 7 to the smart glass 1. In this case, signal transmission between the fisheye camera 3 and the smart glass 1 is performed using a wired cable or a wireless LAN.

First, when a wired cable is used, the arm 7 that can be attached and detached incorporates a cable that connects the fisheye camera 3 or the like and the smart glass 1 and can be connected to the smart glass 1 by various connectors. Further, a cover is provided at the connector insertion port of the smart glass 1 so that the connector insertion port can be closed when the arm 7 is removed. Further, in order to prevent the arm 7 from falling off, the arm 7 can be detached only when the connector removal button 8 is pushed. For example, the circular connector 9 shown in FIG. 5 can be connected to the smart glass 1 by being inserted into a circular connector insertion port 11 provided with a circular lid 10. Also, the flat connector 12 shown in FIG. 6 is connected to the smart glass 1 by being inserted into a flat connector insertion port 14 provided with a flat lid 13. Further, the USB connector 15 shown in FIG. 7 can be connected to the smart glass 1 by being inserted into a USB connector insertion port 17 provided with a rectangular lid 16.

Next, when the wireless LAN is used, the arm 7 shown in FIG. 9 includes a connecting portion 18 with the frame of the smart glass 1 and can be fixed with a tightening screw 19. When the wireless LAN is used, an output signal from the fisheye camera 3 can be received by downloading the application software, so that various types of smart glasses 1 can be supported. At this time, for high-speed processing, it is desirable to mount the distortion correction processing on hardware instead of software. If it cannot be mounted on the smart glass 1, it may be mounted on the fisheye camera 3.
The connecting portion 18 and the tightening screw 19 can be adjusted for each corresponding model of the smart glass 1. In addition, the design of the connection part 18 grade | etc., In FIG. 9 is only an example.
This makes it possible to remove the fisheye camera 3 and the arm 7 except when making a videophone call, and the operability and mobility are improved.

As shown in FIGS. 10 and 11, in the smart glass with a self-photographing function according to the embodiment of the present invention, the swivel arm 20 includes an arm swivel shaft (part 1) 21 at a joint portion with the smart glass 1, The fisheye camera 3 can be turned around the arm turning axis (part 1) 21.

The swivel arm 20 is joined to the vicinity of the temple front end of the smart glass 1 by an arm swivel axis (part 1) 21. The swivel arm 20 is overlapped or fitted on the temple of the smart glass 1 except when making a videophone call. At this time, since the fisheye camera 3 comes to be located behind the ear, it is necessary to design so that the user's head is not uncomfortable.

When making a videophone call or the like, the swivel arm 20 is first swiveled forward 180 ° around the arm swivel axis (part 1) 21 and then viewed from the user around the arm swivel axis (part 2) 22. Turn left 90 °. Thus, the fisheye camera 3 can be fixed in front of the user.

As shown in FIG. 12, the smart glass with a self-photographing function according to the embodiment of the present invention can match the optical axis of the fisheye camera 3 with the line of sight of the user looking at the other user's eye. The position of the fisheye camera 3 is fixed, or the position for displaying the image of the other user is fixed. As a result, the user can make a videophone call or the like while facing the other user and adjusting the line of sight.

The arrows shown in FIG. 12 are the optical axis of the camera and the line of sight 23 of the user looking at the other user's eyes. At this time, in the video 24 of the other user visually recognized by the user, the other user's eye, which is the user's gazing point, forms an image in the fovea of the user's eyeball 25. And in the user's image | video image | photographed with the fisheye camera 3, the user's eye which is the other user's gazing point forms an image in the fovea of the other user's eyeball. As a result, the user and the other user can face each other and face each other.

Next, the smart glass with a self-photographing function according to the embodiment of the present invention is an image captured by the front-view camera 6 and the fish-eye camera 3 for capturing a scene in front of the user as shown in FIG. Line-of-sight detecting means for detecting the line of sight of the user based on this, and the direction of the line of sight can be zoomed and photographed by the front photographing camera 6. In FIG. 13, there are three hills as an example of a scene in front of the user, a house 26 is built on the left hill, and a person 27 is standing on the right hill.

When shooting the direction in which the user is gazing with the front shooting camera 6 provided in the smart glass 1, the direction in which the front shooting camera 6 is facing matches the direction in which the user is gazing. Not exclusively. This is because the front-facing camera 6 is generally oriented in the same direction as the user's face, but the user often moves only the eyeball without facing his face when looking at a specific part of the scene spreading forward. . Therefore, the user's line of sight is detected by the line-of-sight detecting means for detecting the user's line of sight based on the image photographed by the fisheye camera 3, and the direction of the line of sight is zoomed by the front photographing camera 6 so as to be photographed.

For example, when the user's line of sight is directed to the left as shown in the lower left of FIG. 14 and the user is gazing at the house 26, as shown in the upper left of FIG. Further, as shown in the lower right of FIG. 14, when the user's line of sight is directed to the right and watching the person 27, the person 27 can be zoomed and photographed as shown in the upper right of FIG. To.

In the present invention, since the relative positional relationship between the fish-eye camera 3 for self-photographing and the smart glass 1 is constant, the user's line of sight can be easily detected. Since the relative positional relationship with the camera 6 is also constant, it is possible to shoot with the front camera 6 zooming in the direction of the line of sight.

Next, the smart glass with a self-photographing function according to the embodiment of the present invention automatically identifies the area of the smart glass 1 from the image taken by the fisheye camera 3 or the image after correcting the distortion of the image. Smart glass area removing means for removing and area interpolating means for automatically interpolating the missing area after removing the area of the smart glass 1, the face of the user not wearing the smart glass 1 An image can be generated.

As a result, when a user who does not normally wear glasses, such as a videophone, wants to show his / her face without wearing smart glasses to the other user, he / she creates his / her face without wearing smart glasses from the self-photographed image. Then you can show it to the other party.

In the present invention, the relative positional relationship between the fisheye camera 3 and the smart glass 1 is constant. Therefore, it is possible to easily identify and remove the area where the smart glass 1 and the fixing means are reflected. Next, the positional relationship between the smart glass 1 and the user's face when the smart glass 1 is mounted is generally constant. Therefore, the missing area after the area of the smart glass 1 is removed can be interpolated according to the following procedure, for example.
First, the user who is wearing and removing the smart glass 1 is photographed by the front photographing camera 6 or the like. Then, in the user's image taken when the smart glass 1 is removed, the area to be shielded by the smart glass 1 is divided into small areas, and the color or texture of each small area or the continuity of the edge or contour The other region where is most similar is specified in advance.
Next, when making a videophone or the like, the missing area after the area of the smart glass 1 is removed is interpolated with the pixel value and texture of the area at the position corresponding to the area specified in advance.

For example, the image shown in the right of FIG. 15 is obtained by removing the smart glass region from the image of the user wearing the smart glass shown in the left of FIG. 15 and interpolating the defective region after the removal. Can be obtained.

Next, the smart glass with a self-photographing function according to the embodiment of the present invention is a self-photographing fish from a front-photographing camera 6 for photographing the front of the user and an image photographed by the front-photographing camera 6. A smart glass 1 comprising: an area removing unit that automatically removes an area in which the eye camera 3 and the arm 7 are reflected; and an area interpolating unit that automatically interpolates a missing area after removing the area. It is possible to generate a front image of the user that is not worn.

Also in this case, since the relative positional relationship between the self-photographing fisheye camera 3 and the smart glass 1 is constant, the areas of the fisheye camera 3 and the arm 7 can be easily removed. In addition, for the interpolation of the missing area after removing the area, the conventional linear interpolation and smoothing can be used, but the texture cannot be expressed and the boundary discontinuity may occur. It is desirable to interpolate with a region corresponding to the missing region from among the above.

For example, by removing the area in which the fisheye camera 3 and the arm 7 are reflected from the image of the front of the user as shown on the left in FIG. 16, and interpolating the missing area after the removal, FIG. The image shown on the right can be obtained.

Next, a smart glass with a self-photographing function according to an embodiment of the present invention includes a fish-eye camera for stereo image photography, and fixing means such as an arm for connecting and fixing the stereo image photography camera to the smart glass 1. By fixing the stereo image capturing camera at a position away from the fisheye camera 3, the user's stereo image can be captured. In this case, since the fisheye camera 3 is fixed at a position where the user wearing the smart glass 1 can be photographed from the front, the newly provided fisheye camera for stereo image photography is, for example, a temple already equipped with an arm. Secure to the temple on the opposite side. As a result, the user can be photographed simultaneously with the two fisheye cameras on the left and right. In this case, in general, the stereo base (the distance between the left and right cameras) is often about 6 to 7 cm, which is the same as the human eye width. However, in the present invention, the distance to the user who is the subject is short. It is desirable to set a small value. In addition, it is desirable to fix the direction of the camera so that parallel photographing can be performed. For example, as shown in FIG. 17, a stereo image of the user can be taken by fixing the right image taking fisheye camera 28 to the smart glass 1 with the right image taking arm 29.

Since smart glasses that are expected to be widely used in the future can be used for close communication such as videophones and video conferences, the industrial applicability is extremely high.

DESCRIPTION OF SYMBOLS 1 Smart glass 2 Fisheye lens 3 Fisheye camera 4 Extendable arm 5 Arm telescopic switch button 6 Front camera 7 Arm 8 Connector removal button 9 Round connector 10 Round lid 11 Round connector insertion port 12 Flat connector 13 Flat lid 14 Flat connector insertion port 15 USB connector 16 Square lid 17 USB connector insertion port 18 Connecting portion 19 Tightening screw 20 Revolving arm 21 Arm rotation axis (1)
22 Arm rotation axis (2)
23 The optical axis of the camera and the line of sight of the user looking at the other user's eyes 24 The image of the other user visually recognized by the user 25 The user's eyeball 26 The house 27 People 28 The fisheye camera 29 for right image shooting The arm for right image shooting

Claims (10)

A smart glass (see-through type head-mounted display), comprising a camera having a fisheye lens or a wide-angle lens, and a fixing means comprising an arm for connecting and fixing the camera to the smart glass, and a user wearing the smart glass A smart glass with a self-photographing function, characterized in that a videophone or a videoconference can be performed by fixing the camera at a position where the camera can be photographed from directly above the shoulder. The smart glass with a self-photographing function according to claim 1, wherein the fixing means is an extendable arm. The smart glass with a self-photographing function according to claim 1, wherein the fixing means comprises an arm, and the arm can be attached to and detached from the smart glass. The smart glass with a self-photographing function according to claim 1, wherein the fixing means includes a turning shaft at a joint portion with the smart glass, and the camera can turn around the turning shaft. The camera is fixed to the smart glass or the other user's video is displayed so that the optical axis of the camera and the line of sight of the user viewing the other user's eyes displayed on the smart glass can be matched. The smart glass with a self-photographing function according to claim 1, wherein a position to be fixed is fixed, and a videophone or videoconference can be performed while the user faces the other user and adjusts his line of sight. A front shooting camera for shooting the front of the user, and a line-of-sight detecting means for detecting a user's line of sight based on an image taken by the camera having the fisheye lens or the wide-angle lens, and the line of sight with the front shooting camera The smart glass with a self-photographing function according to claim 1, wherein the camera can be photographed by zooming in the direction of. Smart glass area removing means for automatically removing the smart glass area from the image taken by the camera or the image after correcting the distortion of the image, and the defect after removing the smart glass area 2. A self-photographing function according to claim 1, further comprising region interpolation means for automatically interpolating a region, wherein an image of the user's face in a state in which smart glasses are not worn can be generated. Smart glasses. A front shooting camera for shooting the front of the user, a region removing unit that automatically removes a region in which the self-shooting camera and the fixing unit are reflected, from the image shot by the front shooting camera; 2. An area interpolating means for automatically interpolating a missing area after removing the area, so that a front image of the user in a state where smart glasses are not worn can be generated. Smart glasses with the self-photographing function described. 2. The camera according to claim 1, further comprising: a stereo image capturing camera including a fisheye lens or a wide-angle lens; and a fixing unit including an arm that connects and fixes the stereo image capturing camera to the smart glass. The smart glass with a self-photographing function according to claim 1, wherein the stereo image of the user can be photographed by fixing at a position away from the user. * The fixing means used for the smart glass with a self-photographing function according to claim 1, which can be attached to and detached from the smart glass.

Images