WO2022080554A1 - Apparatus for generating three-dimensional map by using stereo frames of lightweight augmented reality device, and method therefor - Google Patents

Abstract

A service server for generating a three-dimensional map by using stereo frames, in the present invention, comprises: a communication module for continuously receiving a plurality of frames including a main frame and subframes of a video captured by an augmented reality device; a pose derivation unit for obtaining pose information from the main frame and the subframes; a depth map calculation unit, which uses a deep learning model to derive a depth map from the main frame and the subframes; and a three-dimensional map generation unit for generating a three-dimensional map on the basis of the pose information and the depth map, and transmitting the generated three-dimensional map to the augmented reality device through the communication module.

Description

Apparatus and method for generating 3D map using stereo frame of lightweight augmented reality device

The present invention relates to a technology for constructing a three-dimensional map, and more particularly, to an apparatus for generating a three-dimensional map using a stereo frame of a lightweight augmented reality device, and a method therefor.

Virtual reality (VR) refers to a specific environment or situation or the technology itself that is similar to reality created by artificial technology using a computer, etc. but is not real. Augmented reality (AR) is a field of virtual reality (VR) and is a computer graphic technique that synthesizes virtual objects or information in an actual environment to make them appear as if they exist in the original environment. In other words, augmented reality is a technology that superimposes virtual objects on the real world that users see with their eyes. It is also called mixed reality (MR) because the real world and the virtual world with additional information are combined in real time and displayed as a single image. Augmented reality, a concept that complements the real world with a virtual world, uses a virtual environment created with computer graphics, but the main character is the real environment. Computer graphics serve to provide additional information necessary for the real environment. This means that the distinction between the real environment and the virtual screen is blurred by overlapping the 3D virtual image on the actual image the user is viewing.

Virtual reality technology immerses the user in the virtual environment, making it impossible to see the real environment. However, augmented reality technology, in which the real environment and virtual objects are mixed, allows users to see the real environment, providing better realism and additional information.

In the case of a lightweight augmented reality device, the width is less than 20 cm because it will have a size similar to that of the largest glasses. For augmented reality service, it is necessary to track the position and gaze of the augmented reality device. To this end, if a stereo camera is used, in the case of a lightweight augmented reality device, the stereo camera (camera unit) must be installed in a narrow range of 20 cm or less, and in this state, there is no choice but to have a narrow parallax. With such a narrow disparity, distance detection in a distant space becomes inaccurate, which limits the augmented reality service type. In most cases, the focus is on real-time detection rather than precision of location detection due to the characteristics of an augmented reality device that can provide a service without a sense of heterogeneity only by detecting the location of the augmented reality device at every moment. In addition, in the case of techniques for improving the precision of position detection, it is difficult to obtain desired performance from the computing power of the lightweight augmented reality device. Moreover, in the case of a lightweight augmented reality device, its performance is more limited due to a power (battery) problem.

An object of the present invention in consideration of the above is to provide an apparatus for generating a three-dimensional map using a stereo frame of a lightweight augmented reality device and a method therefor.

A service server for generating a three-dimensional map using a stereo frame according to a preferred embodiment of the present invention for achieving the above object is a plurality of frames including a main frame and a sub-frame of an image photographed from an augmented reality device. a communication module continuously receiving and a 3D map generator for generating a 3D map based on the pose information and the depth map, and transmitting the generated 3D map to the augmented reality device through the communication module.

The pose derivation unit extracts a feature point representing the same object from the main frame and the sub frame, and sequentially generates the pose information and the pose matrix through a change between the coordinates of the feature point in the main frame and the coordinates of the feature point in the sub frame. It is characterized in that it is derived by .

The depth derivation unit derives a transformation matrix using the known camera matrix and the pose matrix of the augmented reality device, and uses the transformation matrix to generate a simulating sub-frame simulating the sub-frame from the main frame, and the depth It is characterized in that the depth map is derived according to the difference in coordinates between the pixels of the sub-frame and the pixels of the simulated sub-frame through the learning model.

The depth derivation unit derives a transformation matrix using a pose matrix and a known camera matrix derived from the main frame for learning and the sub-frame for learning, and uses the transformation matrix to simulate the sub-frame for learning from the main frame for learning. Create a frame, learn the correlation between the coordinate difference and the depth of the pixel of the sub-frame for learning of the part representing the same object in the real world with respect to the prototype of the model and the pixel of the sub-frame for imitation learning, thereby learning the pixel of the sub-frame for training And it is characterized in that it generates a deep learning model for deriving a depth map according to the coordinate difference of the pixel of the sub-frame for simulation learning.

The method for generating a three-dimensional map using a stereo frame of a service server according to a preferred embodiment of the present invention for achieving the object as described above is the main frame of the image captured by the pose derivation unit from the augmented reality device through the communication module. and deriving pose information from the main frame and the sub-frame when receiving a plurality of stereo frames including a sub-frame, and a depth map calculating unit using a deep learning model to obtain a depth map from the main frame and the sub-frame deriving a 3D map generating unit generating a 3D map based on the pose information and the depth map; It includes a 3D map generator for transmitting to the device.

The step of deriving the pose information from the main frame and the sub-frame includes: the pose extracting unit extracting a feature point representing the same object from each of the main frame and the sub-frame; and sequentially deriving pose information and a pose matrix through a change between the coordinates and the coordinates of the feature point in the sub-frame.

The step of generating a three-dimensional map based on the pose information and the depth map may include deriving a transformation matrix by the depth derivation unit using a known camera matrix and the pose matrix of the augmented reality device, and the depth derivation unit using the pose matrix. generating an imitation sub-frame simulating a sub-frame from the main frame using a transformation matrix, and the depth derivation unit through the deep learning model deriving a map.

The method includes, before deriving the pose information from the main frame and the sub-frames, the depth derivation unit deriving a transformation matrix using a pose matrix derived from the main frame for learning and the sub-frame for learning and a known camera matrix; The depth derivation unit generates a sub-frame for imitation learning that simulates the sub-frame for learning from the main frame for learning using the transformation matrix, and the depth derivation unit represents the same object in the real world with respect to the prototype of the model. Deep learning the correlation between the depth and the coordinate difference between the pixels of the sub-frame for learning and the pixels of the sub-frame for simulation learning to derive a depth map according to the coordinate difference between the pixels of the sub-frame for learning and the pixels of the sub-frame for simulating learning It further comprises the step of generating a learning model.

In the present invention, as described above, the service server receives the stereo frame of the image captured by the augmented reality device, and derives a three-dimensional map therefrom. And according to the present invention, the augmented reality device can match the virtual object to the image taken by using the 3D map derived from the service server. Since the 3D map of the present invention provides precise 3D coordinates, precise registration is possible when registering virtual objects. Accordingly, it is possible to provide augmented reality with higher realism. Moreover, since the 3D map of the present invention is generated by the service server rather than directly derived by the augmented reality device, it is possible to provide a high-quality service even when the augmented reality device is lightweight.

1 is a diagram for explaining the configuration of a system for providing an augmented reality image according to an embodiment of the present invention.

2 is a view for explaining the configuration of a lightweight augmented reality device according to an embodiment of the present invention.

3 is a diagram for explaining the configuration of a service server for providing a virtual reality image according to an embodiment of the present invention.

4 is a view for explaining a detailed configuration of a control unit according to an embodiment of the present invention.

5 and 6 are diagrams for explaining a method of deriving a pose matrix according to an embodiment of the present invention.

7 is a diagram for explaining a method of deriving a transformation matrix using a pose matrix and a camera matrix according to an embodiment of the present invention.

8 is a diagram for explaining a method of generating a simulated frame using a transform matrix according to an embodiment of the present invention.

9 is a diagram for explaining a method for learning a correlation between pixels of two frames according to an embodiment of the present invention.

10 is a flowchart illustrating a method for generating a 3D map using a stereo frame of a lightweight augmented reality device according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term for explanation. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical spirit of the present invention, so various equivalents that can be substituted for them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that in the accompanying drawings, the same components are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, a system for providing an augmented reality (AR) image according to an embodiment of the present invention will be described. 1 is a diagram for explaining the configuration of a system for providing an augmented reality image according to an embodiment of the present invention.

Referring to FIG. 1 , a system for providing a virtual reality image according to an embodiment of the present invention includes an augmented reality device 10 and a service server 20 . The augmented reality device 10 and the service server 20 are connected through wireless communication.

The augmented reality device 10 may be any device capable of providing an augmented reality image to a user. Representatively, the augmented reality device 10 may be exemplified by smart glasses.

The service server 20 may be an edge cloud server located closest to the augmented reality device 10 or a high-performance PC connected to the augmented reality device 10 through Wi-Fi, etc.

According to the present invention, the service server 20 receives the stereo frame of the image captured by the augmented reality device 10 , generates a 3D map from the stereo frame, and transmits the generated 3D map to the augmented reality device 10 . to provide. As described above, according to the present invention, the augmented reality device 10 does not require high performance by generating a three-dimensional map using the resources of the service server 20, that is, computing operation. Therefore, the augmented reality device 10 can be lightweight.

Then, the augmented reality device 10 according to an embodiment of the present invention will be described. 2 is a view for explaining the configuration of a lightweight augmented reality device according to an embodiment of the present invention.

Referring to FIG. 2 , the augmented reality device 10 according to an embodiment of the present invention includes a camera unit 11 , a communication unit 12 , a sensor unit 13 , an audio unit 14 , an input unit 15 , and a display unit ( 16), a storage unit 17 and a control unit 18 .

The camera unit 11 is for capturing an image. In particular, the camera unit 11 according to an embodiment of the present invention may be a stereo camera. Accordingly, the image captured by the camera unit 11 includes a main frame and a sub frame. To this end, the camera unit 12 may include two lenses and two image sensors corresponding to each of the main frame and the sub-frame. Each image sensor receives light reflected from a subject and converts it into an electrical signal, and may be implemented based on a Charged Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS), or the like. The camera unit 11 may further include one or more analog-to-digital converters, and may convert an electrical signal output from the image sensor into a digital sequence and output it to the control unit 18 .

The communication unit 12 is for communication with the service server 20 . The communication unit 12 may include a radio frequency (RF) transmitter (Tx) that up-converts and amplifies the frequency of the transmitted signal, and an RF receiver (Rx) that low-noise amplifies the received signal and down-converts the frequency. In addition, the communication unit 12 may include a modem that modulates a transmitted signal and demodulates a received signal.

The sensor unit 13 is for measuring inertia. The sensor unit 13 includes an Inertial Measurement Unit (IMU), a Doppler Velocity Log (DVL), an Attitude and Heading Reference System (AHRS), and the like. The sensor unit 13 measures inertial information including the position and speed of rotation and movement of the augmented reality device 10 and provides the measured inertial information of the augmented reality device 10 to the control unit 18 .

The audio unit 14 includes a speaker SPK for outputting an audio signal and a microphone MIKE for receiving an audio signal. The audio unit 14 may output an audio signal through the speaker SPK or transmit an audio signal input through the microphone MIKE to the control unit 18 under the control of the control unit 18 . In particular, the audio unit 14 serves to output an audio signal of the virtual reality image.

The input unit 15 receives a user's key manipulation for controlling the augmented reality device 10 , generates an input signal, and transmits it to the control unit 18 . The input unit 15 may include various types of keys for controlling the augmented reality device 10 . In the input unit 15, when the display unit 16 is formed of a touch screen, the functions of various keys can be performed on the display unit 16, and when all functions can be performed only with the touch screen, the input unit 15 may be omitted. may be

The display unit 16 visually provides a menu of the augmented reality device 10, input data, function setting information, and various other information to the user. The display unit 16 performs a function of outputting a boot screen, a standby screen, a menu screen, and the like of the augmented reality device 10 . In particular, the display unit 16 performs a function of outputting a 3D map according to an embodiment of the present invention to the screen. The display unit 16 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 16 may be implemented as a touch screen. In this case, the display unit 16 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all types of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor may detect a user's touch input, generate a detection signal including input coordinates indicating the touched position, and transmit it to the controller 18 . In particular, when the display unit 16 is formed of a touch screen, some or all of the functions of the input unit 15 may be performed through the display unit 16 .

The storage unit 17 serves to store programs and data necessary for the operation of the augmented reality device 10 . In particular, the storage unit 17 may store a camera matrix, a pose matrix, and the like. Also, the storage unit 17 may store a virtual object to be matched with a real image in order to provide an augmented reality image. Various types of data stored in the storage unit 17 may be deleted, changed, or added according to a user's operation of the augmented reality device 10 .

The controller 18 may control the overall operation of the augmented reality device 10 and the signal flow between internal blocks of the augmented reality device 10 , and perform a data processing function of processing data. In addition, the control unit 18 basically performs a role of controlling various functions of the augmented reality device (10). The controller 18 may include a central processing unit (CPU), a baseband processor (BP), an application processor (AP), a graphic processing unit (GPU), a digital signal processor (DSP), and the like.

The controller 18 takes an image through the camera unit 11 and transmits the captured image to the service server 20 through the communication unit 11 in units of stereo frames including a main frame and a sub frame. Then, the service server 20 will generate a 3D map from a plurality of stereo frames including a main frame and a sub frame, and transmit the generated 3D map to the augmented reality device 10 . Accordingly, the controller 18 may provide the augmented reality image by receiving the 3D map through the communication unit 11 and matching the virtual object to the image captured using the received 3D map.

Next, the service server 20 for generating a three-dimensional map using the stereo frame of the lightweight augmented reality device 10 according to an embodiment of the present invention will be described. 3 is a diagram for explaining the configuration of a service server for providing a virtual reality image according to an embodiment of the present invention. Referring to FIG. 3 , the service server 20 according to an embodiment of the present invention includes a communication module 21 , a storage module 22 , and a control module 23 .

The communication module 21 is for communicating with the augmented reality device 10 through a network. The communication module 21 may transmit/receive data to and from the augmented reality device 10 . The communication module 21 may include an RF (Radio Frequency) transmitter (Tx) for up-converting and amplifying the frequency of a transmitted signal, and an RF receiver (Rx) for low-noise amplifying a received signal and down-converting the frequency. . Also, the communication module 21 may include a modem for modulating a signal to be transmitted and demodulating a signal to be received in order to transmit/receive data. The communication module 21 may transmit data received from the control module 23 , for example, a 3D map to the augmented reality device 10 . In addition, the communication module 21 may transmit data received from the augmented reality device 10 , for example, a stereo frame to the control module 23 .

The storage module 22 serves to store programs and data necessary for the operation of the service server 20 . For example, the storage module 22 may store various parameters for generating a 3D map corresponding to the augmented reality device 10 , for example, a camera matrix of the augmented reality device 10 . Various types of data stored in the storage module 121 may be registered, deleted, changed, or added according to the operation of the service server 20 administrator.

The control module 23 may control the overall operation of the service server 20 and the signal flow between internal blocks of the service server 20 , and may perform a data processing function of processing data. The control module 130 may be a central processing unit, a digital signal processor, or the like. In addition, the control module 23 may further include an image processor or a graphic processing unit (GPU).

Then, the detailed configuration of the above-described control module 23 will be described in more detail. 4 is a view for explaining a detailed configuration of a control unit according to an embodiment of the present invention. 5 and 6 are diagrams for explaining a method of deriving a pose matrix according to an embodiment of the present invention. 7 is a diagram for explaining a method of deriving a transformation matrix using a pose matrix and a camera matrix according to an embodiment of the present invention. 8 is a diagram for explaining a method of generating a simulated frame using a transform matrix according to an embodiment of the present invention. 9 is a diagram for explaining a method for learning a correlation between pixels of two frames according to an embodiment of the present invention.

Referring to FIG. 4 , the control module 23 includes a pose extracting unit 210 , a depth extracting unit 220 , and a 3D map generating unit 230 .

The pose derivation unit 210 is for obtaining a pose of the augmented reality device 10 . The pose derivation unit 210 extracts a feature point representing the same object in each of the main frame and the sub frame of the stereo frame of the image captured through the camera unit 11 of the augmented reality device 10, and extracts the extracted Pose information is calculated by calculating a change in coordinates of a feature point.

For example, it is assumed that the first pose Pose1 corresponding to the main frame MF and the second pose Pose2 corresponding to the sub frame SF are as shown in FIG. 5 . In this case, a screen example of the main frame MF and the sub-frame SF is shown in FIG. 6 . For example, in FIG. 6 , it is assumed that a feature point P indicating the same object is extracted from the main frame MF and the sub-frame SF. It can be seen that there is a difference between the coordinates of the feature point P(t-1) in the main frame MF and the coordinates of the feature point P(t) in the sub frame SF. Accordingly, the pose derivation unit 210 extracts a feature point P representing the same object, and the coordinates of the feature point P(m) in the main frame MF of the extracted feature point P and the feature point P(s) in the sub frame SF. Pose information can be derived by changing the coordinates of . In this way, the pose derivation unit 210 derives pose information (position, rotation information) by calculating the change in feature points, and expresses the pose information as a matrix to derive a pose matrix.

The depth derivation unit 220 is for obtaining a depth map using a deep learning model (DLM). The depth derivation unit 220 generates a deep learning model (DLM) by using learning data including a main frame for learning (MF) and a sub-frame for learning (SF), and such a method is as follows. As shown in FIG. 7 , the depth derivation unit 220 first derives a transition matrix (TM) using a pose matrix (PM) and a camera matrix (CM). Here, the pose matrix PM is a matrix expression of pose information derived from the main frame MF for learning and the subframe SF for learning by the pose derivation unit 210 . In addition, the camera matrix CM is an internal parameter of the camera unit 11 of the augmented reality device 10 . This camera matrix (CM) is received in advance from the augmented reality device 10 is stored in the storage module (220). And, as shown in FIG. 8, the depth derivation unit 220 transforms the main frame MF for learning using a transformation matrix TM to simulate the subframe SF for learning (SF') create And, as shown in FIG. 9, the depth derivation unit 220 includes a pixel of a sub-frame for learning (SF) and a pixel of a sub-frame (SF') for imitation of a part representing the same object in the real world with respect to the prototype of the model. A deep learning model that learns the correlation between the coordinate difference and the depth (deep learning) and derives a depth map according to the coordinate difference between the pixels of the sub-frame for learning (SF) and the pixels of the sub-frame (SF') for simulation learning (DLM) is created.

As described above, after generating the deep learning model (DLM), the depth derivation unit 220 receives the main frame MF and the sub frame SF from the augmented reality device 10 through the communication module 21 . When this is input, as shown in FIG. 7 , a transformation matrix TM is derived using the pose matrix PM and the camera matrix CM. Then, as shown in FIG. 8 , the depth derivation unit 220 converts the main frame MF using the transformation matrix TM to obtain a replica sub-frame SF' that mimics the sub-frame SF. create Next, the depth derivation unit 220 inputs the sub-frame SF and the simulated sub-frame SF' to the deep learning model DLM. Then, the deep learning model (DLM) derives a depth map according to the difference in coordinates between the pixels of the sub-frame SF and the pixels of the simulated sub-frame SF'.

The 3D map generator 230 derives a 3D map from the frame of the photographed image by using the pose information obtained by the pose derivation unit 210 and the depth map derived by the depth derivation unit 220 . That is, the position and rotation information between the main frame MF and the sub-frame SF of the augmented reality device 10 can be known from the pose information, and the depth between the main frame MF and the sub-frame F1 through the depth map. can be known, the 3D map generator 230 may generate a 3D map by converting the 2D coordinates of the pixels of the corresponding frame into 3D coordinates using the position and rotation information and the depth.

Next, a method for generating a three-dimensional map using a stereo frame of the lightweight augmented reality device 10 according to an embodiment of the present invention will be described. 10 is a flowchart illustrating a method for generating a 3D map using a stereo frame of a lightweight augmented reality device according to an embodiment of the present invention. In the embodiment of FIG. 10 , as described above, it is assumed that a deep learning model (DLM) that derives a depth map according to a difference in pixel coordinates of a plurality of frames by deep learning the model prototype is generated. .

Referring to FIG. 10 , the communication module 21 receives a plurality of frames including a main frame and a sub frame of a stereo image from the augmented reality device 10 in step S110 .

Then, the pose derivation unit 210 derives the pose information and the pose matrix based on the main frame MF and the sub frame SF in step S120 . That is, with reference to FIGS. 5 and 6 , the pose derivation unit 210 extracts a feature point P indicating the same object in each of the main frame MF and the sub frame SF, and the main frame MF of the extracted feature point P. Pose information can be derived through the change of the coordinates of the feature point P(m) in ) and the coordinates of the feature point P(s) in the sub-frame SF. In this way, the pose derivation unit 210 derives pose information (position, rotation information) by calculating the change in feature points, and expresses the pose information as a matrix to derive a pose matrix.

Next, the depth derivation unit 220 derives a depth map from the main frame MF and the sub-frame SF using the pose matrix and the deep learning model (DLM) in step S130 . The step S130 will be described in more detail as follows.

The depth derivation unit 220 is first, as shown in FIG. 7 , the pose matrix PM and the augmented reality device 10 derived from the main frame MF and the sub frame SF by the pose derivation unit 210 . ), a transformation matrix TM is derived using the known camera matrix CM for the camera part 11 . Then, as shown in FIG. 8 , the depth derivation unit 220 converts the main frame MF using the transformation matrix TM to obtain a replica sub-frame SF' that mimics the sub-frame SF. create Next, the depth derivation unit 220 inputs the sub-frame SF and the simulated sub-frame SF' to the deep learning model DLM. Then, the deep learning model (DLM) may derive a depth map according to the difference in coordinates between the pixels of the sub-frame SF and the pixels of the simulated sub-frame SF'.

Next, the 3D map generator 230 uses the pose information derived by the pose derivation unit 210 and the depth map derived from the depth derivation unit 220 in step S140 to provide a three-dimensional map of the frame of the photographed image. create That is, the position and rotation information between the main frame MF and the sub-frame SF of the augmented reality device 10 can be known from the pose information, and the depth between the main frame MF and the sub-frame F1 through the depth map. can be known, the 3D map generator 230 may convert the 2D coordinates of the pixels of the corresponding frame into 3D coordinates using the position and rotation information and the depth.

Next, the 3D map generation unit 230 transmits the generated 3D map to the augmented reality device 10 through the communication module 21 in step S150 . Accordingly, the control unit 18 of the augmented reality device 10 may receive the 3D map through the communication unit 11 . Then, the controller 180 calls the virtual object stored in the storage unit 17 in step S160 and matches the virtual object to the image captured by the camera unit 11, but matches the virtual object according to the received 3D map. to create an augmented reality image. Then, the control unit 18 may output the augmented reality image through the display unit 16 in step S170.

According to the present invention, a virtual object can be registered with an image captured by using the 3D map derived as described above. Since the 3D map of the present invention provides precise 3D coordinates, precise registration is possible when registering virtual objects. Accordingly, it is possible to provide augmented reality with higher realism. Moreover, since the three-dimensional map of the present invention is generated by the service server 20, not directly derived by the augmented reality device 10, the augmented reality device 10 can be reduced in weight.

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level languages that can be executed by a computer using an interpreter or the like as well as machine language such as generated by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

Claims (8)

In the service server for generating a 3D map using a stereo frame, a communication module for continuously receiving a plurality of frames including a main frame and a sub-frame of an image taken from the augmented reality device; a pose derivation unit for obtaining pose information from the main frame and the sub-frame; a depth map calculation unit for deriving a depth map from the main frame and the sub frame using a deep learning model; and a 3D map generator for generating a 3D map based on the pose information and the depth map, and transmitting the generated 3D map to the augmented reality device through the communication module; characterized in that it comprises A service server for creating 3D maps. According to claim 1, The pose derivation unit extracting feature points representing the same object from the main frame and the sub-frame; Pose information and pose matrix are sequentially derived through a change between the coordinates of the feature point in the main frame and the coordinates of the feature point in the sub frame. A service server for creating 3D maps. 3. The method of claim 2, The depth derivation part Deriving a transformation matrix using the known camera matrix and the pose matrix of the augmented reality device, generating an imitation sub-frame simulating the sub-frame from the main frame using the transformation matrix; Through the deep learning model, characterized in that the depth map is derived according to the coordinate difference between the pixel of the sub-frame and the pixel of the imitation sub-frame A device for generating three-dimensional maps. According to claim 1, The depth derivation part Derive a transformation matrix using the known camera matrix and the pose matrix derived from the training main frame and the training subframe, generating a sub-frame for imitation learning that simulates the sub-frame for training from the main frame for training using the transformation matrix; For the prototype of the model, by learning the correlation between the coordinate difference and the depth of the pixel of the sub-frame for learning of the part representing the same object in the real world and the pixel of the sub-frame for imitation learning, the pixel of the sub-frame for training and the sub-frame for imitation learning Characterized in generating a deep learning model that derives a depth map according to the difference in coordinates of pixels of a frame A device for generating three-dimensional maps. In the method for generating a 3D map using the stereo frame of the service server, deriving pose information from the main frame and the sub-frame when the pose derivation unit receives a plurality of stereo frames including the main frame and the sub-frames of the image taken from the augmented reality device through the communication module; deriving, by a depth map calculation unit, a depth map from the main frame and the sub-frame using a deep learning model; generating, by a 3D map generator, a 3D map based on the pose information and the depth map; and a three-dimensional map generator for transmitting the three-dimensional map generated by the three-dimensional map generator to the augmented reality device through the communication module; characterized in that it comprises A method for creating a three-dimensional map. 6. The method of claim 5, The step of deriving pose information from the main frame and the sub-frame comprises: extracting, by the pose extracting unit, a feature point representing the same object in each of the main frame and the sub-frame; and sequentially deriving, by the pose derivation unit, pose information and a pose matrix through a change between the coordinates of the feature point in the main frame and the coordinates of the feature point in the sub frame; characterized in that it comprises A method for creating a three-dimensional map. 7. The method of claim 6, The step of generating a 3D map based on the pose information and the depth map comprises: deriving a transformation matrix by the depth derivation unit using the known camera matrix and the pose matrix of the augmented reality device; generating, by the depth derivation unit, a simulating sub-frame that mimics a sub-frame from the main frame by using the transform matrix; and deriving, by the depth derivation unit, a depth map according to the difference in coordinates between the pixels of the sub-frame and the pixels of the simulated sub-frame through the deep learning model; characterized in that it comprises A method for creating a three-dimensional map. 6. The method of claim 5, Before the step of deriving the pose information from the main frame and the sub-frame, deriving a transformation matrix by the depth derivation unit using a pose matrix derived from a main frame for learning and a sub-frame for learning and a known camera matrix; generating, by the depth derivation unit, a sub-frame for imitation learning by using the transformation matrix to simulate the sub-frame for training from the main frame for training; and The depth derivation unit learns the correlation between the coordinate difference and the depth of the pixel of the sub-frame for learning of the part representing the same object in the real world with respect to the prototype of the model and the pixel of the sub-frame for imitation learning, and the pixel of the sub-frame for learning generating a deep learning model for deriving a depth map according to a difference in coordinates of pixels of the sub-frame for simulation learning; characterized in that it further comprises A method for creating a three-dimensional map.

Images