KR102870997B1 - Computing device for indoor odometry and method of operation thereof - Google Patents

Abstract

The present disclosure relates to a computing device for indoor odometry and an operating method thereof, which can be configured to estimate an indirect movement motion due to boarding a vehicle and a position change amount by the vehicle corresponding to the indirect movement motion by using surrounding space information and inertial measurement information, and to detect state information based on the position change amount.

Description

Computing device for indoor odometry and method of operation thereof {Computing device for indoor odometry and method of operation therefor}

The present disclosure relates to a computing device for indoor odometry and a method of operating the same.

Indoor odometry is a technology that estimates the state information of a device by using the sensing information acquired while the device moves in an indoor environment. Typically, the device acquires sensing information from at least one of a camera or a lidar and an inertial sensor, and an indoor odometry algorithm estimates state information from the sensing information. For example, indoor odometry algorithms include the visual inertial odometry (VIO) algorithm, which uses sensing information from a camera and an inertial sensor, the lidar inertial odometry (LIO) algorithm, which uses sensing information from a lidar sensor and an inertial sensor, and the visual lidar inertial odometry (VLIO) algorithm, which uses sensing information from a camera, a lidar sensor, and an inertial sensor.

However, the above indoor odometry algorithm has a problem in that it cannot accurately estimate state information for a specific motion of the device. More specifically, for a motion in which the device moves by a means of transportation, such as an elevator, escalator, or moving walkway, the indoor odometry algorithm detects the stationary state of the device based on sensing information from a camera or lidar. This is because the surrounding spatial features generally do not change while the device is in the means of transportation. Meanwhile, for the corresponding motion, the indoor odometry algorithm detects the movement of the device based on sensing information from an inertial sensor. This is because the device moves at a constant speed along with the means of transportation while in the means of transportation. Thus, the simultaneous detection of the stationary and moving state of the device leads to a failure in estimating the state information.

The present disclosure provides a computing device and an operating method thereof for indoor odometry for estimating state information with high accuracy for all motions.

The present disclosure provides a computing device and an operating method thereof for indoor odometry for estimating state information with high accuracy for not only direct movement motion but also indirect movement motion due to boarding a vehicle.

In the present disclosure, an operating method of a computing device for indoor odometry may include a step of estimating an indirect movement motion according to boarding a vehicle and a position change amount by the vehicle corresponding to the indirect movement motion, using surrounding space information and inertial measurement information, and a step of detecting state information according to the position change amount.

In the present disclosure, a computing device for indoor odometry includes a memory, and a processor connected to the memory and configured to execute at least one command stored in the memory, wherein the processor may be configured to estimate an indirect movement motion according to boarding a vehicle and a position change amount by the vehicle corresponding to the indirect movement motion by using surrounding space information and inertial measurement information, and to detect state information according to the position change amount.

According to the present disclosure, a computing device can distinguish between direct and indirect movement motions by estimating motion types using surrounding spatial information and inertial measurement information. That is, the computing device can identify indirect movement motions by a vehicle even if surrounding spatial features do not change. Furthermore, the computing device can estimate state information for indirect movement motions by estimating the amount of positional change by the vehicle corresponding to the indirect movement motion. As a result, the computing device can estimate state information with high accuracy for both direct and indirect movement motions. Therefore, indoor odometry performance can be improved in the computing device.

FIG. 1 is a block diagram illustrating a computing device for indoor odometry according to various embodiments.
Figure 2 is a block diagram illustrating the processor of Figure 1.
FIG. 3 is a flowchart illustrating an operation method of a computing device for indoor odometry according to various embodiments.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions of widely known functions or configurations will be omitted if they may unnecessarily obscure the gist of the present disclosure.

In the attached drawings, identical or corresponding components are assigned the same reference numerals. Furthermore, in the description of the embodiments below, duplicate descriptions of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

The advantages and features of the disclosed embodiments, and methods for achieving them, will become clearer with reference to the embodiments described below, along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure the completeness of the disclosure and to fully inform those skilled in the art of the scope of the invention.

The terms used in this disclosure will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this disclosure have been selected from widely used and common terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of engineers working in the relevant field, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant invention. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

In this disclosure, singular expressions include plural expressions unless the context clearly dictates otherwise. Furthermore, plural expressions include singular expressions unless the context clearly dictates otherwise. Throughout the specification, when a part is said to include a certain component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated.

In addition, the term 'module' or 'part' used in the present disclosure means a software or hardware component, and the 'module' or 'part' performs certain roles. However, the 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to be on an addressable storage medium and may be configured to play one or more processors. Thus, as an example, the 'module' or 'part' may include at least one of components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and 'modules' or 'parts' may be combined into a smaller number of components and 'modules' or 'parts', or may be further separated into additional components and 'modules' or 'parts'.

According to the present disclosure, a 'module' or 'unit' may be implemented as a processor and a memory. 'Processor' should be broadly construed to include a general purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some circumstances, a 'processor' may also refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), and the like. A 'processor' may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such combination of configurations. In addition, 'memory' should be broadly construed to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. Memory is said to be in electronic communication with the processor if the processor can read information from, and/or write information to, the memory. Memory integrated in a processor is in electronic communication with the processor.

Hereinafter, the present disclosure provides a computing device (100) for indoor odometry for estimating state information with high accuracy for all motions, and an operating method thereof. The computing device (100) may be implemented in a device such as a device that is carried or worn by a user and can be moved by the user, a robot capable of autonomous driving, or may be implemented in a server that communicates with such a device. The types of motions may include direct movement motion of the device, indirect movement motion due to boarding a means of transportation, and stationary motion. Here, the means of transportation may include, for example, an elevator, an escalator, a moving walkway, etc.

FIG. 1 is a block diagram illustrating a computing device (100) for indoor odometry according to various embodiments.

Referring to FIG. 1, the computing device (100) may include at least one of a camera module (110), a sensor module (120), a communication module (130), an input module (140), an output module (150), a memory (160), or a processor (170). In some embodiments, at least one of the components of the computing device (100) may be omitted. In some embodiments, at least one other component may be added to the computing device (100). In some embodiments, at least two of the components of the computing device (100) may be implemented as a single integrated circuit.

The camera module (110) can capture images of the surrounding environment of the computing device (100). For example, the images can include moving images and still images. According to one embodiment, the camera module (110) can include at least one lens, at least one image sensor, an image signal processor, or a flash.

The sensor module (120) can detect the status of the computing device (100) and generate an electrical signal or data value therefor. In various embodiments, the sensor module (120) includes an inertial measurement unit (IMU), and the inertial measurement unit can generate inertial measurement information. For example, the inertial measurement unit can include an accelerometer, a gyrometer, a magnetometer, and an altimeter. Additionally, the sensor module (120) can further include at least one of a lidar sensor, a radar sensor, an infrared (IR) sensor, a distance sensor, a gesture sensor, a barometric pressure sensor, a magnetic sensor, a grip sensor, a proximity sensor, a color sensor, an infrared sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The communication module (130) can communicate with an external device in the computing device (100). For example, the external device may include at least one of a device, a base station, a server, or a satellite. The communication module (130) may include at least one of a wired communication module and a wireless communication module. The wired communication module may be wired and connected to the external device via a connection terminal (not shown) to communicate with the external device via a wire. The wireless communication module may include at least one of a short-range communication module and a long-range communication module. The short-range communication module may communicate with the external device via a short-range communication method. For example, the short-range communication method may include at least one of Bluetooth, Wi-Fi Direct, or infrared communication. The long-range communication module may communicate with the external device via a long-range communication method. Here, the long-range communication module may communicate with the external device via a network. For example, the network may include at least one of a cellular network, the Internet, or a computer network such as a LAN or WAN.

The input module (140) can input a signal to be used in at least one component of the computing device (100). In some embodiments, the input module (140) can include at least one of a microphone, a mouse, or a keyboard. In other embodiments, the input module (140) can include at least one of touch circuitry configured to detect a touch, or a sensor circuitry configured to measure the intensity of a force generated by a touch.

The output module (150) can output information from the computing device (100). At this time, the output module (150) can include at least one of a display module for visually displaying information or an audio module for audibly reproducing information. For example, the display module can include at least one of a display, a holographic device, or a projector. As an example, the display module can be implemented as a touch screen by being assembled with at least one of a touch circuit or a sensor circuit of the input module (140). For example, the audio module can include at least one of a speaker, a receiver, an earphone, or a headphone.

The memory (160) can store various data used by at least one component of the computing device (100). For example, the memory (160) can include at least one of volatile memory and non-volatile memory. The data can include at least one program and input data or output data related thereto. The program can be stored in the memory (160) as software including at least one command.

The processor (170) can execute a program in the memory (160) to control at least one component of the computing device (100). Through this, the processor (170) can perform data processing or calculation. Here, the processor (170) can execute a command stored in the memory (160). In various embodiments, the processor (170) can perform indoor odometry. In some embodiments, when the computing device (100) is implemented in a device such as a device that is portable or worn by a user and can be moved by the user or a robot capable of autonomous driving, at least one of the camera module (110) or the sensor module (120) may be disposed in the computing device (100) and connected to the processor (170). In other embodiments, when the computing device (100) is implemented in a server, at least one of the camera module (110) or the sensor module (120) may be disposed in the device, and the processor (170) may be connected to the device via the communication module (130).

FIG. 2 is a block diagram illustrating the processor (170) of FIG. 1.

Referring to FIG. 2, the processor (170) may include an odometry module (210), a motion classification module (220), and a state estimation module (230).

Each of the odometry module (210) and the motion classification module (220) may receive surrounding spatial information and inertial measurement information. The surrounding spatial information represents surrounding spatial features and may be generated from at least one of an image from the camera module (110) or lidar data from the sensor module (120), particularly, a lidar sensor. The inertial measurement information may be generated from the sensor module (120), particularly, an inertial measurement unit.

The odometry module (210) can measure odometry information using surrounding spatial information and inertial measurement information. Specifically, the odometry module (210) can accumulate surrounding spatial information and inertial measurement information that are continuously acquired, and analyze the accumulated surrounding spatial information and inertial measurement information to measure odometry information. For example, the odometry module (210) may be a VIO module (see Qin, Tong, Peiliang Li, and Shaojie Shen. "Vins-mono: A robust and versatile monocular visual-inertial state estimator." IEEE Transactions on Robotics 34.4 (2018): 1004-1020) that uses images and inertial measurement information from a camera module (110) that are continuously acquired. In this case, the odometry module (210) can measure odometry information by combining spatial features generated from images and inertial measurement information. As another example, the odometry module (210) may be a LIO module that uses continuously acquired LIDAR data and inertial measurement information from a LIDAR sensor. In this case, the odometry module (210) may measure odometry information by combining spatial features generated from the LIDAR data and inertial measurement information. As another example, the odometry module (210) may be a VLIO module that uses continuously acquired images, sensing data from a LIDAR sensor, and inertial measurement information. In this case, the odometry module (210) may measure odometry information by combining spatial features generated from images and LIDAR data and inertial measurement information.

The motion classification module (220) can classify motion types using surrounding spatial information and inertial measurement information. In some embodiments, the motion classification module (220) can be implemented as a pre-trained neural network. The motion types can be classified into direct movement motion of the device, indirect movement motion due to boarding a vehicle, or stationary motion. Specifically, the motion classification module (220) can analyze continuously acquired surrounding spatial information to determine whether spatial features have changed. In addition, the motion classification module (220) can analyze continuously acquired inertial measurement information to determine whether the inertial measurement information has changed. In this case, if the spatial features do not change but the inertial measurement information has changed, the motion classification module (220) can detect indirect movement motion as the motion type. On the other hand, if both the spatial features and the inertial measurement information have changed, the motion classification module (220) can detect direct movement motion as the motion type. Meanwhile, if both spatial features and inertial measurement information remain unchanged, the motion classification module (220) can detect stationary motion as a motion type.

In addition, the motion classification module (220) can estimate the amount of position change using surrounding space information and inertial measurement information. Specifically, if the motion type is an indirect movement motion, the motion classification module (220) can detect a temporal section of the indirect movement motion from continuously acquired surrounding space information. In addition, the motion classification module (220) can estimate the amount of position change for the corresponding temporal section from continuously acquired inertial measurement information. The motion classification module (220) can detect the start and end points of the corresponding temporal section, and calculate the amount of position change from inertial measurement information corresponding to the start and end points, respectively. Here, the motion classification module (220) can estimate the amount of position change using angular velocity and linear acceleration among the inertial measurement information.

As described above, the motion classification module (220) can be implemented as a pre-trained neural network. The neural network can be trained based on the following. The input value to the neural network can include surrounding spatial information of a representative point in time within the temporal section of the indirect movement motion, and all inertial measurement information in the temporal section. For example, the surrounding spatial information can include at least one of an image captured at a representative point in time or lidar data detected at a representative point in time. Here, the input value can include angular velocity and linear acceleration among the inertial measurement information. The output value to the neural network can include the position change amount (dx, dy, dz) between the start and end points of the temporal section, and parameters (ux, uy, uz) constituting the uncertainty about the position change amount (here, covariance is ), and the probability of the label that determines the motion type (probability of direct movement motion, probability of indirect movement motion by means of transportation, probability of stationary motion). As training data for the neural network, the actual value for the label of the motion type can be manually generated, and the actual value for the position change amount can be obtained from the result of simultaneous localization and map-building (SLAM). Here, the actual value for the label of the motion type can be generated differently for each means of transportation, and thereby the neural network can be trained to distinguish indirect movement motion by means of transportation. In training the neural network, a cross entropy loss function such as [Mathematical Formula 1 below] can be utilized as a loss function for training the motion type, and a log likelihood loss function that considers covariance such as [Mathematical Formula 2 below] can be utilized as a loss function for training the position change amount.

The state estimation module (230) can detect state information differently depending on the motion type. In other words, the state estimation module (230) can detect state information by complementarily using the odometry module (210) and the motion classification module (220) depending on the motion type. The state information can include at least one of rotation (R), velocity (v), position (p), or bias (b). Here, the bias can include at least one of the bias (b _a ) of an accelerometer or the bias (b _g ) of a geomagnetic field. In this case, the state estimation module (230) can use either the position change amount from the motion classification module (220) or the odometry information from the odometry module (210) in a switch manner depending on the motion type. In some embodiments, the state estimation module (230) may be implemented as a Kalman filter (see Liu, Wenxin, et al. "Tlio: Tight learned inertial odometry." IEEE Robotics and Automation Letters 5.4 (2020): 5653-5660).

Specifically, the state estimation module (230) may include a prediction module (231), a switch module (233), and a correction module (235). As illustrated in FIG. 2, the switch module (233) may be configured as an internal element of the state estimation module (230), but is not limited thereto. That is, although not illustrated, the switch module (233) may be configured as an external element of the state estimation module (230).

The prediction module (231) can predict temporary state information using inertial measurement information. At this time, the prediction module (231) can estimate the temporary state information without considering the motion type. For example, the prediction module (231) can estimate the temporary state information as shown in [Mathematical Formula 3] below.

The switch module (233) can be driven according to the motion type. Specifically, the switch module (233) is connected to the motion classification module (220) and can control the connection between the motion classification module (220) and the correction module (235), and the connection between the odometry module (210) and the correction module (235). If the motion type is an indirect movement motion, the switch module (233) can connect the motion classification module (220) and the correction module (235), and separate the odometry module (210) and the correction module (235). In this case, the position change amount from the motion classification module (220) can be provided to the correction module (235). On the other hand, if the motion type is a direct movement motion, the switch module (233) can separate the motion classification module (220) and the correction module (235), and connect the odometry module (210) and the correction module (235). In this case, odometry information from the odometry module (210) can be provided to the correction module (235). Meanwhile, if the motion type is a stationary motion, the switch module (233) can separate the motion classification module (220) and the correction module (235), and separate the odometry module (210) and the correction module (235).

The correction module (235) can detect the final state information by correcting the predicted temporary state information using the inertial measurement information. In other words, the correction module (235) can correct the temporary state information output from the prediction module (231). Specifically, when connected to the motion classification module (220) by the switch module (233) and separated from the odometry module (210), the correction module (235) can update the position in the temporary state information according to the position change amount from the motion classification module (220). For example, the correction module (235) can calculate the displacement according to the position change amount as in [Mathematical Formula 4] below and update the position based on the displacement. Meanwhile, when separated from the motion classification module (220) by the switch module (233) and connected to the odometry module (210), the correction module (235) can update at least one of the rotation, the speed, or the position in the temporary state information according to the odometry information. For example, the correction module (235) can update the speed and position as in [Mathematical Formula 5] below. Meanwhile, when separated from the motion classification module (220) and the odometry module (210) by the switch module (233), the correction module (235) can maintain temporary state information. As a result, the correction module (235) can output final state information.

Here, can represent the displacement between two points in time.

FIG. 3 is a flowchart illustrating an operation method of a computing device (100) for indoor odometry according to various embodiments.

Referring to FIG. 3, the computing device (100) can acquire surrounding space information and inertial measurement information in step 310. Specifically, the processor (170) can acquire surrounding space information and inertial measurement information that are continuously acquired. The surrounding space information and inertial measurement information are input to each of the odometry module (210) and the motion classification module (220), and the inertial measurement information can also be input to the state estimation module (230). The surrounding space information represents spatial features of the surroundings and can be generated from at least one of an image from the camera module (110) or lidar data from the sensor module (120), particularly, a lidar sensor. The inertial measurement information can be generated from the sensor module (120), particularly, an inertial measurement unit.

In some embodiments, when the computing device (100) is implemented in a device such as a device that is carried or worn by a user and can be moved by the user or a robot capable of autonomous driving, at least one of the camera module (110) or the sensor module (120) may be disposed in the computing device (100) and connected to the processor (170). Thus, the processor (170) may collect surrounding space information and inertial measurement information within the computing device (100). In other embodiments, when the computing device (100) is implemented in a server, at least one of the camera module (110) or the sensor module (120) may be disposed in the device, and the processor (170) may be connected to the device via the communication module (130). Thus, the processor (170) may receive surrounding space information and inertial measurement information from the device via the communication module (130).

Next, the computing device (100) can estimate the motion type and position change amount using the surrounding spatial information and inertial measurement information in step 320. In some embodiments, the motion classification module (220) can be implemented as a pre-trained neural network. The motion type can be classified into direct movement motion of the device, indirect movement motion due to boarding a transportation vehicle, or stationary motion.

Specifically, the motion classification module (220) can analyze continuously acquired surrounding spatial information to determine whether spatial features have changed. In addition, the motion classification module (220) can analyze continuously acquired inertial measurement information to determine whether the inertial measurement information has changed. At this time, if the spatial features do not change but the inertial measurement information has changed, the motion classification module (220) can detect indirect movement motion as a motion type. Meanwhile, if both the spatial features and the inertial measurement information have changed, the motion classification module (220) can detect direct movement motion as a motion type. Meanwhile, if both the spatial features and the inertial measurement information have not changed, the motion classification module (220) can detect stationary motion as a motion type.

In addition, the computing device (100) can estimate the position change amount using the surrounding space information and inertial measurement information in step 320. Specifically, if the motion type is an indirect movement motion, the motion classification module (220) can detect the temporal section of the indirect movement motion from the continuously acquired surrounding space information. Then, the motion classification module (220) can estimate the position change amount for the corresponding temporal section from the continuously acquired inertial measurement information. The motion classification module (220) can detect the start and end points of the corresponding temporal section, and calculate the position change amount from the inertial measurement information corresponding to the start and end points, respectively. Here, the motion classification module (220) can estimate the position change amount using the angular velocity and linear acceleration among the inertial measurement information.

Next, the computing device (110) can determine whether the motion type is an indirect movement motion in step 330. If the motion type is determined to be an indirect movement motion in step 330, the computing device (110) can detect state information based on the position change amount in step 340. That is, the state estimation module (230) can detect final state information by correcting the temporary state information predicted using inertial measurement information according to the position change amount from the motion classification module (220). Specifically, the prediction module (231) can predict the temporary state information using inertial measurement information. At this time, the prediction module (231) can estimate the temporary state information without considering the motion type. Meanwhile, if the motion classification module (220) detects an indirect movement motion as the motion type, the switch module (233) can connect the motion classification module (220) and the correction module (235), and separate the odometry module (210) and the correction module (235). In this case, the position change amount from the motion classification module (220) can be provided to the correction module (235). The correction module (235) can update the position in the temporary state information according to the position change amount from the motion classification module (220).

Meanwhile, if it is determined in step 330 that the motion type is not an indirect movement motion, the computing device (100) can determine in step 350 whether the motion type is a direct movement motion. At this time, if it is determined in step 350 that the motion type is a direct movement motion, the computing device (110) can detect state information according to the odometry information from the odometry module (210) at step 360. That is, the state estimation module (230) can detect final state information by correcting the temporary state information predicted using the inertial measurement information according to the odometry information from the odometry module (210). Specifically, the prediction module (231) can predict the temporary state information using the inertial measurement information. At this time, the prediction module (231) can estimate the temporary state information without considering the motion type. Meanwhile, if the motion type is a direct movement motion, the switch module (233) may separate the motion classification module (220) and the correction module (235), and connect the odometry module (210) and the correction module (235). In this case, odometry information from the odometry module (210) may be provided to the correction module (235). The correction module (235) may update at least one of the rotation, speed, or position in the temporary state information according to the odometry information.

Meanwhile, if it is determined at step 350 that the motion type is not a direct movement motion, the computing device (100) may determine that the motion type is a stationary motion. In this case, the computing device (100) may detect state information using inertial measurement information. Specifically, the prediction module (231) may predict temporary state information using inertial measurement information. At this time, the prediction module (231) may estimate the temporary state information without considering the motion type. Meanwhile, if the motion type is a stationary motion, the switch module (233) may separate the motion classification module (220) and the correction module (235), and separate the odometry module (210) and the correction module (235). As a result, the correction module (235) may maintain the temporary state information and output the final state information.

According to the present disclosure, a computing device (100) can distinguish between direct and indirect movement motions by estimating motion types using surrounding spatial information and inertial measurement information. That is, the computing device (100) can identify indirect movement motions by a means of transportation even if surrounding spatial features do not change. In addition, the computing device (100) can estimate state information for indirect movement motions by estimating the amount of position change by the means of transportation corresponding to the indirect movement motion. As a result, the computing device (100) can estimate state information with high accuracy for not only direct movement motions but also indirect movement motions. Therefore, the indoor odometry performance of the computing device (100) can be improved.

In summary, the present disclosure provides a computing device (100) for indoor odometry and a method of operating the same.

The method of operating a computing device (100) for indoor odometry of the present disclosure may include a step (step 320) of estimating an indirect movement motion according to boarding a vehicle and a position change amount by the vehicle corresponding to the indirect movement motion, using surrounding space information and inertial measurement information, and a step (step 340) of detecting state information according to the position change amount.

In the present disclosure, the step of estimating indirect movement motion and position change amount (step 320) may include a step of classifying a motion type using surrounding space information and inertial measurement information, and a step of estimating a position change amount using inertial measurement information when the motion type is indirect movement motion.

In the present disclosure, the step of detecting state information according to the amount of change in position (step 340) may include a step of detecting state information by updating a predicted position according to the amount of change in position using inertial measurement information.

In the present disclosure, the operating method of the computing device (100) may further include a step (step 360) of detecting state information based on odometry information measured using surrounding space information and inertial measurement information, if the motion type is direct movement motion.

In the present disclosure, the step of detecting state information according to odometry information (step 360) may include a step of detecting state information by updating at least one of rotation, speed, or position predicted using inertial measurement information according to the odometry information.

In the present disclosure, the step of estimating indirect movement motion and position change amount (step 320) can be performed through a pre-learned neural network.

In the present disclosure, the step of detecting state information according to the amount of change in position (step 340) and the step of detecting state information according to odometry information (step 360) can be performed through a Kalman filter.

In the present disclosure, the surrounding spatial information may include spatial features generated from continuous images or lidar data.

In the present disclosure, the step of estimating indirect movement motion and position change amount (step 320) can estimate the position change amount using angular velocity and linear acceleration among inertial measurement information.

A computing device (100) for indoor odometry of the present disclosure includes a memory (160), and a processor (170) connected to the memory (160) and configured to execute at least one command stored in the memory (160), wherein the processor (170) may be configured to estimate an indirect movement motion according to boarding a vehicle and an amount of position change by the vehicle corresponding to the indirect movement motion, using surrounding space information and inertial measurement information, and to detect state information according to the amount of position change.

In the present disclosure, the processor (170) may include a motion classification module (220) configured to classify a motion type using surrounding space information and inertial measurement information, and to estimate a position change amount using inertial measurement information if the motion type is an indirect movement motion, and a state estimation module (230) configured to detect state information according to a position change amount if the motion type is an indirect movement motion.

In the present disclosure, the processor (170) may further include an odometry module (210) configured to measure odometry information using surrounding space information and inertial measurement information.

In the present disclosure, the state estimation module (230) may be configured to detect state information by updating a predicted position using inertial measurement information according to a position change amount when the motion type is an indirect movement motion, and to detect state information by updating at least one of rotation, speed, or position predicted using inertial measurement information according to odometry information when the motion type is a direct movement motion.

In the present disclosure, the state estimation module (230) can use either the position change amount from the motion classification module (220) or the odometry information from the odometry module (210) in a switch manner depending on the motion type.

In the present disclosure, the motion classification module (220) can be implemented as a pre-trained neural network.

In the present disclosure, the state estimation module (230) can be implemented as a Kalman filter.

In the present disclosure, the surrounding spatial information may include spatial features generated from continuous images or lidar data.

In the present disclosure, the computing device (100) may further include at least one of a camera module (110) configured to capture images or a lidar sensor configured to generate lidar data, and an inertial measurement unit configured to generate inertial measurement information.

In the present disclosure, the motion classification module (220) can be configured to estimate the position change amount by using angular velocity and linear acceleration among inertial measurement information.

The above-described method may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may be one that continuously stores a computer-executable program, or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software will depend on the particular application and design requirements imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementations should not be construed as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, a computer, or a combination thereof.

Accordingly, the various exemplary logical blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

In a firmware and/or software implementation, the techniques may be implemented as instructions stored on a computer-readable medium, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, a compact disc (CD), a magnetic or optical data storage device, etc. The instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described herein.

While the embodiments described above have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include personal computers, network servers, and portable devices.

While this disclosure has been described with reference to certain embodiments, various modifications and variations can be made without departing from the scope of the present disclosure, which would be apparent to those skilled in the art. Furthermore, such modifications and variations should be considered to fall within the scope of the claims appended hereto.

Claims (20)

In a method of operating a computing device for indoor odometry of a device,
A step of predicting temporary state information of the device using inertial measurement information;
A step of estimating the motion type of the device using the surrounding space information and the inertial measurement information;
A step of estimating the amount of position change by the moving means corresponding to the indirect moving motion, as the above motion type is estimated to be an indirect moving motion according to boarding of the moving means of the device; and
A step of detecting the status information of the device as final status information by updating the temporary status information based on the above location change amount.
including,
How a computing device operates.
delete In the first paragraph,
The step of detecting the above status information is:
A step of detecting the state information as the final state information by updating the location included in the temporary state information according to the amount of change in the location.
including,
How a computing device operates.
In the first paragraph,
As the above motion type is estimated to be a direct movement motion, a step of detecting the state information in the direct movement motion of the device as the final state information by updating the temporary state information based on the odometry information using the surrounding space information and the inertial measurement information.
including more,
How a computing device operates.
In paragraph 4,
Based on the above odometry information, the step of detecting the status information of the device is:
A step of detecting state information in the direct movement motion by updating at least one of rotation, speed, or position included in the temporary state information according to the odometry information.
including,
How a computing device operates.
In the first paragraph,
The estimation of the above motion type and the estimation of the position change amount are performed through a pre-learned neural network.
How a computing device operates.
In paragraph 4,
The step of detecting the state information based on the position change amount, and the step of detecting the state information in the direct movement motion based on the odometry information,
It is performed through a Kalman filter,
How a computing device operates.
In the first paragraph,
The above surrounding spatial information includes spatial features generated from at least one of continuous images or lidar data.
How a computing device operates.
In the first paragraph,
The step of estimating the above position change amount is:
Estimating the position change amount using the angular velocity and linear acceleration among the above inertial measurement information,
How a computing device operates.
A computer program stored in a non-transitory computer-readable recording medium for executing the method of any one of claims 1 and 3 to 9 on the computing device.
In a computing device for indoor odometry of a device,
memory; and
A processor connected to the memory and configured to execute at least one instruction stored in the memory,
The above processor,
Predicting the temporary state information of the device using inertial measurement information,
By using the surrounding space information and the inertial measurement information, the motion type of the device is estimated, and as the motion type is estimated to be an indirect movement motion due to the device riding a means of transportation, the amount of position change due to the means of transportation corresponding to the indirect movement motion is estimated,
Based on the above position change amount, the device is configured to detect the status information as final status information by updating the temporary status information.
Computing device.
delete In paragraph 11,
The above processor,
An odometry module configured to measure odometry information using the above-mentioned surrounding space information and the above-mentioned inertial measurement information.
including more,
Computing device.
In paragraph 13,
The above processor,
As the above motion type is estimated to be the indirect movement motion, the position included in the temporary state information is updated according to the position change amount, and the state information is detected as the final state information.
As the motion type is estimated to be a direct movement motion, at least one of rotation, speed, or position included in the temporary state information is updated according to the odometry information, and the state information in the direct movement motion is configured to be detected as the final state information.
Computing device.
In paragraph 13,
The state detection module of the processor configured to detect the above state information,
It is configured to use one of the position change amount or the odometry information in a switch manner according to the above motion type.
Computing device.
In paragraph 11,
The estimation of the above motion type and the estimation of the position change amount are performed through a pre-learned neural network.
Computing device.
In paragraph 14,
Detecting the state information based on the position change amount and detecting the state information in the direct movement motion are performed through a Kalman filter.
Computing device.
In paragraph 11,
The above surrounding spatial information includes spatial features generated from continuous images or lidar data.
Computing device.
In paragraph 18,
a camera module configured to capture the above images, or
A lidar sensor configured to generate the above lidar data
At least one of; and
An inertial measurement unit configured to generate the above inertial measurement information
including more,
Computing device.
In paragraph 11,
Among the above inertial measurement information, the position change amount is estimated using the angular velocity and linear acceleration.
Computing device.