WO2021210492A1 - Information processing device, information processing method, and program - Google Patents

Abstract

The information processing device according to an embodiment is provided with at least one memory and at least one processor. The at least one processor is configured to be capable of executing: acquiring a detection result and an environmental information, the detection result including at least one of the state of the surroundings of the information processing device or the state of the information processing device, the environmental information relating to the environment of the surroundings of the information processing device; and executing estimation of the self-location and generation of map information on the basis of the environmental information and the detection result.

Description

Information processing equipment, information processing methods, and programs

An embodiment of the present invention relates to an information processing device, an information processing method, and a program.

Conventionally, robots and the like that estimate their own position and generate map information by recognizing the position and shape of surrounding objects from the sensing result by the sensor or the captured image are known.

Japanese Unexamined Patent Publication No. 2003-015739

However, in the prior art, it may be difficult to estimate the self-position and generate map information with high accuracy.

The information processing device of the embodiment includes at least one memory and at least one processor. At least one processor acquires a detection result including either the surrounding state of the information processing device or the state of the information processing device, and environmental information about the environment around the information processing device, and the environmental information. Based on the information processing and the detection result, it is possible to estimate the self-position and generate map information.

FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing an example of a function provided in the information processing apparatus according to the first embodiment. FIG. 3 is an image diagram showing an example of tracking processing according to the first embodiment. FIG. 4 is an image diagram showing an example of the positional relationship between the information processing apparatus according to the first embodiment and surrounding objects. FIG. 5 is an image diagram showing an example of bundle adjustment according to the first embodiment. FIG. 6 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the first embodiment. FIG. 7 is a block diagram showing an example of the functions provided in the information processing apparatus according to the second embodiment. FIG. 8 is an image diagram showing an example of the positional relationship between the information processing apparatus according to the second embodiment and surrounding objects. FIG. 9 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the second embodiment. FIG. 10 is a block diagram showing an example of the functions provided in the information processing apparatus according to the third embodiment. FIG. 11 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the third embodiment. FIG. 12 is a diagram showing an example of segmentation of a captured image according to a fourth embodiment. FIG. 13 is a diagram showing an example of map information according to the second modification.

(First Embodiment)
FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus 1 according to the first embodiment. As an example, the information processing device 1 includes a main body 10, a moving device 16, an imaging device 17, and an IMU (Inertial Measurement Unit) sensor 18.

The moving device 16 is a device capable of moving the information processing device 1. As an example, the moving device 16 has a plurality of wheels and a motor for driving these wheels, and is connected to the lower part of the main body 10 so as to support the main body 10.

It is assumed that the information processing device 1 can be moved by the mobile device 16, for example, in a building under construction, a built building, a platform of a station, a factory, or the like. In the present embodiment, the case where the information processing device 1 moves in the building under construction will be described as an example.

The means of transportation of the information processing device 1 is not limited to wheels, and may be caterpillars, propellers, or the like. The information processing device 1 is, for example, a robot, a drone, or the like. In the present embodiment, the information processing device 1 is supposed to move autonomously, but the information processing device 1 is not limited to this.

The image pickup device 17 is, for example, a stereo camera in which two cameras arranged side by side are set as one set. The image pickup device 17 transmits the captured image data captured by the two cameras to the main body 10 in association with each other.

The IMU sensor 18 is a sensor in which a gyro sensor, an acceleration sensor, and the like are integrated, and measures the angular velocity and acceleration of the information processing device 1. The IMU sensor 18 sends the measured angular velocity and acceleration to the main body 10. The IMU sensor 18 may further include not only a gyro sensor and an acceleration sensor, but also a magnetic sensor, a GPS (Global Positioning System) device, and the like.

In the present embodiment, the image pickup device 17 and the IMU sensor 18 are collectively referred to as a detection unit. The detection unit may further include various sensors. As an example, the information processing device 1 may further include a distance measuring sensor such as an ultrasonic sensor or a laser scanner.

In the present embodiment, the term "detection" refers to imaging the surroundings of the information processing device 1, measuring the angular velocity or acceleration of the information processing device 1, and the distance to an object around the information processing device 1. It shall include measuring the distance. Further, in the present embodiment, the detection result by the detection unit includes at least one of the surrounding state of the information processing device 1 and the state of the information processing device 1. In other words, the detection result may include both information about the surrounding state of the information processing device 1 and the state of the information processing device 1, or may relate to the surrounding state of the information processing device 1 and the state of the information processing device 1. It may contain only one of the information.

The surrounding state of the information processing device 1 is, for example, an captured image of the surroundings of the information processing device 1, a distance measurement result of a distance between an object around the information processing device 1 and the information processing device 1. The state of the information processing device 1 is, for example, the angular velocity and acceleration measured by the IMU sensor 18. For example, the captured image captured by the imaging device 17 is an example of the detection result of the surrounding state of the information processing device 1. In the present embodiment, the detection result includes at least the captured image, but may further include other information.

As an example, the main body 10 includes a processor 11, a main storage device 12 (memory), an auxiliary storage device 14 (memory), a network interface 13, and a device interface 15, which are routed via a bus 19. It may be realized as a connected computer. The image pickup device 17 and the IMU sensor 18 may be incorporated in the main body 10.

The processor 11 is an electronic circuit (processing circuit, Processing circuit, Processing circuitry, CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), or ASIC (Processing circuit, Processing circuit, Processing circuitry, CPU (Central Processing Unit)) including a computer control device and a computing device. Application Specific Integrated Circuit), etc.) may be used. Further, the processor 11 may be a semiconductor device or the like including a dedicated processing circuit. The processor 11 is not limited to an electronic circuit using an electronic logic element, and may be realized by an optical circuit using an optical logic element. Further, the processor 11 may include a calculation function based on quantum computing.

The processor 11 can perform arithmetic processing based on the data and software (program) input from each apparatus and the like of the internal configuration of the information processing apparatus 1, and output the arithmetic result and the control signal to each apparatus and the like. The processor 11 may control each component constituting the information processing device 1 by executing an OS (Operating System) of the information processing device 1, an application, or the like.

The main storage device 12 is a storage device that stores instructions executed by the processor 11, various data, and the like, and the information stored in the main storage device 12 is read out by the processor 11. The auxiliary storage device 14 is a storage device other than the main storage device 12. Note that these storage devices mean arbitrary electronic components capable of storing electronic information, and may be semiconductor memories. The semiconductor memory may be either a volatile memory or a non-volatile memory. The storage device for storing various data in the information processing device 1 in the present embodiment may be realized by the main storage device 12 or the auxiliary storage device 14, or may be realized by the built-in memory built in the processor 11. .. The main storage device 12 or the auxiliary storage device 14 is also referred to as a storage unit.

Multiple processors may be connected (combined) to one storage device (memory), or a single processor may be connected. A plurality of storage devices (memory) may be connected (combined) to one processor. When the information processing device 1 in the present embodiment is composed of at least one storage device (memory) and a plurality of processors connected (combined) to the at least one storage device (memory), at least one of the plurality of processors One processor may include a configuration in which it is connected (combined) to at least one storage device (memory). Further, this configuration may be realized by a storage device (memory) and a processor included in a plurality of computers. Further, a configuration in which the storage device (memory) is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

The network interface 13 is an interface for connecting to the communication network 3 wirelessly or by wire. As the network interface 13, an appropriate interface such as one conforming to an existing communication standard may be used. Information may be exchanged with the external device 2 connected via the communication network 3 by the network interface 13. The communication network 3 may be any one of WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, and the information processing device 1 and the external device 2 may be used. It suffices as long as information is exchanged with. An example of WAN is the Internet, an example of LAN is IEEE802.11, Ethernet (registered trademark), etc., and an example of PAN is Bluetooth (registered trademark), NFC (Near Field Communication), etc.

The device interface 15 is an interface that directly connects to the mobile device 16, the image pickup device 17, and the IMU sensor 18. The device interface 15 is an interface that conforms to a standard such as USB (Universal Serial Bus), but is not limited thereto. Further, the device interface 15 may be further connected to an external device other than the various devices shown in FIG.

The external device 2 is, for example, a server device or the like. The external device 2 is connected to the information processing device 1 via a communication network 3.

The external device 2 of this embodiment stores the three-dimensional design information of the building in advance. The three-dimensional design information of the building is, for example, BIM (Building Information Modeling) information. BIM information includes information on the three-dimensional structure of a building and information on materials such as building materials. The three-dimensional design information of the building is not limited to BIM information, and may be 3D CAD (Computer-Aided Design) data or the like.

The three-dimensional design information is an example of environmental information in this embodiment. The environmental information is information about the environment around the information processing device 1. For example, the environmental information includes at least one of information about a building in which the information processing device 1 travels, information about a person or an object existing around the information processing device 1, information about the weather, and information about lighting. More specifically, the above-mentioned three-dimensional design information is an example of information on the building in which the information processing device 1 travels among the environmental information. The environmental information may be a combination of a plurality of types of information, or may include only one type of information. For example, in the present embodiment, the environmental information includes at least three-dimensional design information, but may further include other information regarding the environment around the information processing device 1.

Note that the term "object" in this embodiment includes structures such as walls and pillars, furniture, furniture, moving objects, temporary objects, people, and the like.

In the present embodiment, the information processing device 1 and the external device 2 are wirelessly connected, but the information processing device 1 and the external device 2 may be connected by wire. Further, the information processing device 1 does not have to be always connected to the external device 2.

Next, the functions of the information processing device 1 will be described. FIG. 2 is a block diagram showing an example of the functions included in the information processing apparatus 1 according to the first embodiment.

As shown in FIG. 2, the information processing device 1 includes an acquisition unit 101, a conversion unit 102, a SLAM (Simultaneous Localization and Mapping, or Simultaneously Localization and Mapping) processing unit 120, and a movement control unit 105. Further, the SLAM processing unit 120 includes a tracking unit 103 and a bundle adjustment unit 104.

The acquisition unit 101 acquires the detection result of the surrounding state of the information processing device 1 or the state of the information processing device 1 and the environmental information regarding the environment around the information processing device 1.

More specifically, the acquisition unit 101 acquires BIM information from the external device 2 via, for example, the network interface 13. The acquisition unit 101 stores the acquired BIM information in the auxiliary storage device 14.

Further, the acquisition unit 101 acquires an captured image from the imaging device 17 via the device interface 15. In addition, the acquisition unit 101 acquires the angular velocity and acceleration from the IMU sensor 18 via the device interface 15.

The conversion unit 102 converts the environmental information into an input value of at least one of the self-position estimation process by the SLAM processing unit 120 described later or the map information generation process. The conversion unit 102 may convert the environmental information into the input values of both the self-position estimation process and the map information generation process, or may convert only the input values of either process. In the present embodiment, the term "conversion" includes generating other information from the environmental information or extracting, acquiring or searching the information from the environmental information.

In the present embodiment, the conversion unit 102 generates the initial value of the three-dimensional coordinates (world coordinates) of the points in the three-dimensional space used in the bundle adjustment process by the bundle adjustment unit 104 described later from the BIM information. ..

For example, the conversion unit 102 has an imaging range of the imaging device 17 among the structures such as walls and columns included in the BIM information based on the current position and orientation of the imaging device 17 specified by the tracking unit 103 described later. Identify the structures contained in. Then, the conversion unit 102 specifies the three-dimensional coordinates (world coordinates) of the structure included in the imaging range of the imaging device 17 from the imaging device 17 from the BIM information. As an example, the conversion unit 102 acquires the world coordinates of any one of the buildings represented by the BIM information from an external device or the like, and includes the building represented by the BIM information with the one point as a reference. The three-dimensional coordinates in the BIM information of each point are converted into world coordinates. The method of obtaining the world coordinates of each point included in the building from the BIM information is not limited to this. The conversion unit 102 sends the specified three-dimensional coordinates (world coordinates) to the bundle adjustment unit 104 as the initial value of the three-dimensional coordinates (world coordinates) of the points in the three-dimensional space in the bundle adjustment process described later. The details of the bundle adjustment process will be described later. The three-dimensional coordinates (world coordinates) of the points included in the building 9 specified by the conversion unit 102 from the BIM information are examples of information regarding the positions of surrounding objects in the present embodiment. Hereinafter, unless otherwise specified, the three-dimensional coordinates in the present embodiment are world coordinates.

Note that the conversion unit 102 may specify the range of the initial value without specifying the initial value as a unique value. For example, the conversion unit 102 may provide a range instead of specifying the three-dimensional coordinates of a point in the three-dimensional space as unique coordinates. In this case, a three-dimensional space region that is likely to include a certain point included in the structure included in the imaging range of the imaging device 17 is sent to the bundle adjusting unit 104 as a range of initial values. The initial value or the range of the initial value is an example of the input value generated by the conversion unit 102 in the present embodiment.

In the present embodiment, it is assumed that the three-dimensional coordinate system in the BIM information and the three-dimensional coordinate system (SLAM coordinate system) in the SLAM processing unit 120 are calibrated in advance. In the present embodiment, the calibration means that the correspondence between the position in the BIM information and the position in the SLAM coordinate system is defined. For example, the position of the movement start point of the information processing device 1 in the BIM information in the three-dimensional coordinate system may be stored in the auxiliary storage device 14 as a reference point. Therefore, the conversion unit 102 can specify the position in the building represented by the BIM information corresponding to the position of the image pickup device 17 specified by the tracking unit 103.

The calibration of the 3D coordinate system and the SLAM coordinate system in the BIM information may be executed by an input operation by an administrator or the like, or an index such as an AR (Augmented Reality) marker installed in the building may be used as an index of the SLAM processing unit. It may be executed by recognizing 120 from the captured image.

The SLAM processing unit 120 simultaneously estimates the self-position and generates map information. In the present embodiment, the self-position is the position and orientation of the information processing device 1. Further, in the present embodiment, since the image pickup device 17 is mounted on the information processing device 1, the position and orientation of the image pickup device 17 represent the same as the position and posture of the information processing device 1. When the image pickup apparatus 17 is installed at a position away from the center of the information processing apparatus 1, the SLAM processing unit 120 corrects the displacement between the image pickup apparatus 17 and the center of the information processing apparatus 1 to provide information. The self-position, which is the position of the processing device 1, is estimated.

Further, the map information represents the shape of the surrounding structure along the movement locus of the information processing device 1. More specifically, the map information of the present embodiment represents the internal structure of the building in which the information processing device 1 travels in three dimensions along the movement locus of the information processing device 1. The map information of the present embodiment is, for example, a point cloud map in which the internal structure of the building in which the information processing device 1 travels is represented as a point cloud having three-dimensional coordinates. The type of map information is not limited to this, and the map may be represented by a set of three-dimensional figures instead of a point cloud. The map information is also called an environment map.

The SLAM processing unit 120 is an example of an estimation unit in this embodiment. As a method for estimating the self-position and generating map information, a method other than SLAM may be adopted. Further, the estimation of the self-position and the generation of the map information do not have to be performed at the same time, and the other process may be executed after one process is completed first. In addition, the generation of map information in the present specification includes at least one of newly generating map information, adjusting the generated map information, or updating the generated map information. It shall be.

The SLAM processing unit 120 includes a tracking unit 103 and a bundle adjustment unit 104.

The tracking unit 103 identifies the position and orientation of the image pickup device 17 by tracking a plurality of captured images captured by the image pickup device 17 at different times. The tracking unit 103 is an example of a specific unit in the present embodiment.

For example, in the present embodiment, the image pickup device 17 captures the surroundings while moving as the information processing device 1 moves. The tracking unit 103 calculates changes in the position and posture of the image pickup device 17 by tracking points drawn on a certain captured image on another captured image captured at different times. The tracking unit 103 specifies the current position and orientation of the imaging device 17 by adding changes in the position and orientation specified by the tracking process to the position and orientation of the imaging device 17 at the start of imaging.

FIG. 3 is an image diagram showing an example of tracking processing according to the first embodiment. The reference frame 41 and the target frame 42 are captured images captured at different times by the imaging device 17. The reference frame 41 is an captured image captured before the target frame 42, and the imaging device 17 _{changes from the position Ti at the time when the reference frame 41 is imaged to the position T j} at the time when the target frame 42 is imaged. It is assumed that it has moved. The reference frame 41 is also referred to as a key frame, and the target frame 42 is also referred to as a current frame.

In this case, the tracking unit 103, P point which is depicted in the reference frame 41, by calculating the photometry error in the case where it is depicted in the target frame 42, the image pickup device 17 is moved from the position T _i to the position T _j Calculate the relative amount of movement in the case. The point P is, for example, a feature point on the reference frame 41. The movement of the image pickup apparatus 17 includes both a change in the position of the image pickup apparatus 17 and a change in the posture (orientation).

The position T _i at the time of the reference frame 41 and the image pickup is assumed to be already error-corrected. The point 50a shown in FIG. 3 represents the position where the point P drawn on the reference frame 41 is back-projected on the three-dimensional space.

As an example, the tracking unit 103 calculates _{the photometric error Epj} between the reference frame 41 and the target frame 42 by using the following equation (1).

I _i represents the reference frame 41 and I _j represents the target frame 42. Further, N _p is a neighborhood pattern of pixels including the point P on the reference frame 41. Also, _{t i} is the exposure time of the reference frame 41, _{t j} represents the exposure time of the target frame 42. Further, p'is due to the inverse depth _{d p,} a projected point P in the target frame 42. Further, as shown in the equation (1), the tracking unit 103 calculates the _{photometric error Epj using the Huber norm.} Further, the weighting coefficient W _p is calculated in advance based on the brightness gradient of the pixels. For example, with respect to the gradient is larger pixel by reducing the value of the weight factor W _p, it is possible to reduce noise. As a method for calculating the weighting coefficient W _p , a known method can be applied. The luminance conversion hyperparameter _{a i, a j, b i} , b j is a parameter for converting the luminance of the reference frame 41 and the subject frame 42. Luminance conversion hyperparameter _{a i, a j, b i} , b j may be tuned manually for example by the administrator.

Further, the following equation (2) is a constraint condition of the point P'which is the projection point of the point P used in the equation (1). To calculate the point P', a back projection function that back-projects the point P drawn on the reference frame 41 as a point 50a on the three-dimensional space and a projection function that projects the point 50a on the three-dimensional space onto the target frame 42. And are used. The distance from the point P to the point 50a is the depth (d _p ) of the point 50a in the reference frame 41.

Further, the coefficient R included in the equation (2) represents the amount of rotation of the image pickup apparatus 17. The coefficient t represents the translational amount of the imaging device 17. The coefficient R and the coefficient t are defined by the relative position of the image pickup apparatus 17 according to the following constraint condition (3).

The tracking unit 103 takes an image at the time when the target frame I _j is imaged by solving the model of the photometric error _{Epj between the reference frame I i} and the target frame I _j shown in the above equations (1) to (3). The position _{Tj of the} device 17 is specified. _{The position Ti} and the position T _j shown in the equation (3) and FIG. 3 include the position and orientation of the image pickup apparatus 17. In this way, the tracking unit 103 tracks changes in the position and posture of the imaging device 17 by repeatedly executing such tracking processing on a plurality of captured images captured in time series by the imaging device 17. do.

The tracking method is not limited to the above example. For example, the tracking method is an indirect method (indirect method) in which the position and orientation of the image pickup device 17 at the time of imaging of each frame are acquired by acquiring the feature points on the captured image and then solving the matching problem of the feature points. ) And the Direct method (direct method) in which the position and orientation of the image pickup apparatus 17 at the time of imaging of each frame are estimated by directly estimating the conversion between the captured images without the feature point extraction process. In the above example, the movement of the position and the posture of the image pickup apparatus 17 is calculated by projecting the feature points, but the tracking unit 103 may execute the tracking by the direct method. Further, the tracking unit 103 may specify the position and orientation of the image pickup device 17 in consideration of not only the captured image but also the detection result of the IMU sensor 18.

The tracking unit 103 sends the current position and orientation of the specified imaging device 17 to the bundle adjustment unit 104 and the conversion unit 102.

Returning to FIG. 2, the bundle adjustment unit 104 corrects the position and orientation of the image pickup device 17 specified by the tracking unit 103 and the position information of surrounding objects by the bundle adjustment process. The bundle adjustment unit 104 outputs the self-position of the information processing device 1 and the map information as the processing result.

More specifically, the bundle adjustment unit 104 sends the captured image captured by the image pickup device 17 to the bundle adjustment unit 104 that minimizes the reprojection error for each frame.

More specifically, the bundle adjustment unit 104 optimizes the world coordinate points (three-dimensional position coordinates) of each point in the surrounding environment, the position and orientation of the image pickup device 17, and the internal parameters of the image pickup device 17, respectively. Minimize frame reprojection error.

Note that the internal parameters of the imaging device 17 do not have to be updated by the bundle adjustment unit 104 if the camera has been calibrated in advance. The internal parameters of the image pickup device 17 are, for example, the focal length and the principal point. In bundle adjustment, the position and orientation of the imaging device 17 are also referred to as external parameters.

The bundle adjustment unit 104 of the present embodiment uses the three-dimensional coordinates indicating the positions of the surrounding structures converted from the BIM information by the conversion unit 102 as the initial values of the world coordinate points of each point in the surrounding environment as described above. adopt.

Further, the bundle adjustment unit 104 adjusts the error of the position and orientation of the image pickup device 17 specified by the tracking unit 103 by this bundle adjustment. The bundle adjustment unit 104 obtains the world coordinate points of each point at which the reprojection error is minimized by the bundle adjustment, the position and orientation of the image pickup device 17, and the internal parameters of the image pickup device 17, so that the error is reduced as a result. The position and orientation of the image pickup device 17 are calculated. In addition, the set of world coordinate points after bundle adjustment becomes map information.

FIG. 4 is an image diagram showing an example of the positional relationship between the information processing device 1 and surrounding objects according to the first embodiment. In the example shown in FIG. 4, the information processing device 1 is assumed to move in the building 9 in which the pillars 90a to 90c are installed. Pillars 90a to 90c are examples of objects. The distance d in FIG. 4 is the distance from the image pickup device 17 to the point 52 on the plane 901 facing the information processing device 1 of the pillar 90c.

For example, it is assumed that the conversion unit 102 specifies the initial value of the three-dimensional coordinates of the point 52 on the plane 901. In this case, the bundle adjustment unit 104 starts the adjustment process from the initial value, and based on the position and orientation of the image pickup device 17 specified by the tracking unit 103 and the captured image, the self-position and the position of the point 52 Adjust the error. For example, the conversion unit 102 corrects the three-dimensional coordinates of the point 52 by adjusting the error between the self-position and the position of the point 52, and obtains the three-dimensional coordinates with higher accuracy. By the adjustment process, the bundle adjustment unit 104 estimates the self-position and the three-dimensional coordinates of the point 52.

In addition, there may be other objects such as fixtures between the pillar 90c and the image pickup device 17. Although the information of such an object is not included in the BIM information, it is drawn in the captured image. Therefore, the bundle adjustment unit 104 changes the initial value based on the BIM information by the bundle adjustment, thereby causing the BIM. The position of an object that is not included in the information can also be estimated.

FIG. 5 is an image diagram showing an example of bundle adjustment according to the first embodiment. For example, the bundle adjustment unit 104 captures the projection points 401a and 401b on which the points 52 in the three-dimensional space are projected on the two captured images 43 and 44 shown in FIG. 5 by the following equation (4). The position of the image pickup apparatus 17 and the three-dimensional coordinates of the point 52 are estimated so as to minimize the error from the feature points 402a and 402b corresponding to the points 52 drawn on the images 43 and 44. Hereinafter, when the captured image 43 and the captured image 44 are distinguished, the captured image 43 is referred to as a first image and the captured image 44 is referred to as a second image for convenience. Further, in the present embodiment, the internal parameters of the image pickup apparatus 17 are assumed to have been calibrated in advance, and are not included in the parameters to be optimized in the equation (4).

The initial value generated by the conversion unit 102 described above, as an initial value in the world coordinate point (X _i point in the three-dimensional _space) shown as point 52, it is used in equation (4). Further, in FIG. 5, the lines connecting the reference points 170a and 170b representing the position of the image pickup apparatus 17 and the points 52 are referred to as ray bundles (Bundle) 6a and 6b. Also, when the range of the initial value is set by the converter 102, the equation (4), the point X _i on the three-dimensional space, calculation is started world coordinate included in the range is set .. In the case where the error in the calculation of the range of the initial values is not minimized, beyond the scope of the initial value, the value of the point X _i on the optimum three-dimensional space may be determined.

Further, the bundle adjusting unit 104 estimates the position of the plane or the curved surface of the surrounding object based on the BIM information, and based on the constraint condition that a plurality of points existing in the surroundings are located on the plane or the curved surface. The distance from the image pickup device 17 to the surrounding objects is calculated. For example, the points 50b to 50d shown in FIG. 4 all exist on the plane 901. In such a case, the bundle adjustment unit 104 imposes a constraint condition by the equation of a plane when executing the bundle adjustment process based on the captured image with the plane 901 as the imaging range. Hereinafter, when the individual points 50a to 50d in the three-dimensional space are not particularly limited, they are simply referred to as points 50.

For example, as shown in Eqs. (5) and (6), the bundle adjustment unit 104 solves the optimization problem by the nonlinear least squares method by the nonlinear functions f (x) and g (x), thereby presenting a plane. The position of the point 50 and the position of the imaging device 17 are estimated with the above as a constraint condition. The function f (x) corresponds to the above equation (4). As the solution of equations (5) and (6), the penalty method or the extended Lagrange method can be applied, but other solutions may be adopted.

Note that the structure around the information processing device 1 may have a curved surface as well as a flat surface. For example, the outer surface of the pillar 90b is a curved surface. In such a case, the bundle adjustment unit 104 may impose a constraint condition by a curved surface equation so that a point on a three-dimensional space is on a curved surface based on BIM information.

The bundle adjustment unit 104 generates a point cloud having the three-dimensional coordinates as map information based on the three-dimensional coordinates of the plurality of points 50 after the bundle adjustment. Further, the bundle adjustment unit 104 updates the map information by adding or deleting a new point 50 to the map information. In addition, the bundle adjustment unit 104 may adjust the self-position estimation result and the map information in consideration of the detection result of the IMU sensor 18.

In the bundle adjustment process, the bundle adjustment unit 104 calculates the positions of surrounding objects as the spatial coordinates of the plurality of points 50 in the three-dimensional space, and outputs the calculated spatial coordinates of the plurality of points 50 as map information. .. In the present embodiment, the term "output" includes storage in the auxiliary storage device 14 or transmission to the external device 2.

For example, the bundle adjustment unit 104 stores the estimated self-position and the generated map information in the auxiliary storage device 14. Further, the bundle adjustment unit 104 may transmit the estimated self-position and the generated map information to the external device 2.

The movement control unit 105 moves the information processing device 1 by controlling the movement device 16. For example, the movement control unit 105 searches for a movable route based on the map information stored in the auxiliary storage device 14 and the current self-position. The movement control unit 105 controls the movement device 16 based on the search result.

When the information processing device 1 is provided with a distance measuring sensor such as an ultrasonic sensor or a laser scanner, the movement control unit 105 moves to avoid obstacles based on the detection results of obstacles or the like by these sensors. A route may be generated. The movement control method of the information processing device 1 is not limited to these, and various autonomous movement methods can be applied.

Next, the flow of self-position estimation and map information generation processing executed by the information processing device 1 of the present embodiment configured as described above will be described.

FIG. 6 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the first embodiment.

First, the acquisition unit 101 acquires BIM information from the external device 2 (S1). The acquisition unit 101 stores the acquired BIM information in the auxiliary storage device 14.

Then, the movement control unit 105 starts the movement of the information processing device 1 by controlling the movement device 16 (S2).

Next, the acquisition unit 101 acquires an captured image from the imaging device 17. In addition, the acquisition unit 101 acquires sensing results such as angular velocity and acceleration from the IMU sensor 18 (S3).

Next, the tracking unit 103 identifies the current position and orientation of the image pickup device 17 based on the captured image (S4).

Then, the conversion unit 102 generates an initial value of the three-dimensional coordinates of the point in the structure from the BIM information based on the current position and orientation of the image pickup device 17 specified by the tracking unit 103 (S5).

Then, the bundle adjustment unit 104 executes the bundle adjustment process (S6). Specifically, the initial values of the three-dimensional coordinates of the points in the structure around the image pickup device 17 generated from the BIM information, the position and orientation of the image pickup device 17 specified by the tracking unit 103, and the captured image. Based on the above, the distance from the image pickup device 17 to the surrounding object is calculated, and the position and orientation of the image pickup device 17 and the three-dimensional coordinates of the surrounding object are estimated. In addition, the bundle adjustment unit 104 generates map information based on the estimated three-dimensional coordinates of surrounding objects.

The bundle adjustment unit 104 stores the estimated self-position and the generated map information in, for example, the auxiliary storage device 14.

Further, the movement control unit 105 searches for a movement route based on the map information stored in the auxiliary storage device 14 and the current self-position, and controls the movement device 16 based on the search result to obtain an information processing device. Move 1

Then, the movement control unit 105 determines whether or not to end the movement of the information processing device 1 (S7). The movement control unit 105 determines, for example, that when the information processing device 1 arrives at a predetermined end point, the movement control unit 105 ends the movement of the information processing device 1. The conditions for determining the end of movement are not particularly limited. For example, when the movement control unit 105 receives an instruction to end movement from the outside via the communication network 3, the information processing device 1 It may be determined that the movement of is completed.

If the movement control unit 105 does not determine that the movement is completed (S7 "No"), the process returns to S3 and the processes S3 to S7 are repeated. Further, when the movement control unit 105 determines that the movement is completed (S7 “Yes”), the processing of this flowchart ends.

As described above, the information processing device 1 of the present embodiment executes self-position estimation and map information generation based on the BIM information and the captured image captured around the information processing device 1. Therefore, according to the information processing apparatus 1 of the present embodiment, by using the BIM information for the processing of self-position estimation and map information generation, it is possible to improve the self-position estimation and the accuracy of the map information. ..

For example, the information processing apparatus 1 of the present embodiment converts BIM information into at least one input value of self-position estimation processing by SLAM processing unit 120 or map information generation processing, and based on the input value, By executing self-position estimation and map information generation, self-position estimation and map information accuracy are higher than self-position estimation and map information generation based only on peripheral detection results such as captured images. Can be improved.

Further, the information processing device 1 of the present embodiment identifies the position and orientation of the image pickup device 17 by tracking a plurality of captured images captured at different times, and from the image pickup device 17 based on the BIM information to a surrounding object. By changing the initial value or the range of the initial value of the three-dimensional coordinates of the point based on the position and orientation of the image pickup device 17 identified by tracking the captured image and the captured image, from the image pickup device 17 to the surrounding object. Calculate the distance, self-position, and position of surrounding objects. More specifically, the information processing device 1 of the present embodiment uses the distance from the image pickup device 17 based on the BIM information to the surrounding object as the initial value or the range of the initial value in the bundle adjustment process. Therefore, according to the information processing device 1 of the present embodiment of the present embodiment, the distance from the image pickup device 17 based on the BIM information to the surrounding object is the distance from the image pickup device 17 to the surrounding object in three-dimensional coordinates. By using it as the initial value or the range of the initial value of, the amount of calculation in the bundle adjustment process can be reduced.

For example, as a comparative example, in a general bundle adjustment process, the initial value of the three-dimensional coordinates of a point in the three-dimensional space may be assumed to be infinite. In such a case, for example, the bundle adjustment process starts without specifying whether the distance between the point in the three-dimensional space and the image pickup device is 1 m or 1000 m, so that the amount of calculation until the calculation result converges. May increase. On the other hand, since the information processing apparatus 1 of the present embodiment uses the initial value based on the BIM information, the processing result can be converged with a small amount of calculation.

Further, the information processing apparatus 1 of the present embodiment estimates the position of the plane or the curved surface of the surrounding object based on the BIM information, and the constraint condition that a plurality of points existing in the surroundings are located on the plane or the curved surface. The distance from the image pickup apparatus 17 to the surrounding objects is calculated based on the above. Therefore, according to the information processing apparatus 1 of the present embodiment, the amount of calculation can be reduced as compared with the case where the positions of a plurality of points existing on the same plane or curved surface are separately obtained.

Further, the information processing device 1 of the present embodiment calculates the positions of surrounding objects as the spatial coordinates of the plurality of points 50 in the three-dimensional space, and outputs the calculated spatial coordinates of the plurality of points 50 as map information. do. According to the information processing device 1 of the present embodiment, more accurate map information can be provided by outputting the positions of surrounding objects calculated by the bundle adjustment process using BIM information as map information.

The information processing device 1 may be a robot or the like having functions such as monitoring, security, cleaning, and delivery of luggage. In this case, the information processing device 1 realizes various functions by moving the building 9 based on the estimated self-position and map information. Further, the map information generated by the information processing device 1 may be used not only for generating the movement route of the information processing device 1 itself, but also for monitoring or managing the building 9 from a remote location. Further, the map information generated by the information processing device 1 may be used to generate a movement route of a robot or drone other than the information processing device 1.

The image pickup device 17 is not limited to the stereo camera. For example, the image pickup device 17 may be an RGB-D camera having an RGB (Red Green Blue) camera and a three-dimensional measurement camera (Dept camera), a monocular camera, or the like.

Further, the sensor included in the information processing device 1 is not limited to the IMU sensor 18, and a gyro sensor, an acceleration sensor, a magnetic sensor, or the like may be individually provided.

Further, in the present embodiment, the SLAM processing unit 120 executes the image SLAM (Visual SLAM) using the captured image, but the SLAM that does not use the captured image may be adopted. For example, the information processing device 1 may detect surrounding structures by Lidar (Light Detection and Ringing or Laser Imaging Detection and Ringing) or the like instead of the image pickup device 17. In this case, the SLAM processing unit 120 may specify the position and orientation of the information processing device 1 based on the distance measurement result by Lidar.

Further, in the present embodiment, the SLAM processing unit 120 is supposed to generate three-dimensional map information, but it may be possible to generate two-dimensional map information.

Further, the equations (1) to (6) illustrated in the present embodiment are examples, and the mathematical expressions used in the tracking process or the bundle adjustment process are not limited to these. Further, the bundle adjustment unit 104 may perform bundle adjustment according to the equation (4) without imposing the constraint conditions according to the equations (5) and (6). Further, in the tracking process, without using the proximity pattern N _p, it may perform the tracking process.

Further, the SLAM processing unit 120 may estimate its own position and generate map information by a method other than tracking processing or bundle adjustment processing. For example, in the present embodiment, the tracking unit 103 is used as an example of the specific unit, but a method of specifying a change in the position and posture of the information processing device 1 by a method other than tracking may be adopted.

Further, in addition to the processes exemplified in this embodiment, various processes for improving the accuracy of self-position estimation or map information may be added to the SLAM process. For example, the SLAM processing unit 120 may further execute a loop closing process or the like.

A part or all of the information processing device 1 in the above-described embodiment may be composed of hardware, or may be composed of information processing of software (program) executed by a CPU, GPU, or the like. good. When it is composed of software information processing, software that realizes at least a part of the functions of each device in the above-described embodiment is a flexible disk, a CD-ROM (Compact Disc-Read Only Memory), a USB memory, or the like. The software may process information by storing it in a non-temporary storage medium (non-temporary computer-readable medium) and loading it into a computer. In addition, the software may be downloaded via a communication network. Further, information processing may be executed by hardware by implementing the software in a circuit such as an ASIC or FPGA.

The type of storage medium that stores the software is not limited. The storage medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk or a memory. Further, the storage medium may be provided inside the computer or may be provided outside the computer.

Further, in FIG. 1, the information processing device 1 includes one component, but may include a plurality of the same components. Further, although one information processing device 1 is shown in FIG. 1, software is installed on a plurality of computers, and each of the plurality of computers executes the same or different part of the processing of the software. You may. In this case, it may be a form of distributed computing in which each computer communicates via a network interface 13 or the like to execute processing. That is, the information processing device 1 in the above-described embodiment may be configured as a system that realizes a function by executing instructions stored in one or a plurality of storage devices by one or a plurality of computers. Further, the information transmitted from the terminal may be processed by one or a plurality of computers provided on the cloud, and the processing result may be transmitted to the terminal.

Various operations of the information processing device 1 in the above-described embodiment may be executed in parallel processing by using one or a plurality of processors or by using a plurality of computers via a network. Further, various operations may be distributed to a plurality of arithmetic cores in the processor and executed in parallel processing. In addition, some or all of the processes, means, etc. of the present disclosure may be executed by at least one of a processor and a storage device provided on the cloud capable of communicating with the information processing device 1 via the network. As described above, each device in the above-described embodiment may be in the form of parallel computing by one or a plurality of computers.

The information processing device 1 in the above-described embodiment may be realized by one or a plurality of processors 11. Here, the processor 11 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. You may point. When a plurality of electronic circuits are used, each electronic circuit may communicate by wire or wirelessly.

Multiple processors may be connected (combined) to one storage device (memory), or a single processor may be connected. A plurality of storage devices (memory) may be connected (combined) to one processor. When the information processing device 1 in the above-described embodiment is composed of at least one storage device (memory) and a plurality of processors connected (combined) to the at least one storage device (memory), among the plurality of processors At least one processor may include a configuration in which it is connected (combined) to at least one storage device (memory). Further, this configuration may be realized by a storage device (memory) and a processor included in a plurality of computers. Further, a configuration in which the storage device (memory) is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

Further, the external device 2 is not limited to the server device. Further, the external device 2 may be provided in a cloud environment. Further, the external device 2 may be used as an example of the information processing device in the claims.

Further, as another example, the external device 2 may be an input device. Further, the device interface 15 may be connected not only to the mobile device 16, the image pickup device 17, and the IMU sensor 18, but also to the input device. The input device is, for example, a device such as a camera, a microphone, a motion capture, various sensors, a keyboard, a mouse, or a touch panel, and gives the acquired information to the information processing device 1. Further, it may be a device including an input unit, a memory and a processor such as a personal computer, a tablet terminal, or a smartphone.

Further, as another example, the external device 2 may be an output device. Further, the device interface 15 may be connected to the output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, and outputs audio or the like. It may be a speaker or the like. Further, it may be a device including an output unit such as a personal computer, a tablet terminal, or a smartphone, a memory, and a processor.

Further, as another example, the external device 2 may be a storage device (memory). Further, the device interface 15 may be connected to a storage device (memory). For example, the external device 2 may be a network storage or the like, and a storage such as an HDD may be connected to the device interface 15.

Further, the external device 2 or the external device connected to the device interface 15 may be a device having some functions of the components of the information processing device 1 in the above-described embodiment. That is, the information processing device 1 may transmit or receive a part or all of the processing results of the external device 2 or the external device connected to the device interface 15.

Further, while the self-position estimation process and the map information generation process are being executed, the information processing device 1 may be constantly connected to the external device 2 via the communication network 3, but is limited to this. is not it. For example, the information processing device 1 may take the connection with the external device 2 offline while executing the self-position estimation process and the map information generation process.

In the present embodiment, the world coordinates of the points included in the building 9 specified by the conversion unit 102 from the BIM information are used as an example of the information regarding the positions of the surrounding objects, but the information regarding the positions of the surrounding objects is included in this. It is not limited.

For example, the conversion unit 102 may generate information indicating the distance between the information processing device 1 and a surrounding object based on the BIM information and the position and orientation of the image pickup device 17. As the position and orientation of the image pickup apparatus 17, for example, the position and orientation specified by the tracking unit 103 can be adopted.

In this case, generated by (4) and _{X i} point in the three-dimensional space in the expression, position of the imaging device 17 _{(R j,} t _j), the distance between the _{(R j + 1, t j} + 1) is, the conversion unit 102 It is identified by information that indicates the distance taken. In this case, at the start of the calculation, the bundle adjustment unit 104 determines _{the position (R j} , t _j ) _{of the point X i} imaging device 17 in the three-dimensional space specified by the information indicating the distance generated by the conversion unit 102. Set a value for each parameter so that it matches the distance between (R _{j + 1} , t _{j + 1).} Also in this method, the position of a point X _i and the imaging device 17 in a three-dimensional space obtained result of the adjustment of the error by bundle adjustment may be different from the result obtained from the BIM data.

Further, for example, the conversion unit 102 may specify the dimension of the building 9 from the BIM information, and the dimension may be used as an example of information regarding the position of a surrounding object. In this case, bundle adjustment unit 104 (4) and _{X i} point in the three-dimensional space in the expression, position of the imaging device 17 _{(R j,} t _j), although the _{(R j + 1, t j} + 1), the BIM information Bundle adjustment is performed to be within the dimensions of the identified building 9. For example, if the total length of the building 9 identified from BIM information 50m, a width of 50m, _{X i} point in the three-dimensional space are included in the building structure 9 moves away from the position of the imaging device 17, 50 m or more Therefore, the range of values that can be taken by _{the points X i} in the three-dimensional space and the positions (R _j , t _j ) and (R _{j + 1} , t _{j + 1) of the imaging device 17 is limited.} In this way, the bundle adjustment unit 104 can reduce the amount of calculation by performing bundle adjustment using the information regarding the position of the surrounding object converted from the BIM information by the conversion unit 102.

(Second Embodiment)
In the first embodiment described above, the environmental information is three-dimensional design information such as BIM information or 3D CAD data. In this second embodiment, the environmental information includes at least one of the entry / exit information of the person in the building 9 and the image recognition result of the person in the captured image. The environmental information may include both the entry / exit information of the person in the building 9 and the image recognition result of the person in the captured image, or may include either one.

FIG. 7 is a block diagram showing an example of the functions included in the information processing device 1 according to the second embodiment. As shown in FIG. 7, the information processing apparatus 1 of the present embodiment includes an acquisition unit 1101, a conversion unit 1102, a SLAM processing unit 1120, and a movement control unit 105. Further, the SLAM processing unit 1120 includes a tracking unit 1103 and a bundle adjusting unit 1104. Further, the conversion unit 1102 includes an initial value generation unit 106 and a mask information generation unit 107.

The movement control unit 105 has the same function as that of the first embodiment.

The acquisition unit 1101 of the present embodiment has the same function as that of the first embodiment, and acquires the entry / exit information of the person in the building 9.

The entry / exit information is information indicating the number of people entering / exiting each room or floor of the building 9 and the time of entering / exiting. For example, it is assumed that a sensor for detecting entry / exit is installed at the entrance / exit of a room or floor of the building 9, and the detection result by the sensor is transmitted to the external device 2. In this case, the acquisition unit 1101 acquires the entry / exit information from the external device 2. The method of detecting the entry / exit of a person is not limited to the sensor. For example, the entry / exit information may be a reading record of a security card by a card reader or a detection result of a person from an image captured by a surveillance camera installed in a building 9.

The acquisition unit 1101 stores the acquired entry / exit information in the auxiliary storage device 14.

The conversion unit 1102 of the present embodiment has the same functions as those of the first embodiment, and is excluded from the target of map information generation in the building 9 where the information processing device 1 is located based on the environmental information. Generates mask information that represents.

In the present embodiment, the environmental information includes at least one of the entry / exit information and the image recognition result of the person. The image recognition result of a person is a result of recognizing a person by image processing from the captured image captured by the image pickup device 17.

Specifically, the environmental information of the present embodiment includes both the entry / exit information and the image recognition result, and the same three-dimensional design information as that of the first embodiment.

More specifically, the conversion unit 1102 includes an initial value generation unit 106 and a mask information generation unit 107. The initial value generation unit 106 has the same function as the conversion unit 102 in the first embodiment.

The mask information is information representing an area excluded from the target of map information generation.

The mask information generation unit 107 determines the area where the person is located in the building 9 based on the entry / exit information or the image recognition result of the person, and sets the determined area as an area to be excluded from the target of map information generation. ..

The mask information generation unit 107 recognizes a person by image processing from the captured image captured by the imaging device 17. Further, when it is difficult to determine whether or not the object depicted in the captured image is a person in the image processing, the mask information generation unit 107 may move to the room or floor where the captured image is captured based on the entry / exit information. , It is determined whether or not a person exists at the time when the captured image is captured. When the mask information generation unit 107 determines that a person exists in the room or floor where the captured image is captured at the time when the captured image is captured, the mask information generation unit 107 draws the captured image on the captured image rather than determining that the person does not exist. It is highly probable that the object is a person.

FIG. 8 is an image diagram showing an example of the positional relationship between the information processing device 1 and surrounding objects according to the second embodiment. In the example shown in FIG. 8, in the building 9, the person 70 exists in the room where the information processing device 1 exists. Unlike the pillars 90a to 90c, the person 70 moves, so if the presence of the person 70 is included in the map information, the accuracy of the map information may decrease.

The mask information generation unit 107 generates mask information representing the area 80 in which the person 70 exists. The mask information represents, for example, the area 80 in which the person 70 exists in three-dimensional coordinates.

In the present embodiment, the mask information generation unit 107 generates mask information using both the entry / exit information and the image recognition result of the person, but the mask information is generated based on only one of them. You may.

Further, the mask information generation unit 107 detects a moving body such as a vehicle or an object such as a device temporarily existing in the building 9 such as a dolly from the image captured by the image pickup device 17 by image recognition. You may. In this case, the mask information generation unit 107 sets the area where it is determined that these objects exist as an area to be excluded from the target of map information generation.

The mask information generation unit 107 generates mask information representing an area to be excluded from the target of map information generation, and sends it to the SLAM processing unit 1120.

Returning to FIG. 7, the SLAM processing unit 1120 of the present embodiment has the functions of the first embodiment and does not generate map information for the area corresponding to the mask information.

More specifically, the tracking section 1103 of the SLAM processor 1120 of the present embodiment, when performing the tracking process by the same equation (1) in the first embodiment, near the pattern N _p is represented by the mask information If the area that is the image area to be rendered, with respect to the vicinity pattern N _p, is multiplied by a mask value. The mask value is, for example, “0” or “1”, but is not limited thereto. The method of applying the mask is not limited to this, and other methods may be adopted.

Further, the bundle adjustment unit 1104 of the SLAM processing unit 1120 of the present embodiment has the functions of the first embodiment, and the area corresponding to the mask information is excluded from the bundle adjustment.

Next, the flow of self-position estimation and map information generation processing executed by the information processing device 1 of the present embodiment configured as described above will be described.

FIG. 9 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the second embodiment.

The process of acquiring the BIM information in S1 is the same as that of the first embodiment described with reference to FIG. Next, the acquisition unit 1101 acquires the entry / exit information acquisition (S21). The acquisition unit 1101 stores the acquired entry / exit information in the auxiliary storage device 14.

The process of acquiring the captured image of S3 and the sensing results such as angular velocity and acceleration from the process of starting the movement of the information processing device 1 of S2 is the same as that of the first embodiment.

Next, the mask information generation unit 107 of the conversion unit 1102 of the present embodiment generates mask information representing an area to be excluded from the target of map information generation based on the entry / exit information or the image recognition result of the person 70 ( S22).

Then, the tracking unit 1103 of the SLAM processing unit 1120 of the present embodiment identifies the current position and orientation of the image pickup device 17 based on the captured image (S4). At this time, the tracking unit 1103 excludes the area corresponding to the mask information from the tracking process.

Then, the initial value generation unit 106 of the conversion unit 1102 of the present embodiment is based on the current position and orientation of the image pickup device 17 specified by the tracking unit 103, and is based on the BIM information in the structure around the image pickup device 17. The initial value of the three-dimensional coordinates of the point is generated (S5). Since the area corresponding to the mask information is not subject to the generation of map information, the initial value generation unit 106 does not generate the initial value of the three-dimensional coordinates of the points in the structure in the area corresponding to the mask information. ..

Next, the bundle adjustment unit 1104 executes the bundle adjustment process (S6). The bundle adjustment unit 1104 excludes the area corresponding to the mask information from the bundle adjustment.

The process of determining whether or not to end the movement of the information processing device 1 in S7 is the same as that of the first embodiment. In the present embodiment, when the movement control unit 105 does not determine that the movement is completed (S7 “No”), the acquisition unit 1101 acquires the latest entry / exit information acquisition again (S23), and returns to the process of S3.

Further, when the movement control unit 105 determines that the movement is completed (S7 “Yes”), the processing of this flowchart ends.

As described above, the information processing device 1 of the present embodiment generates mask information representing the area excluded from the target of map information generation in the building 9 based on the environmental information, and the area corresponding to the mask information is the area corresponding to the mask information. Does not generate map information. Therefore, according to the information processing device 1 of the present embodiment, it is possible to exclude things such as a person 70 and an object that temporarily exist, which may reduce the accuracy of the map information, so that the accuracy of the map information can be reduced. Can be improved.

For example, in the present embodiment, the environmental information includes the entry / exit information of the person 70 in the building 9, or the image recognition result of the person 70 in the image captured by the image pickup device 17 mounted on the information processing device 1. The information processing device 1 of the present embodiment determines an area where the person 70 is located in the building 9 based on the entry / exit information or the image recognition result of the person 70, and determines the determined area from the target of generating map information. The area to be excluded.

For example, when the building 9 is a work site or the like, a person 70 such as a worker may exist in the building 9. In such a case, the information processing device 1 can improve the accuracy of the map information by not reflecting the worker in the map information. Further, with such a configuration, the information processing apparatus 1 of the present embodiment can robustly execute processing even when the surrounding environment changes depending on a person or the like.

In the present embodiment, the bundle adjustment process is performed based on the environmental information as in the first embodiment, but the information processing device 1 of the second embodiment is the same as that of the first embodiment. It does not have to have all the functions. For example, the information processing device 1 may use the environmental information only for generating the mask information and may not use it for the bundle adjustment process. When the configuration is adopted, the environmental information does not have to include the three-dimensional design information.

Further, in the present embodiment, an example in which the mask information is used when the information processing device 1 is moving to generate map information in real time has been described, but the timing of using the mask information is limited to this. is not it. For example, when updating the map information generated by the information processing device 1 later, mask information based on entry / exit information at a past time may be used.

(Third Embodiment)
In the first and second embodiments described above, it is assumed that the correspondence relationship between the three-dimensional coordinate system and the SLAM coordinate system in the three-dimensional design information such as BIM information is defined in advance. In this third embodiment, the correspondence between the three-dimensional coordinate system and the SLAM coordinate system in the three-dimensional design information is adjusted while the information processing device 1 is moving.

FIG. 10 is a block diagram showing an example of the functions included in the information processing device 1 according to the third embodiment. As shown in FIG. 10, the information processing apparatus 1 of the present embodiment includes an acquisition unit 1101, a marker detection unit 108, a calibration unit 109, a conversion unit 2102, a SLAM processing unit 2120, and a movement control unit 105. To be equipped. Further, the SLAM processing unit 2120 includes a tracking unit 2103 and a bundle adjustment unit 1104. Further, the conversion unit 2102 includes an initial value generation unit 1106 and a mask information generation unit 1107.

The movement control unit 105 has the same functions as those of the first and second embodiments. The acquisition unit 1101 has the same function as that of the second embodiment.

The marker detection unit 108 detects the AR marker from the captured image. The AR marker has, for example, information on three-dimensional coordinates representing the position where the AR marker is described. The three-dimensional coordinates are consistent with the coordinate system in the BIM information. For example, the AR marker is a coordinate system in BIM information and represents a position where the AR marker is installed.

The AR marker is an example of index information in this embodiment. It is assumed that the AR marker is installed on a wall, a pillar, or the like along the passage of the building 9. Specifically, the AR marker is, for example, a QR code (registered trademark) or the like, but is not limited thereto. The number of AR markers is not particularly limited, but it is assumed that a plurality of AR markers are installed per building 9. Further, the marker detection unit 108 is an example of the index detection unit in the present embodiment. The marker detection unit 108 sends the detection result of the AR marker to the calibration unit 109.

Based on the detection result of the AR marker, the calibration unit 109 adjusts the coordinate system representing the self-position held internally so as to match the coordinate system of the BIM information. The calibration unit 109 is an example of the coordinate adjustment unit in this embodiment.

For example, the locus of change in self-position estimated by the SLAM processing unit 2120 is stored in the auxiliary storage device 14, but an error in self-position may be accumulated as the information processing device 1 moves. In such a case, there is a difference in the correspondence between the estimated self-position and the position of the building 9 on the three-dimensional model in the BIM information. The calibration unit 109 eliminates the accumulation of such errors by adjusting the current position of the information processing device 1 based on the three-dimensional coordinates represented by the AR marker detected by the marker detection unit 108.

The calibration unit 109 sends the calibration result to the conversion unit 2102. For example, the calibration unit 109 sends a conversion matrix for correcting the self-position to the conversion unit 2102.

Further, the conversion unit 2102 of the present embodiment has the same functions as those of the first and second embodiments, and also provides environmental information to the SLAM processing unit 2120 based on the self-position adjusted by the calibration unit 109. Converts to the input value of the self-position estimation process or the map information generation process.

More specifically, the initial value generation unit 1106 has the same function as that of the second embodiment, and is an imaging device specified by the tracking unit 2103 based on the self-position adjusted by the calibration unit 109. An input that aligns the position of 17 and the position of the image pickup device 17 in the BIM information, and represents the initial value or the range of the initial value of the bundle adjustment process from the BIM information based on the position and orientation of the image pickup device after the alignment. Generate a value.

For example, the initial value generation unit 1106 specified the three-dimensional coordinates indicating the position of the information processing device 1 in the building 9 on the three-dimensional model of the building 9 in the BIM information by the transformation matrix generated by the calibration unit 109. Above, the initial value of the bundle adjustment process or the input value representing the range of the initial value is generated.

Further, the mask information generation unit 1107 has the same function as that of the second embodiment, and generates mask information based on the self-position adjusted by the calibration unit 109.

For example, the mask information generation unit 1107 identifies the three-dimensional coordinates indicating the position of the information processing device 1 in the building 9 on the three-dimensional model of the building 9 in the BIM information by the conversion matrix generated by the calibration unit 109. Above, generate mask information.

The bundle adjustment unit 1104 of the SLAM processing unit 2120 has the same functions as those of the first and second embodiments, and the bundle adjustment unit 2102 generates the bundle adjustment based on the self-position adjusted by the calibration unit 109. Use the initial value of processing or the range of initial values for bundle adjustment.

Next, the flow of self-position estimation and map information generation processing executed by the information processing device 1 of the present embodiment configured as described above will be described.

FIG. 11 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the third embodiment.

The process from the process of acquiring the BIM information of S1 to the process of acquiring the captured image and the sensing result of S3 is the same as that of the second embodiment.

The marker detection unit 108 detects the AR marker from the captured image (S31). The marker detection unit 108 sends the detection result of the AR marker to the calibration unit 109.

The calibration unit 109 executes the calibration process based on the detection result of the AR marker (S32). For example, the calibration unit 109 generates a transformation matrix for adjusting the self-position of the BIM information in the coordinate system. The calibration unit 109 sends the generated transformation matrix to the conversion unit 2102.

The mask information generation unit 1107 identifies the three-dimensional coordinates indicating the position of the information processing device 1 in the building 9 on the three-dimensional model of the building 9 in the BIM information by the conversion matrix generated by the calibration unit 109. , Generates mask information (S22).

The tracking process of S4 is the same as that of the first and second embodiments, but the calibration result by the calibration unit 109 may also be used in the process.

For example, in the present embodiment, the calibration unit 109 sends the calibration result to the conversion unit 2102, but may further send the calibration result to the SLAM processing unit 2120. When adopting this configuration, the tracking unit 2103 of the SLAM processing unit 2120 executes the tracking process using the three-dimensional coordinates based on the calibration result.

Then, the initial value generation unit 1106 specifies the three-dimensional coordinates indicating the position of the information processing device 1 in the building 9 on the three-dimensional model of the building 9 in the BIM information by the transformation matrix generated by the calibration unit 109. Above, the initial value of the bundle adjustment process or the input value representing the range of the initial value is generated (S5).

Also in the bundle adjustment process of S6, the bundle adjustment unit 1104 may execute the tracking process and the bundle adjustment process using the three-dimensional coordinates based on the calibration result.

The process of determining whether or not to end the movement of the information processing device 1 in S7 and the process of acquiring the entry / exit information in S23 are the same as those in the second embodiment.

As described above, the information processing apparatus 1 of the present embodiment detects the index information whose position is represented by the coordinate system in the BIM information from the detection result of the captured image or the like, and is based on the coordinate system adjusted by the index information. Then, the environmental information is converted into an input value of the self-position estimation process by the SLAM processing unit 2120 or the map information generation process. Therefore, according to the information processing device 1 of the present embodiment, the error between the BIM information and the internal SLAM coordinate system of the information processing device 1 is reduced, and self-position estimation and map information generation can be performed with higher accuracy. can do.

In the present embodiment, the AR marker is illustrated as the index information, but the index information is not limited to this. For example, the index information may be a sign or the like that can be captured by Lidar or various sensors, or may be a beacon or the like.

Further, in the present embodiment, the information processing device 1 is described as having the functions of both the first embodiment and the second embodiment, but the information processing device 1 of the present embodiment is described as having the functions of both the first embodiment and the second embodiment. It is not necessary to have all the functions of the second embodiment. For example, the information processing apparatus 1 may use the environmental information only for either bundle adjustment or generation of mask information. Further, the environmental information may include any of three-dimensional design information, entry / exit information, and image recognition result of a person.

(Fourth Embodiment)
In the second embodiment described above, the information processing apparatus 1 uses the captured image for recognizing a person, but the use of the captured image is not limited to this. In this fourth embodiment, the information processing apparatus 1 segmentes the captured image based on the recognition result of the object drawn on the captured image, and performs SLAM processing based on the segmentation result.

The information processing device 1 of the present embodiment includes an acquisition unit 101, a conversion unit 102, a SLAM processing unit 120, and a movement control unit 105.

The acquisition unit 101 has the same function as that of the first embodiment. Specifically, the acquisition unit 101 acquires an captured image from the imaging device 17 via the device interface 15.

The conversion unit 102 has the same function as that of the first embodiment, and then segments the captured image acquired by the acquisition unit 101 based on the recognition result of the object drawn on the captured image.

FIG. 12 is a diagram showing an example of segmentation of the captured image 60 according to the fourth embodiment. As shown on the left side of FIG. 12, the captured image 60 depicts the environment around the information processing device 1. The conversion unit 102 recognizes the image area in which the object is drawn and the type of each object from the captured image 60. In the present embodiment, the recognition result of the object is information in which the two-dimensional coordinates of the image area in which the object is drawn and the type of each object are associated with each other.

In the present embodiment, the environmental information includes at least the captured image 60. Alternatively, the segmentation result of the captured image 60 may be used as an example of the environmental information instead of the captured image 60 itself.

The conversion unit 102 recognizes the object depicted in the captured image 60. As in the first embodiment, in the present embodiment as well, the term "object" includes structures such as walls and pillars, furniture, furniture, moving objects, temporary objects, people, and the like.

The conversion unit 102 recognizes the individual objects drawn on the captured image 60 by inputting the captured image 60 into the trained model configured by, for example, a neural network or the like. In the example shown in FIG. 12, the captured image 60 depicts a person 70, boxes 75a and 75b, a pillar 90, a wall 91, and a floor 92. The conversion unit 102 recognizes these objects. "People," "boxes," "pillars," "walls," and "floors" are examples of object types. The recognition of the person 70 and the recognition of other objects may be executed separately.

The conversion unit 102 segmentes the captured image 60 based on the recognition result of the object. On the right side of FIG. 12, the segmentation result 61 of the captured image 60 is shown. In the example shown in FIG. 12, the conversion unit 102 sets the image area where the pillar 90 and the wall 91 are drawn as the area A1, the image area where the floor 92 is drawn, the area A2, and the image area where the boxes 75a and 75b are drawn. The image region in which the area A3 and the person 70 are drawn is set as the area A4, and the captured image 60 is segmented into four. The division unit is not limited to the example shown in FIG. Hereinafter, when the regions A1 to A4 are not particularly distinguished, they are simply referred to as regions A.

Further, the conversion unit 102 classifies the recognized objects, that is, the person 70, the boxes 75a and 75b, the pillar 90, the wall 91, and the floor 92 according to whether or not they are permanent objects. For example, the pillar 90, the wall 91, and the floor 92 are permanent objects because they are part of the building 9. The person 70 and the boxes 75a and 75b are non-permanent objects. Whether or not each object is permanently installed is determined by, for example, a trained model.

A permanently installed object is an object that does not move from the installation position once it is installed. For example, an object that is a part of the building 9, such as the pillar 90, the wall 91, and the floor 92, is basically a permanent object because it does not move. An object that is not permanently installed is an object that is likely to move from the installation position. For example, a person 70, a moving body such as a cart or a forklift, temporarily installed fixtures, luggage boxes 75a, 75b, etc. are non-permanent objects.

Further, the conversion unit 102 associates the segmented areas A1 to A4 with whether or not the object drawn in each area is a permanently installed object. In the present embodiment, the information in which the segmented areas A1 to A4 are associated with whether or not the object drawn in each area is a permanently installed object is referred to as a segmentation result. The conversion unit 102 sends the segmentation result to the SLAM processing unit 120.

In the above, the object recognition and the segmentation process based on the result of the object recognition have been described separately, but these processes may be integrated. For example, a trained model that outputs the segmentation result of the captured image 60 when the captured image 60 is input may be adopted. In this case, the conversion unit 102 inputs the captured image 60 into the trained model and obtains the segmentation result output from the trained model.

Further, the method of object recognition from the captured image 60 and the segmentation of the captured image 60 is not limited to the above example. For example, the conversion unit 102 may apply a machine learning or deep learning technique other than the neural network to perform object recognition from the captured image 60 and segmentation of the captured image 60.

Further, the SLAM processing unit 120 of the present embodiment has the functions of the first embodiment, and then estimates the self-position and generates map information based on the segmentation result of the captured image 60.

For example, the SLAM processing unit 120 identifies a three-dimensional space corresponding to the areas A1 and A2 in which the permanent object is drawn in the captured image 60, and uses the three-dimensional space for self-position estimation processing and map information. Target of generation processing. Further, the SLAM processing unit 120 identifies a three-dimensional space corresponding to the areas A3 and A4 in which a non-permanent object is drawn in the captured image 60, and estimates the self-position and map information in the three-dimensional space. Exclude from the target of the generation process of. In this case, the information representing the areas A3 and A4 in which the non-permanent object is drawn may be used as the mask information representing the area to be excluded from the target of generating the map information.

Alternatively, the SLAM processing unit 120 uses the regions A1 and A2 in which the permanent object is drawn in the captured image 60 for SLAM processing, and the region A3 in which the non-permanent object is drawn in the captured image 60. , A4 may not be used for SLAM processing.

Further, the weighting at the time of SLAM processing may be changed depending on whether or not the objects drawn in each of the areas A1 to A4 are permanent objects. For example, in the SLAM processing unit 120, the weighting coefficient of the regions A1 and A2 in which the permanent object is drawn is larger than the weighting coefficient of the regions A3 and A4 in which the non-permanent object is drawn in the captured image 60. The weighting coefficients are set in each of the regions A1 to A4 so as to be. As a result, in the estimation of the self-position and the generation of the map information, the influence of the areas A1 and A2 in which the permanent object is drawn is affected by the areas A3 and A4 in which the non-permanent object is drawn. Will also grow.

Further, the SLAM processing unit 120 may change the weighting coefficient for each type of object in the area A in which the non-permanent object is drawn. For example, there are an object that is relatively likely to be installed at the same position for a long period of time and an object that is relatively unlikely to be installed even if the object is not permanently installed. The SLAM processing unit 120 is classified as a non-permanent object because, for example, large furniture and furniture may move, but it stays in the same position for a long period of time as compared with the person 70 and the like. It is likely to be installed. Therefore, the SLAM processing unit 120 sets the weighting coefficient so that the weighting coefficient becomes larger as the area A in which the object having a low possibility of movement is drawn out of the area A in which the non-permanent object is drawn. May be.

The weighting coefficient for each area A1 to A4 may be set by the conversion unit 102 instead of the SLAM processing unit 120.

As described above, the information processing device 1 of the present embodiment segments the captured image 60 captured by the imaging device 17 based on the recognition result of the object drawn on the captured image 60, and based on the result of the segmentation. , Performs self-position estimation and map information generation. Therefore, according to the information processing apparatus 1 of the present embodiment, in addition to the effect of the first embodiment, whether it is used for estimating the self-position and generating map information according to the object drawn on the captured image 60. Whether or not, or the strength of the influence on the estimation of the self-position and the generation of the map information can be adjusted, so that the accuracy of the estimation of the self-position and the accuracy of the map information can be improved.

Further, the SLAM processing unit 120 described above may execute self-position estimation and map information generation based on the segmentation result of the captured image 60 and the three-dimensional design information such as BIM information.

For example, when the conversion unit 102 determines that the object depicted in the captured image 60 is not a permanent object, the SLAM processing unit 120 refers to the three-dimensional design information, and the object is included in the design of the building 9. Judge whether or not. If the object determined not to be a permanent object based on the captured image 60 is not registered in the three-dimensional design information, the SLAM processing unit 120 adopts the determination result that the object is not a permanent object as it is. Further, when the object determined to be not a permanent object based on the captured image 60 is registered in the three-dimensional design information, the SLAM processing unit 120 determines that the object is not a permanent object. Is changed to the judgment result that is a permanent object.

Further, the conversion unit 102 may determine the high accuracy of determining whether or not the object depicted in the captured image 60 is a permanent object by a percentage or the like. For example, the SLAM processing unit 120 refers to the three-dimensional design information when the accuracy of determining whether or not the object depicted in the captured image 60 is a permanent object is equal to or less than the reference value, and the object is the building 9 It may be determined whether or not it is included in the design of. The reference value of the determination accuracy is not particularly limited.

Note that the process of comparing the three-dimensional design information with the image recognition result may be executed by the conversion unit 102 instead of the SLAM processing unit 120.

In this way, by using the result recognized from the captured image 60 by the trained model together with the three-dimensional design information, the accuracy of the determination result as to whether or not the object drawn in each region A is a permanent object can be determined. Can be improved.

Further, the SLAM processing unit 120 is either one of the segmentation result of the captured image 60 in the present embodiment, the entry / exit information of the person in the second embodiment described above, or the image recognition result of the person 70 in the captured image. Both may be used in combination.

(Modification example 1)
In the first embodiment described above, the initial value of the three-dimensional coordinates of the points in the surrounding object in the SLAM process is obtained based on the three-dimensional design information such as BIM information. In this modification, the three-dimensional coordinates of the points calculated from the distance values estimated from the captured image 60 captured around the information processing device 1 are used as the initial values of the three-dimensional coordinates of the points in the surrounding objects in the SLAM process. Adopt as.

For example, the conversion unit 102 of the information processing device 1 estimates the distance (depth) between the object drawn on the captured image 60 and the imaging device 17 based on the captured image 60. The estimation process is called a depth estimation process.

In this modification, the environmental information includes at least the captured image 60. Alternatively, the distance information estimated from the captured image 60 may be used as an example of the environmental information instead of the captured image 60 itself.

For example, as described in the first embodiment described above, when the image pickup device 17 is a stereo camera, the conversion unit 102 is based on the stereoscopic difference of the captured image captured by one camera included in the stereo camera. , Calculate the depth.

Further, the imaging device 17 may be a monocular camera. In this case, the conversion unit 102 executes the depth estimation process by using the technique of machine learning or deep learning. For example, when the conversion unit 102 inputs the captured image 60 captured by the monocular camera, the conversion unit 102 outputs the depth map corresponding to the captured image 60. The distance between them may be estimated.

The trained model in this modification is, for example, a model that estimates the depth from a monocular image by estimating an image paired with the monocular image from the monocular image using a stereo image as training data. The method of estimating the depth from a monocular image is not limited to this.

The conversion unit 102 sends the distance estimated from the captured image 60 to the SLAM processing unit 120 as an input value for SLAM processing. More specifically, the conversion unit 102 sets the three-dimensional coordinates of the points estimated from the distance estimated from the captured image 60 as the initial values in the bundle adjustment process by the bundle adjustment unit 104. The conversion unit 102 may specify the range of the initial value without specifying the initial value as a unique value.

When the image pickup device 17 is installed at a position away from the center of the information processing device 1, the conversion unit 102 or the SLAM processing unit 120 processes the distance estimated from the captured image 60 with the image pickup device 17. The correction is made based on the deviation of the position from the center of the device 1.

According to this modification, by using the distance between the object estimated from the captured image 60 and the imaging device 17 as the input value of the SALM process, the bundle adjustment process and the like can be calculated even if there is no three-dimensional design information. The amount can be reduced.

Further, the conversion unit 102 is based on both the distance between the object and the image pickup device 17 estimated from the captured image 60 and the distance between the information processing device 1 and the object calculated from the three-dimensional design information. You may generate an input value for the distance to the object. For example, the conversion unit 102 may use the three-dimensional coordinates of the point obtained from the average of the distance estimated from the captured image 60 and the distance calculated from the three-dimensional design information as the initial value in the bundle adjustment process.

(Modification 2)
In the first to fourth embodiments described above, the SLAM processing unit 120 generates a point cloud map, but the mode of the three-dimensional representation is not limited to the point cloud map.

For example, the SLAM processing units 120, 1120, and 2120 (hereinafter, collectively referred to as the SLAM processing unit 120) may generate a set of a plurality of figures having three-dimensional coordinates as map information.

FIG. 13 is a diagram showing an example of map information according to the modified example 2. The map information 500 shown in FIG. 13 is obtained by applying a plurality of triangular figures (triangular-patch-cloud) 501a to 501f (hereinafter referred to as triangular patch 501) to the two-dimensional captured image 45. Is.

Each triangular patch 501 is a plane figure, but its position and orientation can be changed in three-dimensional space. The orientation of the triangular patch 501 is represented by the normal vector n. The position of the triangular patch 501 is represented by three-dimensional coordinates. The position and orientation of each triangular patch 501 corresponds to the depth of the two-dimensional captured image 45.

The SLAM processing unit 120 generates three-dimensional map information by optimizing the positions of the center points and the normal vectors of the plurality of triangular patches 501 applied to the captured image 45.

By generating map information as a set of triangular patches 501 in this way, the information processing device 1 of this modification reduces the amount of calculation as compared with the case where the three-dimensional coordinates of points in the three-dimensional space are individually calculated. At the same time, it is possible to generate map information that closely expresses the surrounding environment.

Further, in FIG. 13, the triangular patch 501 is applied to the captured image 45, but the triangular patch 501 may be applied to the BIM information. For example, the conversion unit 102, 1102, 2102 (hereinafter, referred to as the conversion unit 102) may apply the triangular patch 501 to the three-dimensional design information such as BIM information. For example, the conversion unit 102 can determine the boundary of the triangular patch 501 as the boundary of the three-dimensional structure based on the BIM information, in addition to the boundary drawn as an edge on the captured image.

When adopting this configuration, the SLAM processing unit 120 generates more accurate map information by correcting the positions and orientations of the plurality of triangular patches 501 applied by the conversion unit 102 based on the SLAM result. be able to.

Note that the figures constituting the map information are not limited to the triangular patch 501, and the SLAM processing unit 120 may generate the map information by mesh representation or three-dimensional polygons.

(Modification example 3)
In the first to third embodiments described above, the environmental information is the three-dimensional design information, the entry / exit information of the person in the building 9, or the image recognition result of the person in the captured image, but the environmental information is limited to these. It's not something.

For example, in this modification, the environmental information includes information on at least one of the ambient lighting and the weather. The information regarding the ambient lighting is, for example, information indicating whether the lighting for each room or floor of the building 9 is on or off. The information on the weather is information on sunshine conditions such as sunny, cloudy, and rainy in the area including the building 9. The environmental information may include both information on ambient lighting and information on weather, or may include only one or the other.

For example, the acquisition units 101 and 1101 (hereinafter referred to as the acquisition unit 101) acquire information on ambient lighting or weather from the external device 2.

Further, the conversion unit 102 generates mask information representing a region where the captured image is likely to be deteriorated, based on the information on the ambient lighting or the weather acquired by the acquisition unit 101. The mask information of the second embodiment may be distinguished as the first mask information, and the mask information of the present modification may be distinguished as the second mask information.

The SLAM processing unit 120 of this modification does not use the captured image at least for either self-position estimation or map information generation in the region corresponding to the mask information. For example, the SLAM processing unit 120 may not use the captured image for both the self-position estimation process and the map information generation process in the region corresponding to the mask information, or may not use it for only one of the processes. It may be a thing. For example, the SLAM processing unit 120 may use the captured image for self-position estimation processing for movement in a region corresponding to mask information, or may not use it for generating map information.

Specifically, overexposed areas or underexposed areas may occur in the captured image depending on the lighting or sunshine conditions. The use of such areas may reduce the accuracy of self-position estimation or map information. In this modified example, the deterioration of the accuracy of the self-position estimation or the map information is reduced by not using the captured image for the self-position estimation or the generation of the map information in the region where such an event may occur. ..

Note that the SLAM processing unit 120 may not use the captured image at all for estimating its own position or generating map information in the region corresponding to the mask information, but may use it with a lower priority. For example, when the information processing device 1 includes a sensor or the like for detecting the surrounding state in addition to the image pickup device 17, the SLAM processing unit 120 displays the detection result by the sensor or the like for the area corresponding to the mask information. It is used to estimate the self-position or generate map information in preference to the captured image.

(Modification example 4)
Further, the conversion unit 102 may change the gradation of the captured image based on the environmental information. For example, the conversion unit 102 reduces overexposure or underexposure by changing the dynamic range of the captured image based on information about ambient lighting or weather.

In this case, the SLAM processing unit 120 executes self-position estimation and map information generation based on the captured image whose gradation has been changed by the conversion unit 102.

Therefore, according to the information processing device 1 of this modification, it is possible to robustly estimate the self-position and generate map information in response to the surrounding environment such as lighting conditions or sunshine conditions.

(Modification 5)
Further, the environmental information may include three-dimensional design information and process information representing the construction process of the building 9.

The process information in this modification is information representing the construction schedule or timeline (schedule) of the building 9. When the building 9 is in the process of being built, by collating the three-dimensional design information such as BIM information with the process information, it becomes possible to distinguish between the area where the building 9 has been constructed and the area where the building 9 is in the process of being constructed. Since the 3D design information basically represents a 3D model of the building 9 in the completed state of construction, there may be a difference between the 3D design information and the actual state of the building 9 in the area in the middle of construction. Highly sexual.

The conversion unit 102 of this modification generates unfinished area information representing an area of the building 9 in which the construction has not been completed, based on the three-dimensional design information and the process information.

The SLAM processing unit 120 of this modified example executes self-position estimation and map information generation for the area corresponding to the unfinished area information without using the three-dimensional design information. For example, the SLAM processing unit 120 executes self-position estimation and map information generation based on the detection result of the captured image or the sensor for the region corresponding to the unfinished region information.

Therefore, the information processing device 1 of this modified example does not use the three-dimensional design information in the region where there is a high possibility that there is a difference between the three-dimensional design information and the actual state of the building 9, so that the building 9 can be used. Even in the middle of construction, it is possible to reduce the decrease in the accuracy of self-position estimation and map information.

Note that the SLAM processing unit 120 may use the three-dimensional design information at a lower priority in the area corresponding to the unfinished area information, instead of not using the three-dimensional design information at all.

(Modification 6)
Further, the use of the unfinished area information described in the modified example 5 is not limited to the above-mentioned example.

For example, when the SLAM processing unit 120 generates the map information of the building 9, the map information may be generated only in the area corresponding to the unfinished area information. That is, the SLAM processing unit 120 estimates that the structure of the building 9 does not change in the area where the construction is completed, and the map information is only in the area where the structure of the building 9 changes, that is, the area corresponding to the unfinished area information. The amount of calculation is reduced by generating.

Further, when the tracking unit 103 of the SLAM processing unit 120 obtains the position and orientation of the image pickup device 17 in the tracking process, the tracking unit 103 uses the captured image obtained by capturing the area other than the area corresponding to the unfinished area information in the tracking process. May be done. This is because, for the area corresponding to the unfinished area information, the structure that is the subject changes depending on the construction work, so it may be difficult to track the point 50 between the captured images captured at different times. Because there is.

(Modification 7)
Further, in the first to third embodiments described above, an example has been described in which the information processing device 1 executes the current self-position estimation process and the map information generation process in real time while moving in the building 9. However, the execution timing of the self-position estimation process and the map information generation process is not limited to this. For example, after the movement is completed, the information processing device 1 estimates its own position or generates map information based on the detection result of the surrounding state of the information processing device 1 or the state of the information processing device 1 detected during the movement. You may execute the process.

(Modification 8)
Further, in the first to third embodiments described above, the information processing device 1 executes the self-position estimation process and the map information generation process, but the external device 2 estimates the self-position. A configuration that executes processing and generation processing of map information may be adopted. For example, the external device 2 may execute the position estimation process of the information processing device 1 and the map information generation process based on the detection result acquired from the information processing device 1 and the environmental information. In this case, the external device 2 may be used as an example of the information processing device.

In the present specification (including claims), the expression "at least one of a, b and c (one)" or "at least one of a, b or c (one)" (including similar expressions). ) Is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as a-a, a-b-b, a-a-b-b-c-c, and the like. It also includes adding elements other than the listed elements (a, b and c), such as having d, such as a-b-c-d.

In the present specification (including claims), when expressions such as "with data as input / based on / according to / according to" (including similar expressions) are used, unless otherwise specified. This includes the case where various data itself is used as an input, and the case where various data are processed in some way (for example, noise-added data, normalized data, intermediate representation of various data, etc.) are used as input. In addition, when it is stated that some result can be obtained "based on / according to / according to the data", it includes the case where the result can be obtained based only on the data, and other data other than the data. It may also include cases where the result is obtained under the influence of factors, conditions, and / or conditions. In addition, when it is stated that "data is output", unless otherwise specified, various data itself is used as output, or various data is processed in some way (for example, noise is added, normal). It also includes the case where the output is output (intermediate representation of various data, etc.).

In the present specification (including claims), when the terms "connected" and "coupled" are used, direct connection / coupling and indirect connection / coupling are used. , Electrically connected / combined, communicatively connected / combined, operatively connected / combined, physically connected / combined, etc. Intended as a term. The term should be interpreted as appropriate according to the context in which the term is used, but any connection / combination form that is not intentionally or naturally excluded is not included in the term. It should be interpreted in a limited way.

When the expression "A configured to B" is used in the present specification (including claims), the physical structure of the element A can execute the operation B. Including that the element A has a configuration and the permanent or temporary setting (setting / configuration) of the element A is set (configured / set) to actually execute the operation B. good. For example, when the element A is a general-purpose processor, the processor has a hardware configuration capable of executing the operation B, and the operation B is set by setting a permanent or temporary program (instruction). It suffices if it is configured to actually execute. Further, when the element A is a dedicated processor, a dedicated arithmetic circuit, or the like, the circuit structure of the processor actually executes the operation B regardless of whether or not the control instruction and data are actually attached. It only needs to be implemented.

In the present specification (including claims), when a term meaning inclusion or possession (for example, "comprising / including" and "having", etc.) is used, the object of the term is used. It is intended as an open-ended term, including the case of containing or owning an object other than the indicated object. If the object of these terms that mean inclusion or possession is an expression that does not specify a quantity or suggests a singular (an expression with a or an as an article), the expression is interpreted as not being limited to a specific number. It should be.

In this specification (including claims), expressions such as "one or more" or "at least one" are used in some places, and the quantity is specified in other places. Even if expressions that do not or suggest the singular (expressions with a or an as an article) are used, the latter expression is not intended to mean "one". In general, expressions that do not specify a quantity or suggest a singular (expressions with a or an as an article) should be interpreted as not necessarily limited to a particular number.

In the present specification, when it is stated that a specific effect (advantage / result) can be obtained for a specific configuration having an embodiment, unless there is a specific reason, another one or more having the configuration. It should be understood that the effect can also be obtained in the examples of. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, etc., and that the effect cannot always be obtained by the configuration. The effect is merely obtained by the configuration described in the examples when various factors, conditions, and / or conditions are satisfied, and in the invention relating to the claim that defines the configuration or a similar configuration. , The effect is not always obtained.

In the present specification (including claims), when terms such as "maximize" are used, the global maximum value is obtained, the approximate value of the global maximum value is obtained, and the local maximum value is obtained. Should be interpreted as appropriate according to the context in which the term is used, including finding an approximation of the local maximum. It also includes probabilistically or heuristically finding approximate values of these maximum values. Similarly, when terms such as "minimize" are used, find the global minimum, find the approximation of the global minimum, find the local minimum, and find the local minimum. It should be interpreted as appropriate according to the context in which the term was used, including finding an approximation of the value. It also includes probabilistically or heuristically finding approximate values of these minimum values. Similarly, when terms such as "optimize" are used, finding a global optimal value, finding an approximation of a global optimal value, finding a local optimal value, and local optimization It should be interpreted as appropriate according to the context in which the term was used, including finding an approximation of the value. It also includes probabilistically or heuristically finding approximate values of these optimal values.

In the present specification (including claims), when a plurality of hardware performs a predetermined process, the respective hardware may cooperate to perform the predetermined process, or some hardware may perform the predetermined process. You may do all of the above. Further, some hardware may perform a part of a predetermined process, and another hardware may perform the rest of the predetermined process. In the present specification (including claims), when expressions such as "one or more hardware performs the first process and the one or more hardware performs the second process" are used. , The hardware that performs the first process and the hardware that performs the second process may be the same or different. That is, the hardware that performs the first process and the hardware that performs the second process may be included in the one or more hardware. The hardware may include an electronic circuit, a device including the electronic circuit, or the like.

In the present specification (including claims), when a plurality of storage devices (memory) store data, each storage device (memory) among the plurality of storage devices (memory) stores only a part of the data. It may be stored or the entire data may be stored.

As described above, according to the first to third embodiments, it is possible to improve the estimation of the self-position and the accuracy of the map information.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, changes, replacements, partial deletions, etc. are possible without departing from the conceptual idea and purpose of the present invention derived from the contents specified in the claims and their equivalents. For example, in all the above-described embodiments, when numerical values or mathematical formulas are used for explanation, they are shown as examples, and the present invention is not limited thereto. Further, the order of each operation in the embodiment is shown as an example, and is not limited to these.

Claims (17)

With at least one memory
With at least one processor,
The at least one processor
Acquiring the detection result including any one of the surrounding state of the information processing device and the state of the information processing device, and the environmental information regarding the environment around the information processing device.
Performing self-position estimation and map information generation based on the environmental information and the detection result, and
Is configured to be executable,
Information processing device. The environmental information includes three-dimensional design information of the building.
The information processing device according to claim 1. The at least one processor
Based on the 3D design information, information about the position of surrounding objects is generated.
Based on the information about the position of the surrounding object, the estimation of the self-position and the generation of the map information are executed.
The information processing device according to claim 2. The detection result includes a plurality of captured images captured by an imaging device mounted on the information processing device.
The at least one processor
Based on the plurality of captured images, the position and orientation of the imaging device are estimated, and the position and orientation of the imaging device are estimated.
Based on the three-dimensional design information and the position and orientation of the imaging device, information regarding the position of the surrounding object is generated.
The information processing device according to claim 3. The at least one processor
Based on the 3D design information, the position of the plane or curved surface of the surrounding object is estimated.
Generates information about the position of the surrounding object based on the constraint that a plurality of points existing in the surroundings are located on the plane or the curved surface.
The information processing device according to claim 4. The at least one processor
From the detection result, the index information whose position is represented by the coordinate system in the three-dimensional design information is detected.
Based on the detection result of the index information, the coordinate system representing the self-position is adjusted so as to match the coordinate system of the three-dimensional design information.
The information processing device according to any one of claims 2 to 5. The at least one processor
Applying a set of a plurality of figures having three-dimensional coordinates to the three-dimensional design information,
Based on the estimation result of the self-position, the positions and orientations of the plurality of figures are corrected.
A set of the plurality of figures whose positions and orientations have been corrected is generated as the map information.
The information processing device according to any one of claims 2 to 6. The environmental information includes the three-dimensional design information and process information representing the construction process of the building.
The at least one processor
Based on the three-dimensional design information and the process information, unfinished area information representing an area where construction has not been completed in the building is generated.
For the region corresponding to the unfinished region information, the estimation of the self-position and the generation of the map information are executed without using the three-dimensional design information.
The information processing device according to any one of claims 2 to 7. The at least one processor
Based on the environmental information, mask information representing an area to be excluded from the generation target of the map information is generated.
The map information is not generated for the area corresponding to the mask information.
The information processing device according to any one of claims 1 to 8. The environmental information includes either one of the entry / exit information of a person in the building where the information processing device is located or the image recognition result of the person in the captured image captured by the image pickup device mounted on the information processing device. ,
The at least one processor
The mask information is generated based on either the entry / exit information or the image recognition result of the person.
The information processing device according to claim 9. The at least one processor
The captured image captured by the image pickup device mounted on the information processing device is divided into regions based on the recognition result of the object drawn on the captured image.
Based on the result of the region division, the estimation of the self-position and the generation of the map information are executed.
The information processing device according to any one of claims 1 to 10. The environmental information includes information about any one of the ambient lighting or the weather.
The detection result includes an image captured by an image pickup device mounted on the information processing device.
The at least one processor
Based on the environmental information, a second mask information representing a region where the captured image is likely to be deteriorated is generated.
For the region corresponding to the second mask information, the captured image is not used at least in either the estimation of the self-position or the generation of the map information.
The information processing device according to any one of claims 1 to 10. The environmental information includes information about any one of the ambient lighting or the weather.
The detection result includes an image captured by an image pickup device mounted on the information processing device.
The at least one processor
Based on the environmental information, the gradation of the captured image is changed.
Based on the captured image whose tone has been changed, the estimation of the self-position and the generation of the map information are executed.
The information processing device according to any one of claims 1 to 12. The environmental information includes an image captured by an image pickup device mounted on the information processing device.
The at least one processor
Based on the captured image, information about the position of surrounding objects is generated.
Based on the information about the position of the surrounding object, the estimation of the self-position and the generation of the map information are executed.
The information processing device according to any one of claims 1 to 13. The at least one processor
The position of the surrounding object is calculated as the spatial coordinates of multiple points in the three-dimensional space.
The calculated spatial coordinates of the plurality of points are output as the map information.
The information processing device according to any one of claims 1 to 14. A step of acquiring a detection result including any one of the surrounding state of the information processing device and the state of the information processing device and environmental information about the environment around the information processing device by at least one processor.
A step of performing self-position estimation and map information generation based on the environmental information and the detection result by the at least one processor.
Information processing methods including. A step of acquiring a detection result including at least one of the surrounding state of the information processing device and the state of the information processing device, and environmental information regarding the environment around the information processing device.
A step of estimating the self-position and generating map information based on the environmental information and the detection result, and
A program that causes at least one computer to run.

Images