[go: up one dir, main page]

WO2021210492A1 - Dispositif de traitement d'informations, procédé de traitement d'informations et programme - Google Patents

Dispositif de traitement d'informations, procédé de traitement d'informations et programme Download PDF

Info

Publication number
WO2021210492A1
WO2021210492A1 PCT/JP2021/014938 JP2021014938W WO2021210492A1 WO 2021210492 A1 WO2021210492 A1 WO 2021210492A1 JP 2021014938 W JP2021014938 W JP 2021014938W WO 2021210492 A1 WO2021210492 A1 WO 2021210492A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
processing device
information processing
unit
self
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/014938
Other languages
English (en)
Japanese (ja)
Inventor
貴之 猿田
亮 水谷
達雄 古賀
仁紀 木内
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kajima Corp
Preferred Networks Inc
Original Assignee
Kajima Corp
Preferred Networks Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kajima Corp, Preferred Networks Inc filed Critical Kajima Corp
Publication of WO2021210492A1 publication Critical patent/WO2021210492A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/10—Simultaneous control of position or course in three dimensions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B29/00—Maps; Plans; Charts; Diagrams, e.g. route diagram

Definitions

  • An embodiment of the present invention relates to an information processing device, an information processing method, and a program.
  • robots and the like that estimate their own position and generate map information by recognizing the position and shape of surrounding objects from the sensing result by the sensor or the captured image are known.
  • the information processing device of the embodiment includes at least one memory and at least one processor. At least one processor acquires a detection result including either the surrounding state of the information processing device or the state of the information processing device, and environmental information about the environment around the information processing device, and the environmental information. Based on the information processing and the detection result, it is possible to estimate the self-position and generate map information.
  • FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus according to the first embodiment.
  • FIG. 2 is a block diagram showing an example of a function provided in the information processing apparatus according to the first embodiment.
  • FIG. 3 is an image diagram showing an example of tracking processing according to the first embodiment.
  • FIG. 4 is an image diagram showing an example of the positional relationship between the information processing apparatus according to the first embodiment and surrounding objects.
  • FIG. 5 is an image diagram showing an example of bundle adjustment according to the first embodiment.
  • FIG. 6 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the first embodiment.
  • FIG. 7 is a block diagram showing an example of the functions provided in the information processing apparatus according to the second embodiment.
  • FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus according to the first embodiment.
  • FIG. 2 is a block diagram showing an example of a function provided in the information processing apparatus according to the first embodiment.
  • FIG. 3 is an image
  • FIG. 8 is an image diagram showing an example of the positional relationship between the information processing apparatus according to the second embodiment and surrounding objects.
  • FIG. 9 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the second embodiment.
  • FIG. 10 is a block diagram showing an example of the functions provided in the information processing apparatus according to the third embodiment.
  • FIG. 11 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the third embodiment.
  • FIG. 12 is a diagram showing an example of segmentation of a captured image according to a fourth embodiment.
  • FIG. 13 is a diagram showing an example of map information according to the second modification.
  • FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus 1 according to the first embodiment.
  • the information processing device 1 includes a main body 10, a moving device 16, an imaging device 17, and an IMU (Inertial Measurement Unit) sensor 18.
  • IMU Inertial Measurement Unit
  • the moving device 16 is a device capable of moving the information processing device 1.
  • the moving device 16 has a plurality of wheels and a motor for driving these wheels, and is connected to the lower part of the main body 10 so as to support the main body 10.
  • the information processing device 1 can be moved by the mobile device 16, for example, in a building under construction, a built building, a platform of a station, a factory, or the like.
  • the case where the information processing device 1 moves in the building under construction will be described as an example.
  • the means of transportation of the information processing device 1 is not limited to wheels, and may be caterpillars, propellers, or the like.
  • the information processing device 1 is, for example, a robot, a drone, or the like. In the present embodiment, the information processing device 1 is supposed to move autonomously, but the information processing device 1 is not limited to this.
  • the image pickup device 17 is, for example, a stereo camera in which two cameras arranged side by side are set as one set.
  • the image pickup device 17 transmits the captured image data captured by the two cameras to the main body 10 in association with each other.
  • the IMU sensor 18 is a sensor in which a gyro sensor, an acceleration sensor, and the like are integrated, and measures the angular velocity and acceleration of the information processing device 1.
  • the IMU sensor 18 sends the measured angular velocity and acceleration to the main body 10.
  • the IMU sensor 18 may further include not only a gyro sensor and an acceleration sensor, but also a magnetic sensor, a GPS (Global Positioning System) device, and the like.
  • the image pickup device 17 and the IMU sensor 18 are collectively referred to as a detection unit.
  • the detection unit may further include various sensors.
  • the information processing device 1 may further include a distance measuring sensor such as an ultrasonic sensor or a laser scanner.
  • the term "detection” refers to imaging the surroundings of the information processing device 1, measuring the angular velocity or acceleration of the information processing device 1, and the distance to an object around the information processing device 1. It shall include measuring the distance.
  • the detection result by the detection unit includes at least one of the surrounding state of the information processing device 1 and the state of the information processing device 1.
  • the detection result may include both information about the surrounding state of the information processing device 1 and the state of the information processing device 1, or may relate to the surrounding state of the information processing device 1 and the state of the information processing device 1. It may contain only one of the information.
  • the surrounding state of the information processing device 1 is, for example, an captured image of the surroundings of the information processing device 1, a distance measurement result of a distance between an object around the information processing device 1 and the information processing device 1.
  • the state of the information processing device 1 is, for example, the angular velocity and acceleration measured by the IMU sensor 18.
  • the captured image captured by the imaging device 17 is an example of the detection result of the surrounding state of the information processing device 1.
  • the detection result includes at least the captured image, but may further include other information.
  • the main body 10 includes a processor 11, a main storage device 12 (memory), an auxiliary storage device 14 (memory), a network interface 13, and a device interface 15, which are routed via a bus 19. It may be realized as a connected computer.
  • the image pickup device 17 and the IMU sensor 18 may be incorporated in the main body 10.
  • the processor 11 is an electronic circuit (processing circuit, Processing circuit, Processing circuitry, CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), or ASIC (Processing circuit, Processing circuit, Processing circuitry, CPU (Central Processing Unit)) including a computer control device and a computing device. Application Specific Integrated Circuit), etc.) may be used. Further, the processor 11 may be a semiconductor device or the like including a dedicated processing circuit. The processor 11 is not limited to an electronic circuit using an electronic logic element, and may be realized by an optical circuit using an optical logic element. Further, the processor 11 may include a calculation function based on quantum computing.
  • the processor 11 can perform arithmetic processing based on the data and software (program) input from each apparatus and the like of the internal configuration of the information processing apparatus 1, and output the arithmetic result and the control signal to each apparatus and the like.
  • the processor 11 may control each component constituting the information processing device 1 by executing an OS (Operating System) of the information processing device 1, an application, or the like.
  • OS Operating System
  • the main storage device 12 is a storage device that stores instructions executed by the processor 11, various data, and the like, and the information stored in the main storage device 12 is read out by the processor 11.
  • the auxiliary storage device 14 is a storage device other than the main storage device 12. Note that these storage devices mean arbitrary electronic components capable of storing electronic information, and may be semiconductor memories.
  • the semiconductor memory may be either a volatile memory or a non-volatile memory.
  • the storage device for storing various data in the information processing device 1 in the present embodiment may be realized by the main storage device 12 or the auxiliary storage device 14, or may be realized by the built-in memory built in the processor 11. ..
  • the main storage device 12 or the auxiliary storage device 14 is also referred to as a storage unit.
  • processors may be connected (combined) to one storage device (memory), or a single processor may be connected.
  • a plurality of storage devices (memory) may be connected (combined) to one processor.
  • the information processing device 1 in the present embodiment is composed of at least one storage device (memory) and a plurality of processors connected (combined) to the at least one storage device (memory), at least one of the plurality of processors
  • One processor may include a configuration in which it is connected (combined) to at least one storage device (memory). Further, this configuration may be realized by a storage device (memory) and a processor included in a plurality of computers. Further, a configuration in which the storage device (memory) is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.
  • the network interface 13 is an interface for connecting to the communication network 3 wirelessly or by wire.
  • an appropriate interface such as one conforming to an existing communication standard may be used.
  • Information may be exchanged with the external device 2 connected via the communication network 3 by the network interface 13.
  • the communication network 3 may be any one of WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, and the information processing device 1 and the external device 2 may be used. It suffices as long as information is exchanged with.
  • WAN Wide Area Network
  • LAN Local Area Network
  • PAN Personal Area Network
  • An example of WAN is the Internet
  • an example of LAN is IEEE802.11, Ethernet (registered trademark), etc.
  • PAN is Bluetooth (registered trademark), NFC (Near Field Communication), etc.
  • the device interface 15 is an interface that directly connects to the mobile device 16, the image pickup device 17, and the IMU sensor 18.
  • the device interface 15 is an interface that conforms to a standard such as USB (Universal Serial Bus), but is not limited thereto. Further, the device interface 15 may be further connected to an external device other than the various devices shown in FIG.
  • the external device 2 is, for example, a server device or the like.
  • the external device 2 is connected to the information processing device 1 via a communication network 3.
  • the external device 2 of this embodiment stores the three-dimensional design information of the building in advance.
  • the three-dimensional design information of the building is, for example, BIM (Building Information Modeling) information.
  • BIM information includes information on the three-dimensional structure of a building and information on materials such as building materials.
  • the three-dimensional design information of the building is not limited to BIM information, and may be 3D CAD (Computer-Aided Design) data or the like.
  • the three-dimensional design information is an example of environmental information in this embodiment.
  • the environmental information is information about the environment around the information processing device 1.
  • the environmental information includes at least one of information about a building in which the information processing device 1 travels, information about a person or an object existing around the information processing device 1, information about the weather, and information about lighting.
  • the above-mentioned three-dimensional design information is an example of information on the building in which the information processing device 1 travels among the environmental information.
  • the environmental information may be a combination of a plurality of types of information, or may include only one type of information.
  • the environmental information includes at least three-dimensional design information, but may further include other information regarding the environment around the information processing device 1.
  • object in this embodiment includes structures such as walls and pillars, furniture, furniture, moving objects, temporary objects, people, and the like.
  • the information processing device 1 and the external device 2 are wirelessly connected, but the information processing device 1 and the external device 2 may be connected by wire. Further, the information processing device 1 does not have to be always connected to the external device 2.
  • FIG. 2 is a block diagram showing an example of the functions included in the information processing apparatus 1 according to the first embodiment.
  • the information processing device 1 includes an acquisition unit 101, a conversion unit 102, a SLAM (Simultaneous Localization and Mapping, or Simultaneously Localization and Mapping) processing unit 120, and a movement control unit 105. Further, the SLAM processing unit 120 includes a tracking unit 103 and a bundle adjustment unit 104.
  • SLAM Simultaneous Localization and Mapping, or Simultaneously Localization and Mapping
  • the acquisition unit 101 acquires the detection result of the surrounding state of the information processing device 1 or the state of the information processing device 1 and the environmental information regarding the environment around the information processing device 1.
  • the acquisition unit 101 acquires BIM information from the external device 2 via, for example, the network interface 13.
  • the acquisition unit 101 stores the acquired BIM information in the auxiliary storage device 14.
  • the acquisition unit 101 acquires an captured image from the imaging device 17 via the device interface 15. In addition, the acquisition unit 101 acquires the angular velocity and acceleration from the IMU sensor 18 via the device interface 15.
  • the conversion unit 102 converts the environmental information into an input value of at least one of the self-position estimation process by the SLAM processing unit 120 described later or the map information generation process.
  • the conversion unit 102 may convert the environmental information into the input values of both the self-position estimation process and the map information generation process, or may convert only the input values of either process.
  • conversion includes generating other information from the environmental information or extracting, acquiring or searching the information from the environmental information.
  • the conversion unit 102 generates the initial value of the three-dimensional coordinates (world coordinates) of the points in the three-dimensional space used in the bundle adjustment process by the bundle adjustment unit 104 described later from the BIM information. ..
  • the conversion unit 102 has an imaging range of the imaging device 17 among the structures such as walls and columns included in the BIM information based on the current position and orientation of the imaging device 17 specified by the tracking unit 103 described later. Identify the structures contained in. Then, the conversion unit 102 specifies the three-dimensional coordinates (world coordinates) of the structure included in the imaging range of the imaging device 17 from the imaging device 17 from the BIM information. As an example, the conversion unit 102 acquires the world coordinates of any one of the buildings represented by the BIM information from an external device or the like, and includes the building represented by the BIM information with the one point as a reference. The three-dimensional coordinates in the BIM information of each point are converted into world coordinates.
  • the method of obtaining the world coordinates of each point included in the building from the BIM information is not limited to this.
  • the conversion unit 102 sends the specified three-dimensional coordinates (world coordinates) to the bundle adjustment unit 104 as the initial value of the three-dimensional coordinates (world coordinates) of the points in the three-dimensional space in the bundle adjustment process described later.
  • the details of the bundle adjustment process will be described later.
  • the three-dimensional coordinates (world coordinates) of the points included in the building 9 specified by the conversion unit 102 from the BIM information are examples of information regarding the positions of surrounding objects in the present embodiment.
  • the three-dimensional coordinates in the present embodiment are world coordinates.
  • the conversion unit 102 may specify the range of the initial value without specifying the initial value as a unique value.
  • the conversion unit 102 may provide a range instead of specifying the three-dimensional coordinates of a point in the three-dimensional space as unique coordinates.
  • a three-dimensional space region that is likely to include a certain point included in the structure included in the imaging range of the imaging device 17 is sent to the bundle adjusting unit 104 as a range of initial values.
  • the initial value or the range of the initial value is an example of the input value generated by the conversion unit 102 in the present embodiment.
  • the calibration means that the correspondence between the position in the BIM information and the position in the SLAM coordinate system is defined.
  • the position of the movement start point of the information processing device 1 in the BIM information in the three-dimensional coordinate system may be stored in the auxiliary storage device 14 as a reference point. Therefore, the conversion unit 102 can specify the position in the building represented by the BIM information corresponding to the position of the image pickup device 17 specified by the tracking unit 103.
  • the calibration of the 3D coordinate system and the SLAM coordinate system in the BIM information may be executed by an input operation by an administrator or the like, or an index such as an AR (Augmented Reality) marker installed in the building may be used as an index of the SLAM processing unit. It may be executed by recognizing 120 from the captured image.
  • an index such as an AR (Augmented Reality) marker installed in the building may be used as an index of the SLAM processing unit. It may be executed by recognizing 120 from the captured image.
  • the SLAM processing unit 120 simultaneously estimates the self-position and generates map information.
  • the self-position is the position and orientation of the information processing device 1.
  • the position and orientation of the image pickup device 17 represent the same as the position and posture of the information processing device 1.
  • the SLAM processing unit 120 corrects the displacement between the image pickup apparatus 17 and the center of the information processing apparatus 1 to provide information.
  • the self-position which is the position of the processing device 1, is estimated.
  • the map information represents the shape of the surrounding structure along the movement locus of the information processing device 1. More specifically, the map information of the present embodiment represents the internal structure of the building in which the information processing device 1 travels in three dimensions along the movement locus of the information processing device 1.
  • the map information of the present embodiment is, for example, a point cloud map in which the internal structure of the building in which the information processing device 1 travels is represented as a point cloud having three-dimensional coordinates.
  • the type of map information is not limited to this, and the map may be represented by a set of three-dimensional figures instead of a point cloud.
  • the map information is also called an environment map.
  • the SLAM processing unit 120 is an example of an estimation unit in this embodiment.
  • a method for estimating the self-position and generating map information a method other than SLAM may be adopted. Further, the estimation of the self-position and the generation of the map information do not have to be performed at the same time, and the other process may be executed after one process is completed first.
  • the generation of map information in the present specification includes at least one of newly generating map information, adjusting the generated map information, or updating the generated map information. It shall be.
  • the SLAM processing unit 120 includes a tracking unit 103 and a bundle adjustment unit 104.
  • the tracking unit 103 identifies the position and orientation of the image pickup device 17 by tracking a plurality of captured images captured by the image pickup device 17 at different times.
  • the tracking unit 103 is an example of a specific unit in the present embodiment.
  • the image pickup device 17 captures the surroundings while moving as the information processing device 1 moves.
  • the tracking unit 103 calculates changes in the position and posture of the image pickup device 17 by tracking points drawn on a certain captured image on another captured image captured at different times.
  • the tracking unit 103 specifies the current position and orientation of the imaging device 17 by adding changes in the position and orientation specified by the tracking process to the position and orientation of the imaging device 17 at the start of imaging.
  • FIG. 3 is an image diagram showing an example of tracking processing according to the first embodiment.
  • the reference frame 41 and the target frame 42 are captured images captured at different times by the imaging device 17.
  • the reference frame 41 is an captured image captured before the target frame 42, and the imaging device 17 changes from the position Ti at the time when the reference frame 41 is imaged to the position T j at the time when the target frame 42 is imaged. It is assumed that it has moved.
  • the reference frame 41 is also referred to as a key frame
  • the target frame 42 is also referred to as a current frame.
  • the tracking unit 103 P point which is depicted in the reference frame 41, by calculating the photometry error in the case where it is depicted in the target frame 42, the image pickup device 17 is moved from the position T i to the position T j Calculate the relative amount of movement in the case.
  • the point P is, for example, a feature point on the reference frame 41.
  • the movement of the image pickup apparatus 17 includes both a change in the position of the image pickup apparatus 17 and a change in the posture (orientation).
  • the position T i at the time of the reference frame 41 and the image pickup is assumed to be already error-corrected.
  • the point 50a shown in FIG. 3 represents the position where the point P drawn on the reference frame 41 is back-projected on the three-dimensional space.
  • the tracking unit 103 calculates the photometric error Epj between the reference frame 41 and the target frame 42 by using the following equation (1).
  • I i represents the reference frame 41 and I j represents the target frame 42.
  • N p is a neighborhood pattern of pixels including the point P on the reference frame 41.
  • t i is the exposure time of the reference frame 41
  • t j represents the exposure time of the target frame 42.
  • p' is due to the inverse depth d p, a projected point P in the target frame 42.
  • the tracking unit 103 calculates the photometric error Epj using the Huber norm.
  • the weighting coefficient W p is calculated in advance based on the brightness gradient of the pixels. For example, with respect to the gradient is larger pixel by reducing the value of the weight factor W p, it is possible to reduce noise.
  • the luminance conversion hyperparameter a i, a j, b i , b j is a parameter for converting the luminance of the reference frame 41 and the subject frame 42.
  • Luminance conversion hyperparameter a i, a j, b i , b j may be tuned manually for example by the administrator.
  • the following equation (2) is a constraint condition of the point P'which is the projection point of the point P used in the equation (1).
  • a back projection function that back-projects the point P drawn on the reference frame 41 as a point 50a on the three-dimensional space and a projection function that projects the point 50a on the three-dimensional space onto the target frame 42. And are used.
  • the distance from the point P to the point 50a is the depth (d p ) of the point 50a in the reference frame 41.
  • the coefficient R included in the equation (2) represents the amount of rotation of the image pickup apparatus 17.
  • the coefficient t represents the translational amount of the imaging device 17.
  • the coefficient R and the coefficient t are defined by the relative position of the image pickup apparatus 17 according to the following constraint condition (3).
  • the tracking unit 103 takes an image at the time when the target frame I j is imaged by solving the model of the photometric error Epj between the reference frame I i and the target frame I j shown in the above equations (1) to (3).
  • the position Tj of the device 17 is specified.
  • the position Ti and the position T j shown in the equation (3) and FIG. 3 include the position and orientation of the image pickup apparatus 17. In this way, the tracking unit 103 tracks changes in the position and posture of the imaging device 17 by repeatedly executing such tracking processing on a plurality of captured images captured in time series by the imaging device 17. do.
  • the tracking method is not limited to the above example.
  • the tracking method is an indirect method (indirect method) in which the position and orientation of the image pickup device 17 at the time of imaging of each frame are acquired by acquiring the feature points on the captured image and then solving the matching problem of the feature points.
  • the Direct method direct method in which the position and orientation of the image pickup apparatus 17 at the time of imaging of each frame are estimated by directly estimating the conversion between the captured images without the feature point extraction process.
  • the movement of the position and the posture of the image pickup apparatus 17 is calculated by projecting the feature points, but the tracking unit 103 may execute the tracking by the direct method. Further, the tracking unit 103 may specify the position and orientation of the image pickup device 17 in consideration of not only the captured image but also the detection result of the IMU sensor 18.
  • the tracking unit 103 sends the current position and orientation of the specified imaging device 17 to the bundle adjustment unit 104 and the conversion unit 102.
  • the bundle adjustment unit 104 corrects the position and orientation of the image pickup device 17 specified by the tracking unit 103 and the position information of surrounding objects by the bundle adjustment process.
  • the bundle adjustment unit 104 outputs the self-position of the information processing device 1 and the map information as the processing result.
  • the bundle adjustment unit 104 sends the captured image captured by the image pickup device 17 to the bundle adjustment unit 104 that minimizes the reprojection error for each frame.
  • the bundle adjustment unit 104 optimizes the world coordinate points (three-dimensional position coordinates) of each point in the surrounding environment, the position and orientation of the image pickup device 17, and the internal parameters of the image pickup device 17, respectively. Minimize frame reprojection error.
  • the internal parameters of the imaging device 17 do not have to be updated by the bundle adjustment unit 104 if the camera has been calibrated in advance.
  • the internal parameters of the image pickup device 17 are, for example, the focal length and the principal point. In bundle adjustment, the position and orientation of the imaging device 17 are also referred to as external parameters.
  • the bundle adjustment unit 104 of the present embodiment uses the three-dimensional coordinates indicating the positions of the surrounding structures converted from the BIM information by the conversion unit 102 as the initial values of the world coordinate points of each point in the surrounding environment as described above. adopt.
  • the bundle adjustment unit 104 adjusts the error of the position and orientation of the image pickup device 17 specified by the tracking unit 103 by this bundle adjustment.
  • the bundle adjustment unit 104 obtains the world coordinate points of each point at which the reprojection error is minimized by the bundle adjustment, the position and orientation of the image pickup device 17, and the internal parameters of the image pickup device 17, so that the error is reduced as a result.
  • the position and orientation of the image pickup device 17 are calculated.
  • the set of world coordinate points after bundle adjustment becomes map information.
  • FIG. 4 is an image diagram showing an example of the positional relationship between the information processing device 1 and surrounding objects according to the first embodiment.
  • the information processing device 1 is assumed to move in the building 9 in which the pillars 90a to 90c are installed. Pillars 90a to 90c are examples of objects.
  • the distance d in FIG. 4 is the distance from the image pickup device 17 to the point 52 on the plane 901 facing the information processing device 1 of the pillar 90c.
  • the conversion unit 102 specifies the initial value of the three-dimensional coordinates of the point 52 on the plane 901.
  • the bundle adjustment unit 104 starts the adjustment process from the initial value, and based on the position and orientation of the image pickup device 17 specified by the tracking unit 103 and the captured image, the self-position and the position of the point 52 Adjust the error.
  • the conversion unit 102 corrects the three-dimensional coordinates of the point 52 by adjusting the error between the self-position and the position of the point 52, and obtains the three-dimensional coordinates with higher accuracy.
  • the bundle adjustment unit 104 estimates the self-position and the three-dimensional coordinates of the point 52.
  • the bundle adjustment unit 104 changes the initial value based on the BIM information by the bundle adjustment, thereby causing the BIM.
  • the position of an object that is not included in the information can also be estimated.
  • FIG. 5 is an image diagram showing an example of bundle adjustment according to the first embodiment.
  • the bundle adjustment unit 104 captures the projection points 401a and 401b on which the points 52 in the three-dimensional space are projected on the two captured images 43 and 44 shown in FIG. 5 by the following equation (4).
  • the position of the image pickup apparatus 17 and the three-dimensional coordinates of the point 52 are estimated so as to minimize the error from the feature points 402a and 402b corresponding to the points 52 drawn on the images 43 and 44.
  • the captured image 43 and the captured image 44 are distinguished, the captured image 43 is referred to as a first image and the captured image 44 is referred to as a second image for convenience.
  • the internal parameters of the image pickup apparatus 17 are assumed to have been calibrated in advance, and are not included in the parameters to be optimized in the equation (4).
  • the initial value generated by the conversion unit 102 described above as an initial value in the world coordinate point (X i point in the three-dimensional space) shown as point 52, it is used in equation (4). Further, in FIG. 5, the lines connecting the reference points 170a and 170b representing the position of the image pickup apparatus 17 and the points 52 are referred to as ray bundles (Bundle) 6a and 6b. Also, when the range of the initial value is set by the converter 102, the equation (4), the point X i on the three-dimensional space, calculation is started world coordinate included in the range is set .. In the case where the error in the calculation of the range of the initial values is not minimized, beyond the scope of the initial value, the value of the point X i on the optimum three-dimensional space may be determined.
  • the bundle adjusting unit 104 estimates the position of the plane or the curved surface of the surrounding object based on the BIM information, and based on the constraint condition that a plurality of points existing in the surroundings are located on the plane or the curved surface.
  • the distance from the image pickup device 17 to the surrounding objects is calculated.
  • the points 50b to 50d shown in FIG. 4 all exist on the plane 901.
  • the bundle adjustment unit 104 imposes a constraint condition by the equation of a plane when executing the bundle adjustment process based on the captured image with the plane 901 as the imaging range.
  • points 50a to 50d in the three-dimensional space are not particularly limited, they are simply referred to as points 50.
  • the bundle adjustment unit 104 solves the optimization problem by the nonlinear least squares method by the nonlinear functions f (x) and g (x), thereby presenting a plane.
  • the position of the point 50 and the position of the imaging device 17 are estimated with the above as a constraint condition.
  • the function f (x) corresponds to the above equation (4).
  • the penalty method or the extended Lagrange method can be applied, but other solutions may be adopted.
  • the structure around the information processing device 1 may have a curved surface as well as a flat surface.
  • the outer surface of the pillar 90b is a curved surface.
  • the bundle adjustment unit 104 may impose a constraint condition by a curved surface equation so that a point on a three-dimensional space is on a curved surface based on BIM information.
  • the bundle adjustment unit 104 generates a point cloud having the three-dimensional coordinates as map information based on the three-dimensional coordinates of the plurality of points 50 after the bundle adjustment. Further, the bundle adjustment unit 104 updates the map information by adding or deleting a new point 50 to the map information. In addition, the bundle adjustment unit 104 may adjust the self-position estimation result and the map information in consideration of the detection result of the IMU sensor 18.
  • the bundle adjustment unit 104 calculates the positions of surrounding objects as the spatial coordinates of the plurality of points 50 in the three-dimensional space, and outputs the calculated spatial coordinates of the plurality of points 50 as map information. ..
  • the term "output" includes storage in the auxiliary storage device 14 or transmission to the external device 2.
  • the bundle adjustment unit 104 stores the estimated self-position and the generated map information in the auxiliary storage device 14. Further, the bundle adjustment unit 104 may transmit the estimated self-position and the generated map information to the external device 2.
  • the movement control unit 105 moves the information processing device 1 by controlling the movement device 16. For example, the movement control unit 105 searches for a movable route based on the map information stored in the auxiliary storage device 14 and the current self-position. The movement control unit 105 controls the movement device 16 based on the search result.
  • the movement control unit 105 moves to avoid obstacles based on the detection results of obstacles or the like by these sensors. A route may be generated.
  • the movement control method of the information processing device 1 is not limited to these, and various autonomous movement methods can be applied.
  • FIG. 6 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the first embodiment.
  • the acquisition unit 101 acquires BIM information from the external device 2 (S1).
  • the acquisition unit 101 stores the acquired BIM information in the auxiliary storage device 14.
  • the movement control unit 105 starts the movement of the information processing device 1 by controlling the movement device 16 (S2).
  • the acquisition unit 101 acquires an captured image from the imaging device 17.
  • the acquisition unit 101 acquires sensing results such as angular velocity and acceleration from the IMU sensor 18 (S3).
  • the tracking unit 103 identifies the current position and orientation of the image pickup device 17 based on the captured image (S4).
  • the conversion unit 102 generates an initial value of the three-dimensional coordinates of the point in the structure from the BIM information based on the current position and orientation of the image pickup device 17 specified by the tracking unit 103 (S5).
  • the bundle adjustment unit 104 executes the bundle adjustment process (S6). Specifically, the initial values of the three-dimensional coordinates of the points in the structure around the image pickup device 17 generated from the BIM information, the position and orientation of the image pickup device 17 specified by the tracking unit 103, and the captured image. Based on the above, the distance from the image pickup device 17 to the surrounding object is calculated, and the position and orientation of the image pickup device 17 and the three-dimensional coordinates of the surrounding object are estimated. In addition, the bundle adjustment unit 104 generates map information based on the estimated three-dimensional coordinates of surrounding objects.
  • the bundle adjustment unit 104 stores the estimated self-position and the generated map information in, for example, the auxiliary storage device 14.
  • the movement control unit 105 searches for a movement route based on the map information stored in the auxiliary storage device 14 and the current self-position, and controls the movement device 16 based on the search result to obtain an information processing device. Move 1
  • the movement control unit 105 determines whether or not to end the movement of the information processing device 1 (S7).
  • the movement control unit 105 determines, for example, that when the information processing device 1 arrives at a predetermined end point, the movement control unit 105 ends the movement of the information processing device 1.
  • the conditions for determining the end of movement are not particularly limited. For example, when the movement control unit 105 receives an instruction to end movement from the outside via the communication network 3, the information processing device 1 It may be determined that the movement of is completed.
  • the information processing device 1 of the present embodiment executes self-position estimation and map information generation based on the BIM information and the captured image captured around the information processing device 1. Therefore, according to the information processing apparatus 1 of the present embodiment, by using the BIM information for the processing of self-position estimation and map information generation, it is possible to improve the self-position estimation and the accuracy of the map information. ..
  • the information processing apparatus 1 of the present embodiment converts BIM information into at least one input value of self-position estimation processing by SLAM processing unit 120 or map information generation processing, and based on the input value, By executing self-position estimation and map information generation, self-position estimation and map information accuracy are higher than self-position estimation and map information generation based only on peripheral detection results such as captured images. Can be improved.
  • the information processing device 1 of the present embodiment identifies the position and orientation of the image pickup device 17 by tracking a plurality of captured images captured at different times, and from the image pickup device 17 based on the BIM information to a surrounding object.
  • the information processing device 1 of the present embodiment uses the distance from the image pickup device 17 based on the BIM information to the surrounding object as the initial value or the range of the initial value in the bundle adjustment process.
  • the distance from the image pickup device 17 based on the BIM information to the surrounding object is the distance from the image pickup device 17 to the surrounding object in three-dimensional coordinates.
  • the initial value of the three-dimensional coordinates of a point in the three-dimensional space may be assumed to be infinite.
  • the bundle adjustment process starts without specifying whether the distance between the point in the three-dimensional space and the image pickup device is 1 m or 1000 m, so that the amount of calculation until the calculation result converges. May increase.
  • the information processing apparatus 1 of the present embodiment uses the initial value based on the BIM information, the processing result can be converged with a small amount of calculation.
  • the information processing apparatus 1 of the present embodiment estimates the position of the plane or the curved surface of the surrounding object based on the BIM information, and the constraint condition that a plurality of points existing in the surroundings are located on the plane or the curved surface.
  • the distance from the image pickup apparatus 17 to the surrounding objects is calculated based on the above. Therefore, according to the information processing apparatus 1 of the present embodiment, the amount of calculation can be reduced as compared with the case where the positions of a plurality of points existing on the same plane or curved surface are separately obtained.
  • the information processing device 1 of the present embodiment calculates the positions of surrounding objects as the spatial coordinates of the plurality of points 50 in the three-dimensional space, and outputs the calculated spatial coordinates of the plurality of points 50 as map information. do. According to the information processing device 1 of the present embodiment, more accurate map information can be provided by outputting the positions of surrounding objects calculated by the bundle adjustment process using BIM information as map information.
  • the information processing device 1 may be a robot or the like having functions such as monitoring, security, cleaning, and delivery of luggage. In this case, the information processing device 1 realizes various functions by moving the building 9 based on the estimated self-position and map information. Further, the map information generated by the information processing device 1 may be used not only for generating the movement route of the information processing device 1 itself, but also for monitoring or managing the building 9 from a remote location. Further, the map information generated by the information processing device 1 may be used to generate a movement route of a robot or drone other than the information processing device 1.
  • the image pickup device 17 is not limited to the stereo camera.
  • the image pickup device 17 may be an RGB-D camera having an RGB (Red Green Blue) camera and a three-dimensional measurement camera (Dept camera), a monocular camera, or the like.
  • the sensor included in the information processing device 1 is not limited to the IMU sensor 18, and a gyro sensor, an acceleration sensor, a magnetic sensor, or the like may be individually provided.
  • the SLAM processing unit 120 executes the image SLAM (Visual SLAM) using the captured image, but the SLAM that does not use the captured image may be adopted.
  • the information processing device 1 may detect surrounding structures by Lidar (Light Detection and Ringing or Laser Imaging Detection and Ringing) or the like instead of the image pickup device 17.
  • the SLAM processing unit 120 may specify the position and orientation of the information processing device 1 based on the distance measurement result by Lidar.
  • the SLAM processing unit 120 is supposed to generate three-dimensional map information, but it may be possible to generate two-dimensional map information.
  • the equations (1) to (6) illustrated in the present embodiment are examples, and the mathematical expressions used in the tracking process or the bundle adjustment process are not limited to these.
  • the bundle adjustment unit 104 may perform bundle adjustment according to the equation (4) without imposing the constraint conditions according to the equations (5) and (6). Further, in the tracking process, without using the proximity pattern N p, it may perform the tracking process.
  • the SLAM processing unit 120 may estimate its own position and generate map information by a method other than tracking processing or bundle adjustment processing.
  • the tracking unit 103 is used as an example of the specific unit, but a method of specifying a change in the position and posture of the information processing device 1 by a method other than tracking may be adopted.
  • various processes for improving the accuracy of self-position estimation or map information may be added to the SLAM process.
  • the SLAM processing unit 120 may further execute a loop closing process or the like.
  • a part or all of the information processing device 1 in the above-described embodiment may be composed of hardware, or may be composed of information processing of software (program) executed by a CPU, GPU, or the like. good.
  • software that realizes at least a part of the functions of each device in the above-described embodiment is a flexible disk, a CD-ROM (Compact Disc-Read Only Memory), a USB memory, or the like.
  • the software may process information by storing it in a non-temporary storage medium (non-temporary computer-readable medium) and loading it into a computer.
  • the software may be downloaded via a communication network.
  • information processing may be executed by hardware by implementing the software in a circuit such as an ASIC or FPGA.
  • the type of storage medium that stores the software is not limited.
  • the storage medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk or a memory. Further, the storage medium may be provided inside the computer or may be provided outside the computer.
  • the information processing device 1 includes one component, but may include a plurality of the same components.
  • software is installed on a plurality of computers, and each of the plurality of computers executes the same or different part of the processing of the software. You may. In this case, it may be a form of distributed computing in which each computer communicates via a network interface 13 or the like to execute processing. That is, the information processing device 1 in the above-described embodiment may be configured as a system that realizes a function by executing instructions stored in one or a plurality of storage devices by one or a plurality of computers. Further, the information transmitted from the terminal may be processed by one or a plurality of computers provided on the cloud, and the processing result may be transmitted to the terminal.
  • Various operations of the information processing device 1 in the above-described embodiment may be executed in parallel processing by using one or a plurality of processors or by using a plurality of computers via a network. Further, various operations may be distributed to a plurality of arithmetic cores in the processor and executed in parallel processing. In addition, some or all of the processes, means, etc. of the present disclosure may be executed by at least one of a processor and a storage device provided on the cloud capable of communicating with the information processing device 1 via the network. As described above, each device in the above-described embodiment may be in the form of parallel computing by one or a plurality of computers.
  • the information processing device 1 in the above-described embodiment may be realized by one or a plurality of processors 11.
  • the processor 11 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. You may point. When a plurality of electronic circuits are used, each electronic circuit may communicate by wire or wirelessly.
  • processors may be connected (combined) to one storage device (memory), or a single processor may be connected.
  • a plurality of storage devices (memory) may be connected (combined) to one processor.
  • the information processing device 1 in the above-described embodiment is composed of at least one storage device (memory) and a plurality of processors connected (combined) to the at least one storage device (memory), among the plurality of processors
  • At least one processor may include a configuration in which it is connected (combined) to at least one storage device (memory). Further, this configuration may be realized by a storage device (memory) and a processor included in a plurality of computers. Further, a configuration in which the storage device (memory) is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.
  • the external device 2 is not limited to the server device. Further, the external device 2 may be provided in a cloud environment. Further, the external device 2 may be used as an example of the information processing device in the claims.
  • the external device 2 may be an input device.
  • the device interface 15 may be connected not only to the mobile device 16, the image pickup device 17, and the IMU sensor 18, but also to the input device.
  • the input device is, for example, a device such as a camera, a microphone, a motion capture, various sensors, a keyboard, a mouse, or a touch panel, and gives the acquired information to the information processing device 1.
  • it may be a device including an input unit, a memory and a processor such as a personal computer, a tablet terminal, or a smartphone.
  • the external device 2 may be an output device.
  • the device interface 15 may be connected to the output device.
  • the output device may be, for example, a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, and outputs audio or the like. It may be a speaker or the like. Further, it may be a device including an output unit such as a personal computer, a tablet terminal, or a smartphone, a memory, and a processor.
  • the external device 2 may be a storage device (memory). Further, the device interface 15 may be connected to a storage device (memory).
  • the external device 2 may be a network storage or the like, and a storage such as an HDD may be connected to the device interface 15.
  • the external device 2 or the external device connected to the device interface 15 may be a device having some functions of the components of the information processing device 1 in the above-described embodiment. That is, the information processing device 1 may transmit or receive a part or all of the processing results of the external device 2 or the external device connected to the device interface 15.
  • the information processing device 1 may be constantly connected to the external device 2 via the communication network 3, but is limited to this. is not it.
  • the information processing device 1 may take the connection with the external device 2 offline while executing the self-position estimation process and the map information generation process.
  • the world coordinates of the points included in the building 9 specified by the conversion unit 102 from the BIM information are used as an example of the information regarding the positions of the surrounding objects, but the information regarding the positions of the surrounding objects is included in this. It is not limited.
  • the conversion unit 102 may generate information indicating the distance between the information processing device 1 and a surrounding object based on the BIM information and the position and orientation of the image pickup device 17.
  • the position and orientation of the image pickup apparatus 17 for example, the position and orientation specified by the tracking unit 103 can be adopted.
  • position of the imaging device 17 (R j, t j), the distance between the (R j + 1, t j + 1) is, the conversion unit 102 It is identified by information that indicates the distance taken.
  • the bundle adjustment unit 104 determines the position (R j , t j ) of the point X i imaging device 17 in the three-dimensional space specified by the information indicating the distance generated by the conversion unit 102. Set a value for each parameter so that it matches the distance between (R j + 1 , t j + 1). Also in this method, the position of a point X i and the imaging device 17 in a three-dimensional space obtained result of the adjustment of the error by bundle adjustment may be different from the result obtained from the BIM data.
  • the conversion unit 102 may specify the dimension of the building 9 from the BIM information, and the dimension may be used as an example of information regarding the position of a surrounding object.
  • the bundle adjustment unit 104 can reduce the amount of calculation by performing bundle adjustment using the information regarding the position of the surrounding object converted from the BIM information by the conversion unit 102.
  • the environmental information is three-dimensional design information such as BIM information or 3D CAD data.
  • the environmental information includes at least one of the entry / exit information of the person in the building 9 and the image recognition result of the person in the captured image.
  • the environmental information may include both the entry / exit information of the person in the building 9 and the image recognition result of the person in the captured image, or may include either one.
  • FIG. 7 is a block diagram showing an example of the functions included in the information processing device 1 according to the second embodiment.
  • the information processing apparatus 1 of the present embodiment includes an acquisition unit 1101, a conversion unit 1102, a SLAM processing unit 1120, and a movement control unit 105.
  • the SLAM processing unit 1120 includes a tracking unit 1103 and a bundle adjusting unit 1104.
  • the conversion unit 1102 includes an initial value generation unit 106 and a mask information generation unit 107.
  • the movement control unit 105 has the same function as that of the first embodiment.
  • the acquisition unit 1101 of the present embodiment has the same function as that of the first embodiment, and acquires the entry / exit information of the person in the building 9.
  • the entry / exit information is information indicating the number of people entering / exiting each room or floor of the building 9 and the time of entering / exiting.
  • a sensor for detecting entry / exit is installed at the entrance / exit of a room or floor of the building 9, and the detection result by the sensor is transmitted to the external device 2.
  • the acquisition unit 1101 acquires the entry / exit information from the external device 2.
  • the method of detecting the entry / exit of a person is not limited to the sensor.
  • the entry / exit information may be a reading record of a security card by a card reader or a detection result of a person from an image captured by a surveillance camera installed in a building 9.
  • the acquisition unit 1101 stores the acquired entry / exit information in the auxiliary storage device 14.
  • the conversion unit 1102 of the present embodiment has the same functions as those of the first embodiment, and is excluded from the target of map information generation in the building 9 where the information processing device 1 is located based on the environmental information. Generates mask information that represents.
  • the environmental information includes at least one of the entry / exit information and the image recognition result of the person.
  • the image recognition result of a person is a result of recognizing a person by image processing from the captured image captured by the image pickup device 17.
  • the environmental information of the present embodiment includes both the entry / exit information and the image recognition result, and the same three-dimensional design information as that of the first embodiment.
  • the conversion unit 1102 includes an initial value generation unit 106 and a mask information generation unit 107.
  • the initial value generation unit 106 has the same function as the conversion unit 102 in the first embodiment.
  • the mask information is information representing an area excluded from the target of map information generation.
  • the mask information generation unit 107 determines the area where the person is located in the building 9 based on the entry / exit information or the image recognition result of the person, and sets the determined area as an area to be excluded from the target of map information generation. ..
  • the mask information generation unit 107 recognizes a person by image processing from the captured image captured by the imaging device 17. Further, when it is difficult to determine whether or not the object depicted in the captured image is a person in the image processing, the mask information generation unit 107 may move to the room or floor where the captured image is captured based on the entry / exit information. , It is determined whether or not a person exists at the time when the captured image is captured. When the mask information generation unit 107 determines that a person exists in the room or floor where the captured image is captured at the time when the captured image is captured, the mask information generation unit 107 draws the captured image on the captured image rather than determining that the person does not exist. It is highly probable that the object is a person.
  • FIG. 8 is an image diagram showing an example of the positional relationship between the information processing device 1 and surrounding objects according to the second embodiment.
  • the person 70 in the building 9, the person 70 exists in the room where the information processing device 1 exists. Unlike the pillars 90a to 90c, the person 70 moves, so if the presence of the person 70 is included in the map information, the accuracy of the map information may decrease.
  • the mask information generation unit 107 generates mask information representing the area 80 in which the person 70 exists.
  • the mask information represents, for example, the area 80 in which the person 70 exists in three-dimensional coordinates.
  • the mask information generation unit 107 generates mask information using both the entry / exit information and the image recognition result of the person, but the mask information is generated based on only one of them. You may.
  • the mask information generation unit 107 detects a moving body such as a vehicle or an object such as a device temporarily existing in the building 9 such as a dolly from the image captured by the image pickup device 17 by image recognition. You may. In this case, the mask information generation unit 107 sets the area where it is determined that these objects exist as an area to be excluded from the target of map information generation.
  • the mask information generation unit 107 generates mask information representing an area to be excluded from the target of map information generation, and sends it to the SLAM processing unit 1120.
  • the SLAM processing unit 1120 of the present embodiment has the functions of the first embodiment and does not generate map information for the area corresponding to the mask information.
  • the tracking section 1103 of the SLAM processor 1120 of the present embodiment when performing the tracking process by the same equation (1) in the first embodiment, near the pattern N p is represented by the mask information If the area that is the image area to be rendered, with respect to the vicinity pattern N p, is multiplied by a mask value.
  • the mask value is, for example, “0” or “1”, but is not limited thereto.
  • the method of applying the mask is not limited to this, and other methods may be adopted.
  • the bundle adjustment unit 1104 of the SLAM processing unit 1120 of the present embodiment has the functions of the first embodiment, and the area corresponding to the mask information is excluded from the bundle adjustment.
  • FIG. 9 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the second embodiment.
  • the process of acquiring the BIM information in S1 is the same as that of the first embodiment described with reference to FIG.
  • the acquisition unit 1101 acquires the entry / exit information acquisition (S21).
  • the acquisition unit 1101 stores the acquired entry / exit information in the auxiliary storage device 14.
  • the process of acquiring the captured image of S3 and the sensing results such as angular velocity and acceleration from the process of starting the movement of the information processing device 1 of S2 is the same as that of the first embodiment.
  • the mask information generation unit 107 of the conversion unit 1102 of the present embodiment generates mask information representing an area to be excluded from the target of map information generation based on the entry / exit information or the image recognition result of the person 70 ( S22).
  • the tracking unit 1103 of the SLAM processing unit 1120 of the present embodiment identifies the current position and orientation of the image pickup device 17 based on the captured image (S4). At this time, the tracking unit 1103 excludes the area corresponding to the mask information from the tracking process.
  • the initial value generation unit 106 of the conversion unit 1102 of the present embodiment is based on the current position and orientation of the image pickup device 17 specified by the tracking unit 103, and is based on the BIM information in the structure around the image pickup device 17.
  • the initial value of the three-dimensional coordinates of the point is generated (S5). Since the area corresponding to the mask information is not subject to the generation of map information, the initial value generation unit 106 does not generate the initial value of the three-dimensional coordinates of the points in the structure in the area corresponding to the mask information. ..
  • the bundle adjustment unit 1104 executes the bundle adjustment process (S6).
  • the bundle adjustment unit 1104 excludes the area corresponding to the mask information from the bundle adjustment.
  • the process of determining whether or not to end the movement of the information processing device 1 in S7 is the same as that of the first embodiment.
  • the acquisition unit 1101 acquires the latest entry / exit information acquisition again (S23), and returns to the process of S3.
  • the information processing device 1 of the present embodiment generates mask information representing the area excluded from the target of map information generation in the building 9 based on the environmental information, and the area corresponding to the mask information is the area corresponding to the mask information. Does not generate map information. Therefore, according to the information processing device 1 of the present embodiment, it is possible to exclude things such as a person 70 and an object that temporarily exist, which may reduce the accuracy of the map information, so that the accuracy of the map information can be reduced. Can be improved.
  • the environmental information includes the entry / exit information of the person 70 in the building 9, or the image recognition result of the person 70 in the image captured by the image pickup device 17 mounted on the information processing device 1.
  • the information processing device 1 of the present embodiment determines an area where the person 70 is located in the building 9 based on the entry / exit information or the image recognition result of the person 70, and determines the determined area from the target of generating map information. The area to be excluded.
  • the information processing device 1 can improve the accuracy of the map information by not reflecting the worker in the map information. Further, with such a configuration, the information processing apparatus 1 of the present embodiment can robustly execute processing even when the surrounding environment changes depending on a person or the like.
  • the bundle adjustment process is performed based on the environmental information as in the first embodiment, but the information processing device 1 of the second embodiment is the same as that of the first embodiment. It does not have to have all the functions.
  • the information processing device 1 may use the environmental information only for generating the mask information and may not use it for the bundle adjustment process.
  • the environmental information does not have to include the three-dimensional design information.
  • the timing of using the mask information is limited to this. is not it.
  • mask information based on entry / exit information at a past time may be used.
  • FIG. 10 is a block diagram showing an example of the functions included in the information processing device 1 according to the third embodiment.
  • the information processing apparatus 1 of the present embodiment includes an acquisition unit 1101, a marker detection unit 108, a calibration unit 109, a conversion unit 2102, a SLAM processing unit 2120, and a movement control unit 105.
  • the SLAM processing unit 2120 includes a tracking unit 2103 and a bundle adjustment unit 1104.
  • the conversion unit 2102 includes an initial value generation unit 1106 and a mask information generation unit 1107.
  • the movement control unit 105 has the same functions as those of the first and second embodiments.
  • the acquisition unit 1101 has the same function as that of the second embodiment.
  • the marker detection unit 108 detects the AR marker from the captured image.
  • the AR marker has, for example, information on three-dimensional coordinates representing the position where the AR marker is described.
  • the three-dimensional coordinates are consistent with the coordinate system in the BIM information.
  • the AR marker is a coordinate system in BIM information and represents a position where the AR marker is installed.
  • the AR marker is an example of index information in this embodiment. It is assumed that the AR marker is installed on a wall, a pillar, or the like along the passage of the building 9. Specifically, the AR marker is, for example, a QR code (registered trademark) or the like, but is not limited thereto. The number of AR markers is not particularly limited, but it is assumed that a plurality of AR markers are installed per building 9. Further, the marker detection unit 108 is an example of the index detection unit in the present embodiment. The marker detection unit 108 sends the detection result of the AR marker to the calibration unit 109.
  • the calibration unit 109 Based on the detection result of the AR marker, the calibration unit 109 adjusts the coordinate system representing the self-position held internally so as to match the coordinate system of the BIM information.
  • the calibration unit 109 is an example of the coordinate adjustment unit in this embodiment.
  • the locus of change in self-position estimated by the SLAM processing unit 2120 is stored in the auxiliary storage device 14, but an error in self-position may be accumulated as the information processing device 1 moves.
  • the calibration unit 109 eliminates the accumulation of such errors by adjusting the current position of the information processing device 1 based on the three-dimensional coordinates represented by the AR marker detected by the marker detection unit 108.
  • the calibration unit 109 sends the calibration result to the conversion unit 2102.
  • the calibration unit 109 sends a conversion matrix for correcting the self-position to the conversion unit 2102.
  • the conversion unit 2102 of the present embodiment has the same functions as those of the first and second embodiments, and also provides environmental information to the SLAM processing unit 2120 based on the self-position adjusted by the calibration unit 109. Converts to the input value of the self-position estimation process or the map information generation process.
  • the initial value generation unit 1106 has the same function as that of the second embodiment, and is an imaging device specified by the tracking unit 2103 based on the self-position adjusted by the calibration unit 109.
  • the initial value generation unit 1106 specified the three-dimensional coordinates indicating the position of the information processing device 1 in the building 9 on the three-dimensional model of the building 9 in the BIM information by the transformation matrix generated by the calibration unit 109.
  • the initial value of the bundle adjustment process or the input value representing the range of the initial value is generated.
  • the mask information generation unit 1107 has the same function as that of the second embodiment, and generates mask information based on the self-position adjusted by the calibration unit 109.
  • the mask information generation unit 1107 identifies the three-dimensional coordinates indicating the position of the information processing device 1 in the building 9 on the three-dimensional model of the building 9 in the BIM information by the conversion matrix generated by the calibration unit 109. Above, generate mask information.
  • the bundle adjustment unit 1104 of the SLAM processing unit 2120 has the same functions as those of the first and second embodiments, and the bundle adjustment unit 2102 generates the bundle adjustment based on the self-position adjusted by the calibration unit 109. Use the initial value of processing or the range of initial values for bundle adjustment.
  • FIG. 11 is a flowchart showing an example of the flow of self-position estimation and map information generation processing according to the third embodiment.
  • the process from the process of acquiring the BIM information of S1 to the process of acquiring the captured image and the sensing result of S3 is the same as that of the second embodiment.
  • the marker detection unit 108 detects the AR marker from the captured image (S31).
  • the marker detection unit 108 sends the detection result of the AR marker to the calibration unit 109.
  • the calibration unit 109 executes the calibration process based on the detection result of the AR marker (S32). For example, the calibration unit 109 generates a transformation matrix for adjusting the self-position of the BIM information in the coordinate system. The calibration unit 109 sends the generated transformation matrix to the conversion unit 2102.
  • the mask information generation unit 1107 identifies the three-dimensional coordinates indicating the position of the information processing device 1 in the building 9 on the three-dimensional model of the building 9 in the BIM information by the conversion matrix generated by the calibration unit 109. , Generates mask information (S22).
  • the tracking process of S4 is the same as that of the first and second embodiments, but the calibration result by the calibration unit 109 may also be used in the process.
  • the calibration unit 109 sends the calibration result to the conversion unit 2102, but may further send the calibration result to the SLAM processing unit 2120.
  • the tracking unit 2103 of the SLAM processing unit 2120 executes the tracking process using the three-dimensional coordinates based on the calibration result.
  • the initial value generation unit 1106 specifies the three-dimensional coordinates indicating the position of the information processing device 1 in the building 9 on the three-dimensional model of the building 9 in the BIM information by the transformation matrix generated by the calibration unit 109.
  • the initial value of the bundle adjustment process or the input value representing the range of the initial value is generated (S5).
  • the bundle adjustment unit 1104 may execute the tracking process and the bundle adjustment process using the three-dimensional coordinates based on the calibration result.
  • the information processing apparatus 1 of the present embodiment detects the index information whose position is represented by the coordinate system in the BIM information from the detection result of the captured image or the like, and is based on the coordinate system adjusted by the index information. Then, the environmental information is converted into an input value of the self-position estimation process by the SLAM processing unit 2120 or the map information generation process. Therefore, according to the information processing device 1 of the present embodiment, the error between the BIM information and the internal SLAM coordinate system of the information processing device 1 is reduced, and self-position estimation and map information generation can be performed with higher accuracy. can do.
  • the AR marker is illustrated as the index information, but the index information is not limited to this.
  • the index information may be a sign or the like that can be captured by Lidar or various sensors, or may be a beacon or the like.
  • the information processing device 1 is described as having the functions of both the first embodiment and the second embodiment, but the information processing device 1 of the present embodiment is described as having the functions of both the first embodiment and the second embodiment. It is not necessary to have all the functions of the second embodiment.
  • the information processing apparatus 1 may use the environmental information only for either bundle adjustment or generation of mask information. Further, the environmental information may include any of three-dimensional design information, entry / exit information, and image recognition result of a person.
  • the information processing apparatus 1 uses the captured image for recognizing a person, but the use of the captured image is not limited to this.
  • the information processing apparatus 1 segmentes the captured image based on the recognition result of the object drawn on the captured image, and performs SLAM processing based on the segmentation result.
  • the information processing device 1 of the present embodiment includes an acquisition unit 101, a conversion unit 102, a SLAM processing unit 120, and a movement control unit 105.
  • the acquisition unit 101 has the same function as that of the first embodiment. Specifically, the acquisition unit 101 acquires an captured image from the imaging device 17 via the device interface 15.
  • the conversion unit 102 has the same function as that of the first embodiment, and then segments the captured image acquired by the acquisition unit 101 based on the recognition result of the object drawn on the captured image.
  • FIG. 12 is a diagram showing an example of segmentation of the captured image 60 according to the fourth embodiment.
  • the captured image 60 depicts the environment around the information processing device 1.
  • the conversion unit 102 recognizes the image area in which the object is drawn and the type of each object from the captured image 60.
  • the recognition result of the object is information in which the two-dimensional coordinates of the image area in which the object is drawn and the type of each object are associated with each other.
  • the environmental information includes at least the captured image 60.
  • the segmentation result of the captured image 60 may be used as an example of the environmental information instead of the captured image 60 itself.
  • the conversion unit 102 recognizes the object depicted in the captured image 60.
  • object includes structures such as walls and pillars, furniture, furniture, moving objects, temporary objects, people, and the like.
  • the conversion unit 102 recognizes the individual objects drawn on the captured image 60 by inputting the captured image 60 into the trained model configured by, for example, a neural network or the like.
  • the captured image 60 depicts a person 70, boxes 75a and 75b, a pillar 90, a wall 91, and a floor 92.
  • the conversion unit 102 recognizes these objects.
  • "People,” “boxes,” “pillars,” “walls,” and “floors” are examples of object types.
  • the recognition of the person 70 and the recognition of other objects may be executed separately.
  • the conversion unit 102 segmentes the captured image 60 based on the recognition result of the object. On the right side of FIG. 12, the segmentation result 61 of the captured image 60 is shown. In the example shown in FIG. 12, the conversion unit 102 sets the image area where the pillar 90 and the wall 91 are drawn as the area A1, the image area where the floor 92 is drawn, the area A2, and the image area where the boxes 75a and 75b are drawn. The image region in which the area A3 and the person 70 are drawn is set as the area A4, and the captured image 60 is segmented into four.
  • the division unit is not limited to the example shown in FIG. Hereinafter, when the regions A1 to A4 are not particularly distinguished, they are simply referred to as regions A.
  • the conversion unit 102 classifies the recognized objects, that is, the person 70, the boxes 75a and 75b, the pillar 90, the wall 91, and the floor 92 according to whether or not they are permanent objects.
  • the pillar 90, the wall 91, and the floor 92 are permanent objects because they are part of the building 9.
  • the person 70 and the boxes 75a and 75b are non-permanent objects. Whether or not each object is permanently installed is determined by, for example, a trained model.
  • a permanently installed object is an object that does not move from the installation position once it is installed.
  • an object that is a part of the building 9, such as the pillar 90, the wall 91, and the floor 92, is basically a permanent object because it does not move.
  • An object that is not permanently installed is an object that is likely to move from the installation position.
  • a person 70, a moving body such as a cart or a forklift, temporarily installed fixtures, luggage boxes 75a, 75b, etc. are non-permanent objects.
  • the conversion unit 102 associates the segmented areas A1 to A4 with whether or not the object drawn in each area is a permanently installed object.
  • the information in which the segmented areas A1 to A4 are associated with whether or not the object drawn in each area is a permanently installed object is referred to as a segmentation result.
  • the conversion unit 102 sends the segmentation result to the SLAM processing unit 120.
  • the object recognition and the segmentation process based on the result of the object recognition have been described separately, but these processes may be integrated.
  • a trained model that outputs the segmentation result of the captured image 60 when the captured image 60 is input may be adopted.
  • the conversion unit 102 inputs the captured image 60 into the trained model and obtains the segmentation result output from the trained model.
  • the method of object recognition from the captured image 60 and the segmentation of the captured image 60 is not limited to the above example.
  • the conversion unit 102 may apply a machine learning or deep learning technique other than the neural network to perform object recognition from the captured image 60 and segmentation of the captured image 60.
  • the SLAM processing unit 120 of the present embodiment has the functions of the first embodiment, and then estimates the self-position and generates map information based on the segmentation result of the captured image 60.
  • the SLAM processing unit 120 identifies a three-dimensional space corresponding to the areas A1 and A2 in which the permanent object is drawn in the captured image 60, and uses the three-dimensional space for self-position estimation processing and map information.
  • Target of generation processing Further, the SLAM processing unit 120 identifies a three-dimensional space corresponding to the areas A3 and A4 in which a non-permanent object is drawn in the captured image 60, and estimates the self-position and map information in the three-dimensional space. Exclude from the target of the generation process of. In this case, the information representing the areas A3 and A4 in which the non-permanent object is drawn may be used as the mask information representing the area to be excluded from the target of generating the map information.
  • the SLAM processing unit 120 uses the regions A1 and A2 in which the permanent object is drawn in the captured image 60 for SLAM processing, and the region A3 in which the non-permanent object is drawn in the captured image 60. , A4 may not be used for SLAM processing.
  • the weighting at the time of SLAM processing may be changed depending on whether or not the objects drawn in each of the areas A1 to A4 are permanent objects.
  • the weighting coefficient of the regions A1 and A2 in which the permanent object is drawn is larger than the weighting coefficient of the regions A3 and A4 in which the non-permanent object is drawn in the captured image 60.
  • the weighting coefficients are set in each of the regions A1 to A4 so as to be.
  • the SLAM processing unit 120 may change the weighting coefficient for each type of object in the area A in which the non-permanent object is drawn. For example, there are an object that is relatively likely to be installed at the same position for a long period of time and an object that is relatively unlikely to be installed even if the object is not permanently installed.
  • the SLAM processing unit 120 is classified as a non-permanent object because, for example, large furniture and furniture may move, but it stays in the same position for a long period of time as compared with the person 70 and the like. It is likely to be installed. Therefore, the SLAM processing unit 120 sets the weighting coefficient so that the weighting coefficient becomes larger as the area A in which the object having a low possibility of movement is drawn out of the area A in which the non-permanent object is drawn. May be.
  • the weighting coefficient for each area A1 to A4 may be set by the conversion unit 102 instead of the SLAM processing unit 120.
  • the information processing device 1 of the present embodiment segments the captured image 60 captured by the imaging device 17 based on the recognition result of the object drawn on the captured image 60, and based on the result of the segmentation. , Performs self-position estimation and map information generation. Therefore, according to the information processing apparatus 1 of the present embodiment, in addition to the effect of the first embodiment, whether it is used for estimating the self-position and generating map information according to the object drawn on the captured image 60. Whether or not, or the strength of the influence on the estimation of the self-position and the generation of the map information can be adjusted, so that the accuracy of the estimation of the self-position and the accuracy of the map information can be improved.
  • the SLAM processing unit 120 described above may execute self-position estimation and map information generation based on the segmentation result of the captured image 60 and the three-dimensional design information such as BIM information.
  • the SLAM processing unit 120 refers to the three-dimensional design information, and the object is included in the design of the building 9. Judge whether or not. If the object determined not to be a permanent object based on the captured image 60 is not registered in the three-dimensional design information, the SLAM processing unit 120 adopts the determination result that the object is not a permanent object as it is. Further, when the object determined to be not a permanent object based on the captured image 60 is registered in the three-dimensional design information, the SLAM processing unit 120 determines that the object is not a permanent object. Is changed to the judgment result that is a permanent object.
  • the conversion unit 102 may determine the high accuracy of determining whether or not the object depicted in the captured image 60 is a permanent object by a percentage or the like.
  • the SLAM processing unit 120 refers to the three-dimensional design information when the accuracy of determining whether or not the object depicted in the captured image 60 is a permanent object is equal to or less than the reference value, and the object is the building 9 It may be determined whether or not it is included in the design of.
  • the reference value of the determination accuracy is not particularly limited.
  • the process of comparing the three-dimensional design information with the image recognition result may be executed by the conversion unit 102 instead of the SLAM processing unit 120.
  • the accuracy of the determination result as to whether or not the object drawn in each region A is a permanent object can be determined. Can be improved.
  • the SLAM processing unit 120 is either one of the segmentation result of the captured image 60 in the present embodiment, the entry / exit information of the person in the second embodiment described above, or the image recognition result of the person 70 in the captured image. Both may be used in combination.
  • Modification example 1 In the first embodiment described above, the initial value of the three-dimensional coordinates of the points in the surrounding object in the SLAM process is obtained based on the three-dimensional design information such as BIM information. In this modification, the three-dimensional coordinates of the points calculated from the distance values estimated from the captured image 60 captured around the information processing device 1 are used as the initial values of the three-dimensional coordinates of the points in the surrounding objects in the SLAM process. Adopt as.
  • the conversion unit 102 of the information processing device 1 estimates the distance (depth) between the object drawn on the captured image 60 and the imaging device 17 based on the captured image 60.
  • the estimation process is called a depth estimation process.
  • the environmental information includes at least the captured image 60.
  • the distance information estimated from the captured image 60 may be used as an example of the environmental information instead of the captured image 60 itself.
  • the conversion unit 102 is based on the stereoscopic difference of the captured image captured by one camera included in the stereo camera. , Calculate the depth.
  • the imaging device 17 may be a monocular camera.
  • the conversion unit 102 executes the depth estimation process by using the technique of machine learning or deep learning. For example, when the conversion unit 102 inputs the captured image 60 captured by the monocular camera, the conversion unit 102 outputs the depth map corresponding to the captured image 60. The distance between them may be estimated.
  • the trained model in this modification is, for example, a model that estimates the depth from a monocular image by estimating an image paired with the monocular image from the monocular image using a stereo image as training data.
  • the method of estimating the depth from a monocular image is not limited to this.
  • the conversion unit 102 sends the distance estimated from the captured image 60 to the SLAM processing unit 120 as an input value for SLAM processing. More specifically, the conversion unit 102 sets the three-dimensional coordinates of the points estimated from the distance estimated from the captured image 60 as the initial values in the bundle adjustment process by the bundle adjustment unit 104. The conversion unit 102 may specify the range of the initial value without specifying the initial value as a unique value.
  • the conversion unit 102 or the SLAM processing unit 120 processes the distance estimated from the captured image 60 with the image pickup device 17. The correction is made based on the deviation of the position from the center of the device 1.
  • the bundle adjustment process and the like can be calculated even if there is no three-dimensional design information. The amount can be reduced.
  • the conversion unit 102 is based on both the distance between the object and the image pickup device 17 estimated from the captured image 60 and the distance between the information processing device 1 and the object calculated from the three-dimensional design information. You may generate an input value for the distance to the object. For example, the conversion unit 102 may use the three-dimensional coordinates of the point obtained from the average of the distance estimated from the captured image 60 and the distance calculated from the three-dimensional design information as the initial value in the bundle adjustment process.
  • the SLAM processing unit 120 generates a point cloud map, but the mode of the three-dimensional representation is not limited to the point cloud map.
  • the SLAM processing units 120, 1120, and 2120 may generate a set of a plurality of figures having three-dimensional coordinates as map information.
  • FIG. 13 is a diagram showing an example of map information according to the modified example 2.
  • the map information 500 shown in FIG. 13 is obtained by applying a plurality of triangular figures (triangular-patch-cloud) 501a to 501f (hereinafter referred to as triangular patch 501) to the two-dimensional captured image 45. Is.
  • Each triangular patch 501 is a plane figure, but its position and orientation can be changed in three-dimensional space.
  • the orientation of the triangular patch 501 is represented by the normal vector n.
  • the position of the triangular patch 501 is represented by three-dimensional coordinates.
  • the position and orientation of each triangular patch 501 corresponds to the depth of the two-dimensional captured image 45.
  • the SLAM processing unit 120 generates three-dimensional map information by optimizing the positions of the center points and the normal vectors of the plurality of triangular patches 501 applied to the captured image 45.
  • the information processing device 1 of this modification By generating map information as a set of triangular patches 501 in this way, the information processing device 1 of this modification reduces the amount of calculation as compared with the case where the three-dimensional coordinates of points in the three-dimensional space are individually calculated. At the same time, it is possible to generate map information that closely expresses the surrounding environment.
  • the triangular patch 501 is applied to the captured image 45, but the triangular patch 501 may be applied to the BIM information.
  • the conversion unit 102, 1102, 2102 (hereinafter, referred to as the conversion unit 102) may apply the triangular patch 501 to the three-dimensional design information such as BIM information.
  • the conversion unit 102 can determine the boundary of the triangular patch 501 as the boundary of the three-dimensional structure based on the BIM information, in addition to the boundary drawn as an edge on the captured image.
  • the SLAM processing unit 120 When adopting this configuration, the SLAM processing unit 120 generates more accurate map information by correcting the positions and orientations of the plurality of triangular patches 501 applied by the conversion unit 102 based on the SLAM result. be able to.
  • the figures constituting the map information are not limited to the triangular patch 501, and the SLAM processing unit 120 may generate the map information by mesh representation or three-dimensional polygons.
  • the environmental information is the three-dimensional design information, the entry / exit information of the person in the building 9, or the image recognition result of the person in the captured image, but the environmental information is limited to these. It's not something.
  • the environmental information includes information on at least one of the ambient lighting and the weather.
  • the information regarding the ambient lighting is, for example, information indicating whether the lighting for each room or floor of the building 9 is on or off.
  • the information on the weather is information on sunshine conditions such as sunny, cloudy, and rainy in the area including the building 9.
  • the environmental information may include both information on ambient lighting and information on weather, or may include only one or the other.
  • the acquisition units 101 and 1101 acquire information on ambient lighting or weather from the external device 2.
  • the conversion unit 102 generates mask information representing a region where the captured image is likely to be deteriorated, based on the information on the ambient lighting or the weather acquired by the acquisition unit 101.
  • the mask information of the second embodiment may be distinguished as the first mask information, and the mask information of the present modification may be distinguished as the second mask information.
  • the SLAM processing unit 120 of this modification does not use the captured image at least for either self-position estimation or map information generation in the region corresponding to the mask information.
  • the SLAM processing unit 120 may not use the captured image for both the self-position estimation process and the map information generation process in the region corresponding to the mask information, or may not use it for only one of the processes. It may be a thing.
  • the SLAM processing unit 120 may use the captured image for self-position estimation processing for movement in a region corresponding to mask information, or may not use it for generating map information.
  • overexposed areas or underexposed areas may occur in the captured image depending on the lighting or sunshine conditions.
  • the use of such areas may reduce the accuracy of self-position estimation or map information.
  • the deterioration of the accuracy of the self-position estimation or the map information is reduced by not using the captured image for the self-position estimation or the generation of the map information in the region where such an event may occur. ..
  • the SLAM processing unit 120 may not use the captured image at all for estimating its own position or generating map information in the region corresponding to the mask information, but may use it with a lower priority.
  • the information processing device 1 includes a sensor or the like for detecting the surrounding state in addition to the image pickup device 17, the SLAM processing unit 120 displays the detection result by the sensor or the like for the area corresponding to the mask information. It is used to estimate the self-position or generate map information in preference to the captured image.
  • the conversion unit 102 may change the gradation of the captured image based on the environmental information. For example, the conversion unit 102 reduces overexposure or underexposure by changing the dynamic range of the captured image based on information about ambient lighting or weather.
  • the SLAM processing unit 120 executes self-position estimation and map information generation based on the captured image whose gradation has been changed by the conversion unit 102.
  • the information processing device 1 of this modification it is possible to robustly estimate the self-position and generate map information in response to the surrounding environment such as lighting conditions or sunshine conditions.
  • the environmental information may include three-dimensional design information and process information representing the construction process of the building 9.
  • the process information in this modification is information representing the construction schedule or timeline (schedule) of the building 9.
  • the building 9 is in the process of being built, by collating the three-dimensional design information such as BIM information with the process information, it becomes possible to distinguish between the area where the building 9 has been constructed and the area where the building 9 is in the process of being constructed.
  • the 3D design information basically represents a 3D model of the building 9 in the completed state of construction, there may be a difference between the 3D design information and the actual state of the building 9 in the area in the middle of construction. Highly sexual.
  • the conversion unit 102 of this modification generates unfinished area information representing an area of the building 9 in which the construction has not been completed, based on the three-dimensional design information and the process information.
  • the SLAM processing unit 120 of this modified example executes self-position estimation and map information generation for the area corresponding to the unfinished area information without using the three-dimensional design information. For example, the SLAM processing unit 120 executes self-position estimation and map information generation based on the detection result of the captured image or the sensor for the region corresponding to the unfinished region information.
  • the information processing device 1 of this modified example does not use the three-dimensional design information in the region where there is a high possibility that there is a difference between the three-dimensional design information and the actual state of the building 9, so that the building 9 can be used. Even in the middle of construction, it is possible to reduce the decrease in the accuracy of self-position estimation and map information.
  • the SLAM processing unit 120 may use the three-dimensional design information at a lower priority in the area corresponding to the unfinished area information, instead of not using the three-dimensional design information at all.
  • the map information may be generated only in the area corresponding to the unfinished area information. That is, the SLAM processing unit 120 estimates that the structure of the building 9 does not change in the area where the construction is completed, and the map information is only in the area where the structure of the building 9 changes, that is, the area corresponding to the unfinished area information. The amount of calculation is reduced by generating.
  • the tracking unit 103 of the SLAM processing unit 120 uses the captured image obtained by capturing the area other than the area corresponding to the unfinished area information in the tracking process. May be done. This is because, for the area corresponding to the unfinished area information, the structure that is the subject changes depending on the construction work, so it may be difficult to track the point 50 between the captured images captured at different times. Because there is.
  • the information processing device 1 executes the self-position estimation process and the map information generation process, but the external device 2 estimates the self-position.
  • a configuration that executes processing and generation processing of map information may be adopted.
  • the external device 2 may execute the position estimation process of the information processing device 1 and the map information generation process based on the detection result acquired from the information processing device 1 and the environmental information.
  • the external device 2 may be used as an example of the information processing device.
  • the expression "at least one of a, b and c (one)” or “at least one of a, b or c (one)” (including similar expressions). ) Is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as a-a, a-b-b, a-a-b-b-c-c, and the like. It also includes adding elements other than the listed elements (a, b and c), such as having d, such as a-b-c-d.
  • connection when the terms "connected” and “coupled” are used, direct connection / coupling and indirect connection / coupling are used. , Electrically connected / combined, communicatively connected / combined, operatively connected / combined, physically connected / combined, etc. Intended as a term.
  • the term should be interpreted as appropriate according to the context in which the term is used, but any connection / combination form that is not intentionally or naturally excluded is not included in the term. It should be interpreted in a limited way.
  • the physical structure of the element A can execute the operation B. Including that the element A has a configuration and the permanent or temporary setting (setting / configuration) of the element A is set (configured / set) to actually execute the operation B. good.
  • the element A is a general-purpose processor
  • the processor has a hardware configuration capable of executing the operation B, and the operation B is set by setting a permanent or temporary program (instruction). It suffices if it is configured to actually execute.
  • the element A is a dedicated processor, a dedicated arithmetic circuit, or the like, the circuit structure of the processor actually executes the operation B regardless of whether or not the control instruction and data are actually attached. It only needs to be implemented.
  • finding a global optimal value finding an approximation of a global optimal value, finding a local optimal value, and local optimization It should be interpreted as appropriate according to the context in which the term was used, including finding an approximation of the value. It also includes probabilistically or heuristically finding approximate values of these optimal values.
  • the respective hardware when a plurality of hardware performs a predetermined process, the respective hardware may cooperate to perform the predetermined process, or some hardware may perform the predetermined process. You may do all of the above. Further, some hardware may perform a part of a predetermined process, and another hardware may perform the rest of the predetermined process.
  • the hardware that performs the first process and the hardware that performs the second process when expressions such as "one or more hardware performs the first process and the one or more hardware performs the second process" are used. , The hardware that performs the first process and the hardware that performs the second process may be the same or different. That is, the hardware that performs the first process and the hardware that performs the second process may be included in the one or more hardware.
  • the hardware may include an electronic circuit, a device including the electronic circuit, or the like.
  • each storage device (memory) among the plurality of storage devices (memory) stores only a part of the data. It may be stored or the entire data may be stored.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Business, Economics & Management (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)

Abstract

Le dispositif de traitement d'informations selon un mode de réalisation est pourvu d'au moins une mémoire et d'au moins un processeur. Le ou les processeurs sont configurés pour pouvoir exécuter: l'acquisition d'un résultat de détection et d'informations environnementales, le résultat de détection comprenant au moins l'un de l'état de l'entourage du dispositif de traitement d'informations ou de l'état du dispositif de traitement d'informations, les informations environnementales se rapportant à l'environnement de l'entourage du dispositif de traitement d'informations; et l'exécution d'une estimation de l'auto-localisation et la génération d'informations de carte sur la base des informations environnementales et du résultat de détection.
PCT/JP2021/014938 2020-04-15 2021-04-08 Dispositif de traitement d'informations, procédé de traitement d'informations et programme Ceased WO2021210492A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-073040 2020-04-15
JP2020073040 2020-04-15

Publications (1)

Publication Number Publication Date
WO2021210492A1 true WO2021210492A1 (fr) 2021-10-21

Family

ID=78085146

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/014938 Ceased WO2021210492A1 (fr) 2020-04-15 2021-04-08 Dispositif de traitement d'informations, procédé de traitement d'informations et programme

Country Status (1)

Country Link
WO (1) WO2021210492A1 (fr)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023085183A1 (fr) * 2021-11-10 2023-05-19 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et objet mobile
JP2023070207A (ja) * 2021-11-09 2023-05-19 三菱電機株式会社 画像認識装置、画像認識システム、および画像認識方法
JP2023122807A (ja) * 2022-02-24 2023-09-05 日立グローバルライフソリューションズ株式会社 自律走行型ロボット
JP2025084676A (ja) * 2023-11-22 2025-06-03 台達電子工業股▲ふん▼有限公司 ３ｄモデリングに用いるコンピュータプログラム製品およびその移動体除去方法
WO2025197855A1 (fr) * 2024-03-19 2025-09-25 株式会社アイシン Dispositif de création de carte, procédé de création de carte et programme de création de carte

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004133567A (ja) * 2002-10-09 2004-04-30 Hitachi Ltd 移動体およびその位置検出装置
JP2008129614A (ja) * 2006-11-16 2008-06-05 Toyota Motor Corp 移動体システム
JP2018164966A (ja) * 2017-03-28 2018-10-25 清水建設株式会社 位置推定装置、ロボット及び位置推定方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004133567A (ja) * 2002-10-09 2004-04-30 Hitachi Ltd 移動体およびその位置検出装置
JP2008129614A (ja) * 2006-11-16 2008-06-05 Toyota Motor Corp 移動体システム
JP2018164966A (ja) * 2017-03-28 2018-10-25 清水建設株式会社 位置推定装置、ロボット及び位置推定方法

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023070207A (ja) * 2021-11-09 2023-05-19 三菱電機株式会社 画像認識装置、画像認識システム、および画像認識方法
JP7720771B2 (ja) 2021-11-09 2025-08-08 三菱電機株式会社 画像認識装置、画像認識システム、および画像認識方法
WO2023085183A1 (fr) * 2021-11-10 2023-05-19 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et objet mobile
JP2023122807A (ja) * 2022-02-24 2023-09-05 日立グローバルライフソリューションズ株式会社 自律走行型ロボット
JP2025084676A (ja) * 2023-11-22 2025-06-03 台達電子工業股▲ふん▼有限公司 ３ｄモデリングに用いるコンピュータプログラム製品およびその移動体除去方法
JP7765572B2 (ja) 2023-11-22 2025-11-06 台達電子工業股▲ふん▼有限公司 ３ｄモデリングに用いるコンピュータプログラム製品およびその移動体除去方法
WO2025197855A1 (fr) * 2024-03-19 2025-09-25 株式会社アイシン Dispositif de création de carte, procédé de création de carte et programme de création de carte

Similar Documents

Publication Publication Date Title
WO2021210492A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et programme
US8401242B2 (en) Real-time camera tracking using depth maps
WO2019138678A1 (fr) Dispositif de traitement d'informations, son procédé de commande, programme, et système d'aide à la conduite d'un véhicule
JP7131994B2 (ja) 自己位置推定装置、自己位置推定方法、自己位置推定プログラム、学習装置、学習方法及び学習プログラム
JP5480667B2 (ja) 位置姿勢計測装置、位置姿勢計測方法、プログラム
JP2020030204A (ja) 距離測定方法、プログラム、距離測定システム、および可動物体
CN113870343A (zh) 相对位姿标定方法、装置、计算机设备和存储介质
CN103926933A (zh) 一种无人飞行器室内同时定位与环境建模方法
JP2019125116A (ja) 情報処理装置、情報処理方法、およびプログラム
CN110260866A (zh) 一种基于视觉传感器的机器人定位与避障方法
CN110764110B (zh) 路径导航方法、装置及计算机可读存储介质
CN112967340A (zh) 同时定位和地图构建方法、装置、电子设备及存储介质
KR20210116161A (ko) 단일 체커보드를 이용하는 이종 센서 캘리브레이션 방법 및 장치
Hsu et al. Application of multisensor fusion to develop a personal location and 3D mapping system
JP2020149186A (ja) 位置姿勢推定装置、学習装置、移動ロボット、位置姿勢推定方法、学習方法
JP2022138037A (ja) 情報処理装置、情報処理方法およびプログラム
WO2022014322A1 (fr) Système de traitement d'informations et dispositif de traitement d'informations
WO2022198508A1 (fr) Procédé et appareil de suggestion d'anomalie d'objectif, plate-forme mobile et support de stockage lisible
KR20240123863A (ko) 온실용 로봇 및 이의 위치 및 자세 추정 방법
US20240106998A1 (en) Miscalibration detection for virtual reality and augmented reality systems
US11294379B2 (en) Systems and methods for controlling intelligent wheelchair
CN120510212A (zh) 一种融合点线流特征的鲁棒视觉惯性slam方法、存储介质及设备
US20190369631A1 (en) Intelligent wheelchair system based on big data and artificial intelligence
US20240069203A1 (en) Global optimization methods for mobile coordinate scanners
CN117541655A (zh) 融合视觉语义消除radar建图z轴累积误差的方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21788847

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21788847

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP