[go: up one dir, main page]

WO2019170166A1 - 深度相机标定方法以及装置、电子设备及存储介质 - Google Patents

深度相机标定方法以及装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2019170166A1
WO2019170166A1 PCT/CN2019/085515 CN2019085515W WO2019170166A1 WO 2019170166 A1 WO2019170166 A1 WO 2019170166A1 CN 2019085515 W CN2019085515 W CN 2019085515W WO 2019170166 A1 WO2019170166 A1 WO 2019170166A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
pose
frame
label
tag
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/085515
Other languages
English (en)
French (fr)
Inventor
方璐
苏卓
韩磊
戴琼海
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua Berkeley Shenzhen Institute
Original Assignee
Tsinghua Berkeley Shenzhen Institute
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua Berkeley Shenzhen Institute filed Critical Tsinghua Berkeley Shenzhen Institute
Publication of WO2019170166A1 publication Critical patent/WO2019170166A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • Embodiments of the present invention relate to machine vision technology, for example, to a depth camera calibration method and apparatus, an electronic device, and a storage medium.
  • RGB-D cameras ie depth cameras
  • RGB-D camera combines traditional RGB camera with depth camera, which has the advantages of high precision, small size, large amount of information, passivity and rich information.
  • a camera calibration method based on a specific calibration object is generally adopted, which is relatively cumbersome, and requires a field of view between adjacent cameras to have a large overlapping range, and cannot adapt to a situation in which the field of view overlap is small.
  • the calibration method based on Simultaneous Localization And Mapping (SLAM) uses the camera to move according to the preset trajectory to capture images, and then offline processing the image for calibration. It is not possible to quickly calibrate 360° panoramic RGB-D cameras online.
  • the embodiment of the invention provides a depth camera calibration method and device, an electronic device and a storage medium, so as to realize real-time online panoramic multi-camera self-calibration without human intervention.
  • an embodiment of the present invention provides a depth camera calibration method, including:
  • an embodiment of the present invention further provides a depth camera calibration apparatus, including:
  • a camera control module configured to control at least two depth cameras in the panoramic depth camera system to synchronously acquire images during motion, wherein each depth camera is provided with a label;
  • a pose obtaining module configured to acquire a pose of the at least one first label camera when acquiring a plurality of frame images
  • the relative pose calculation module is configured to overlap the corresponding image according to the historical perspective and the pose of the first label camera when each frame image is captured, in a case where the second label camera overlaps with the first label camera appearance history angle Determining a relative pose of the second tag camera at the same time as the first tag camera.
  • an embodiment of the present invention further provides an electronic device, including:
  • One or more processors are One or more processors;
  • a memory set to store one or more programs
  • a panoramic depth camera system comprising at least two depth cameras, the at least two depth cameras covering a panoramic field of view for acquiring images
  • the one or more programs are executed by the one or more processors such that the one or more processors implement a depth camera calibration method as described in any of the embodiments herein.
  • the embodiment of the present invention further provides a computer readable storage medium, where a computer program is stored, and when the program is executed by the processor, the depth camera calibration method according to any embodiment of the present application is implemented.
  • FIG. 1 is a flowchart of a depth camera calibration method according to Embodiment 1 of the present invention.
  • FIG. 2 is a schematic diagram of a panoramic depth camera system according to Embodiment 1 of the present invention.
  • FIG. 3 is a flowchart of a depth camera calibration method according to Embodiment 2 of the present invention.
  • FIG. 5 is a flowchart of obtaining a first label camera pose according to Embodiment 4 of the present invention.
  • FIG. 6 is a structural block diagram of a depth camera calibration apparatus according to Embodiment 5 of the present invention.
  • FIG. 7 is a schematic structural diagram of an electronic device according to Embodiment 6 of the present invention.
  • Embodiment 1 is a flowchart of a depth camera calibration method according to Embodiment 1 of the present invention.
  • the present embodiment is applicable to a multi-camera self-calibration.
  • the calibration in this embodiment refers to calculating a relative pose between cameras.
  • the method may be performed by a depth camera calibration device or an electronic device, and the depth camera calibration device may be implemented by software and/or hardware, for example, by a central processing unit (CPU), and the camera is controlled and calibrated by the CPU. Further, the device can be integrated in a portable mobile electronic device. As shown in FIG. 1, the method may include, for example, step S101, step S102, and step S103.
  • step S101 at least two depth cameras in the panoramic depth camera system are controlled to synchronously acquire images during the motion.
  • each depth camera is set with a corresponding label.
  • the panoramic depth camera system (which may be simply referred to as a camera system) includes at least two depth cameras (RGB-D cameras) that cover a 360-degree panoramic field of view.
  • the size of the camera and the number of cameras to be used can be selected according to specific needs, and multiple cameras can be fixed on the platform (for example, components of rigid structure) according to the size of the camera and the number of cameras to meet the field of view coverage.
  • the number of cameras required is determined according to the field of view of the single camera and the required field of view. The sum of the fields of view of all cameras needs to be larger than the required field of view.
  • a single camera with the same viewing angle is used.
  • Six cameras can be used in consideration of the vertical field of view. Depending on the size of the camera used and the number of cameras, the camera can be reasonably laid out, exemplarily, with a length of 10 to 15 cm, a width of 3 to 5 cm, a height of 3 to 5 cm, and a resolution of 640 ⁇ 480 RGB.
  • a positive hexagonal prism with a base length of 5 cm is selected as a shaft, and the camera lens is fixed outwardly on the shaft by a fixing member, as shown in FIG.
  • adjacent cameras may have no overlap of views or a small overlap of views, for example, one or two degrees of view overlap.
  • the depth camera in the panoramic depth camera system synchronously acquires images
  • the synchronization mode may be hardware synchronization or software synchronization.
  • hardware synchronization refers to using a signal (such as a rising edge signal) to simultaneously trigger all cameras to capture images at the same time
  • the software mode refers to time stamping multiple images when buffering images captured by multiple cameras.
  • the image with the closest timestamp is considered to be the image acquired at the same time, that is, maintaining a buffer area, and outputting the image frames of the most similar time stamps of multiple cameras at a time.
  • the panoramic depth camera system needs to be in motion to acquire images as observation information.
  • a robot-equipped panoramic depth camera system or a user-held panoramic depth camera system to perform free motion in a conventional room, and to ensure as much as possible to increase the rotational motion during the movement to increase the overlap of historical angles between the cameras during the exercise, and obtain more observations.
  • Information for easy calibration is a feature that is associated with a robot-equipped panoramic depth camera system or a user-held panoramic depth camera system.
  • Each depth camera is provided with a corresponding attribute label, which is used to distinguish the role of the camera.
  • the first label indicates that the camera is a reference camera
  • the second label indicates that the camera is a non-reference camera.
  • the specific value of the label may be 0. Or 1, for example, 0 represents a non-reference camera and 1 represents a reference camera.
  • step S102 the pose of the at least one first tag camera when acquiring each frame image is acquired.
  • the camera character may be determined according to the camera tag.
  • At least one of the at least two depth cameras included in the panoramic depth camera system may be preset as any reference camera in the system, and the first tag camera is the reference camera.
  • the first tag camera is the reference camera.
  • a reference camera can be preset in consideration of the accuracy of the calibration. For the reference camera in the camera system, in the whole calibration process, it is necessary to acquire the pose of the reference camera when capturing multiple frame images in real time.
  • the pose is the relative position when the reference camera captures the current frame image relative to the image of the previous frame.
  • the pose changes may include a position (translation matrix T) and a pose (rotation matrix R) involving six degrees of freedom X, Y, Z, ⁇ , ⁇ , and ⁇ , where X, Y, and Z are camera positions
  • T position
  • R pose
  • X, Y, and Z are camera positions
  • the parameters, ⁇ , ⁇ , and ⁇ are parameters of the camera pose.
  • step S103 if the second tag camera overlaps with the first tag camera appearance history angle, the corresponding image is overlapped according to the historical view angle and the pose of the first tag camera when the plurality of frame images are acquired, and the second tag camera is determined to be the first The relative pose of the label camera at the same time.
  • each second label camera overlaps with the historical perspective of the first label camera, thereby obtaining the relative position of each second label camera and the first label camera at the same time. That is, the relative pose between multiple cameras is obtained, and the real-time online camera self-calibration process is completed.
  • the depth cameras to be calibrated in the panoramic depth camera system are in motion, and the cameras of these motions synchronously acquire images in real time and maintain key frames under multiple cameras.
  • the single camera pose estimation is performed according to the acquired multiple frame images, and the camera pose when each frame image is acquired is obtained; at the same time, for the non-reference camera, the single camera pose estimation is not performed.
  • the collected plurality of frame images are used as observation information to determine whether or not the historical perspective overlaps with any of the reference cameras. If they overlap, the relative pose of the non-reference camera and the corresponding reference camera at the same time can be calculated.
  • the depth camera calibration method of the embodiment is based on a first label camera preset in the panoramic depth camera system, and uses a second label camera and a first label camera in a process of capturing images while the panoramic depth camera system is moving. Overlap, determining the relative pose of the second label camera and the first label camera, enabling fast online self-calibration of multiple cameras.
  • the method does not require a calibration object, and does not require a fixed large field of view overlap between adjacent cameras. When the camera system is built, there is a small overlapping viewing angle between adjacent cameras or no overlapping viewing angle, and the motion of the panoramic depth camera system is used to make different
  • the camera collects images with overlapping historical views and can be calibrated accordingly.
  • the method has a small amount of calculation and can be calibrated online based on the CPU.
  • the depth camera calibration method of the embodiment of the present invention is applicable to an application background such as indoor robot navigation or three-dimensional scene reconstruction.
  • the present embodiment further optimizes the determination of the overlap of historical angles of view and the calculation of the relative pose between cameras.
  • the method further includes: performing feature point matching on the current frame image collected by the plurality of depth cameras and the previous key frame thereof respectively. And obtaining a conversion relationship matrix between the two frames of images; if the conversion relationship matrix is greater than or equal to the preset conversion threshold, determining that the current frame image is the current key frame under the corresponding depth camera, and storing the key frame.
  • the first frame image acquired by each depth camera defaults to a key frame.
  • the frame image is compared with the closest key frame of the camera to determine whether the frame image is a key. frame.
  • the preset conversion threshold is set in advance according to the motion condition when the image is acquired by the depth camera. For example, if the pose changes greatly when the camera takes two adjacent frames, the preset conversion threshold is set larger.
  • Feature point matching can use related matching algorithms, for example, feature matching of color images acquired by RGB-D cameras based on Oriented FAST and Rotated BRIEF (ORB) algorithm (sparse algorithm), or using direct method Perform dense registration.
  • FIG. 3 is a flowchart of a depth camera calibration method according to Embodiment 2 of the present invention. As shown in FIG. 3, the method includes: Step S301 to Step S306.
  • step S301 at least two depth cameras in the panoramic depth camera system are controlled to synchronously acquire images during the motion.
  • each depth camera is set with a corresponding label.
  • step S302 a pose of the at least one first tag camera when acquiring a plurality of frame images is acquired.
  • step S303 the current frame image acquired by the second tag camera is matched with the historical key frame of the at least one first tag camera; if the historical key frame and the current frame image reach the matching threshold, the second tag camera is determined.
  • the historical perspective overlaps with the corresponding first label camera.
  • the second tag camera performs feature point matching on the key frame (ie, historical key frame) stored under the first tag camera every time a new frame image is acquired, and if the matching threshold is reached, it is considered that the historical view overlap occurs.
  • Feature point matching can use related matching algorithms, for example, feature matching of color images acquired by RGB-D cameras based on sparse ORB algorithm, or dense registration using direct method.
  • the matching threshold may be the number of preset matching feature points.
  • step S304 the abnormal data in the correspondence between the current frame image acquired by the second tag camera and the feature point of the corresponding historical key frame is removed, and the relative position of the current frame image and the corresponding historical key frame is calculated according to the remaining feature point correspondence relationship. relationship.
  • step S305 a transformation relationship between the pose when the second label camera captures the current frame image and the pose of the first label camera when acquiring the corresponding historical key frame is determined according to the relative positional relationship.
  • step S306 determining the relative pose of the second label camera and the first label camera at the current frame time according to the transformation relationship and the plurality of poses between the acquisition of the corresponding historical key frame and the acquisition of the current frame image by the first label camera.
  • the random sample Consensus (RANSAC) algorithm may be used to remove the anomaly data, and the relative positional relationship between the two frames of the overlapping view angles may be calculated according to the remaining feature point correspondence relationship; according to the relative positional relationship between the images, Find the corresponding camera pose change relationship. Since the pose of the first tag camera when collecting a plurality of frame images is always estimated, the first tag camera pose between the historical keyframe and the current frame overlapped by the historical perspective is known, thereby deriving the history The relative position of the second tag camera with the overlapping angles of view and the first tag camera at the current frame time (ie, the moment when the historical perspective overlaps). The relative pose also includes six degrees of freedom, involving the rotation matrix R and the translation matrix T.
  • the panoramic depth camera system includes three depth cameras A, B, and C, with camera A being the first label camera and cameras B and C being the second label camera.
  • the three cameras synchronously acquire images.
  • the 10th frame of camera B overlaps with the 5th frame (historical key frame) of camera A, and the features between the images overlapped according to the historical perspective of the two frames.
  • Point matching can be used to calculate the transformation relationship between the pose of the 10th frame captured by camera B and the pose of the 5th frame acquired by camera A. Since the camera A is always performing pose estimation and recording the poses of the 5th frame to the 10th frame, thereby deriving one frame, the relative pose of the camera B and the camera A in the 10th frame can be obtained.
  • the historical key frame is used to determine that the second tag camera overlaps with the historical view of the first tag camera, and the pose relationship between the second tag camera and the first tag camera is calculated by the two frames of the overlapping angle of view. Then, using the first label camera pose between the corresponding historical key frame and the current frame under the first label camera, the relative pose of the second label camera and the first label camera at the same moment can be derived, and the calculation is simple and capable Guarantee fast online calibration.
  • the calculated Global optimization of the relative pose may include: if there is a key frame in the current frame image acquired by the depth camera synchronously calculated relative to the pose, the history under the depth camera according to the key frame and the calculated relative pose The key frame performs loopback detection; if the loopback is successful (that is, the matching historical keyframe is found), the relative pose between the corresponding depth cameras is updated according to the key frame and the corresponding historical keyframe.
  • the loopback detection is to determine whether the depth camera moves to a place that has been reached or has a large overlap with the historical angle of view.
  • Loopback detection is performed for the key frame rate. For each frame of image captured by the depth camera, it is necessary to determine whether the frame image is a key frame. If yes, loopback detection is performed. If not, wait for the next key frame to arrive for loopback detection. .
  • the determination of the key frame may be that the image of each frame captured by the depth camera is matched with the last key frame corresponding to the camera to obtain a transformation relationship matrix between the two frames; if the conversion relationship matrix is greater than or equal to the pre-predetermined Setting the conversion threshold determines that the current frame image is the current key frame under the camera.
  • the comparison object of the loopback detection in this embodiment is a historical key frame under the depth camera in which the relative pose is calculated in the camera system, instead of merely comparing its own historical key frame, if loopback Successfully, the relative pose between the corresponding depth cameras is updated according to the current key frame and the corresponding historical key frame to reduce the cumulative error and improve the accuracy of the relative pose between the cameras.
  • the loopback detection of the current key frame and the historical key frame may be performed by matching the current key frame with the ORB feature point of the historical key frame. If the matching degree is high, the loopback is successful.
  • one or more historical key frames with high matching degree may be selected to perform an optimized update of the relative pose between the corresponding cameras. It should be noted that if the matching historical key frame belongs to the depth camera itself, the depth camera's own pose is optimized according to the current key frame and the historical key frame.
  • the above-mentioned relative pose optimization update process can be started after obtaining the relative pose of a pair of cameras in the camera system, and the relative pose between the cameras is updated as the camera moves to acquire images, if the camera system
  • the relative pose between all the cameras is calculated, and when the preset condition is met (for example, the number of optimizations of the relative pose between multiple cameras reaches the preset number of times, or the preset error requirement is met), the optimization update is stopped. Get the final more accurate calibration parameters.
  • the relative pose between cameras B and A is also optimized based on the key frames captured by camera A and/or camera B. If the relative pose of the camera C and the camera A is obtained at the same time, the relative pose of the camera B and the camera C can be further derived, thereby obtaining the relative pose between the plurality of cameras, and then the optimization is performed. . For the current frame time, the three cameras synchronously acquire corresponding images.
  • the key frame a and b performs loopback detection, both of which are loopback success, and are optimized and updated according to key frames a and b respectively.
  • the calculation of relative poses during the optimization and update process is the same as the initial relative pose calculation method, and will not be described here.
  • the calculated relative position and orientation of the camera can be optimized in real time according to the image acquired by the camera, and the accuracy of the camera calibration is improved.
  • the depth camera calibration method of each of the above embodiments is to use a first label camera preset in the panoramic depth camera system as a reference to obtain a relative pose between the cameras.
  • This embodiment provides another depth camera calibration method based on the above embodiments to further speed up the online calibration speed.
  • 4 is a flowchart of a depth camera calibration method according to Embodiment 3 of the present invention. As shown in FIG. 4, the method includes: Step S401 to Step S405.
  • step S401 at least two depth cameras in the panoramic depth camera system are controlled to synchronously acquire images during the motion.
  • each depth camera is set with a corresponding label.
  • step S402 a pose of the at least one first tag camera when acquiring a plurality of frame images is acquired.
  • step S403 if the second tag camera overlaps with the first tag camera's historical view angle, the corresponding image is overlapped according to the historical view angle and the pose of the first tag camera when the plurality of frame images are acquired, and the second tag camera is determined to be the first The relative pose of the label camera at the same time.
  • step S404 the label of the second tag camera is modified to the first tag.
  • the second tag camera overlapping with the historical view angle of the first tag camera is included in the reference range to expand the reference range.
  • step S405 the pose posture when acquiring the plurality of frame images by the at least one first label camera is repeatedly performed, determining the relative pose of the second label camera with the historical perspective overlap and the corresponding first label camera at the same time and modifying The operation of the tag (ie, repeating S402 to S404) until the second tag camera is not included in the at least two depth cameras.
  • the labels of the depth camera in the camera system are all modified to the first label, indicating that each second label camera calculates the relative pose with other cameras, and the calibration result is obtained.
  • the second label camera is also used as a reference to expand the reference range, and the history of the other second label cameras overlaps. Probability, which can further speed up the calibration.
  • FIG. 5 is a flowchart of acquiring a first label camera pose according to Embodiment 4 of the present invention. As shown in FIG. 5, the method includes: Step S501, Step S502, and Step S503.
  • step S501 for each first tag camera, feature extraction is performed on each frame image acquired by the first tag camera to obtain at least one feature point of each frame image.
  • the feature extraction of the image is to find some pixel points (ie, feature points) having landmark features in the frame image, for example, may be corner points, textures, and pixel points in the edge of the image.
  • Feature extraction for each frame of image may use an ORB algorithm to find at least one feature point in the frame image.
  • step S502 feature point matching is performed on two adjacent frames of images to obtain a feature point correspondence relationship between adjacent two frames of images.
  • the sparse ORB feature registration or the direct method dense registration can be used to obtain the correspondence between the feature points of the adjacent two frames, that is, the correspondence between the feature points of the adjacent two frames is obtained.
  • H represents the Hamming distance between two feature points X1, X2, XORing the two feature points, and counting the number of 1 as the Hamming distance of a feature point between adjacent two frames of images (ie feature point correspondence).
  • iterative calculation can be performed using the Gauss-Newton method.
  • it may be a posture in which the calculation of the reprojection error is minimized.
  • r( ⁇ ) denotes a vector containing all reprojection errors
  • J( ⁇ ) is the Jacobian matrix of r( ⁇ )
  • ⁇ represents the Lie algebra of the camera pose
  • ⁇ represents the increase of r( ⁇ ) at each iteration.
  • R i represents the rotation matrix of the camera when the image of the ith frame is acquired
  • R j represents the rotation matrix of the camera when the image of the jth frame is acquired
  • C i,j represents a set of correspondence points of the feature points of the i-th frame image and the j-th frame image
  • -1 represents the i-th frame image
  • [] ⁇ represents the vector product
  • represents the norm of C i,j .
  • the nonlinear term The expression is:
  • each frame image must exist in another frame.
  • the feature points not included in one frame of image are subjected to the matching operation in step S502, and an abnormal correspondence relationship occurs.
  • Calculating the camera pose is to solve the nonlinear least squares problem between two frames of images with the following cost function:
  • E represents a re-projection error of the ith frame image in the Euclidean space compared to the j-th frame image (in the present embodiment, the previous frame image);
  • T i represents the pose of the camera when the ith frame image is acquired (according to the foregoing
  • the explanation of the camera pose can be seen that the actual position refers to the pose change of the image of the i-th frame relative to the image of the previous frame)
  • T j represents the pose of the camera when the image of the j-th frame is captured
  • N represents the total frame acquired by the camera.
  • the present embodiment does not directly calculate the cost function of the equation (2), but calculates the linear component including the second-order statistic relationship of the remaining feature points and the nonlinear component including the camera pose.
  • Nonlinear term The expression of the nonlinear term The linear part that is fixed between two frames of image when calculating As a whole, W is calculated. It does not need to be calculated according to the number of feature points. It reduces the complexity of camera pose calculation and enhances the real-time performance of camera pose calculation.
  • Equation (1) The derivation process of equation (1) is described below, and the principle of reducing the complexity of the algorithm is analyzed in combination with the derivation process.
  • the camera pose T i [R i /t i ] when the camera captures the ith frame image in the Euclidean space, in fact, T i refers to the image of the jth frame when the camera captures the ith frame image (this embodiment)
  • the pose transformation matrix of the middle frame image of the middle finger includes a rotation matrix R i and a translation matrix t i .
  • the rigid transformation T i in the Euclidean space is represented by the Lie algebra ⁇ i on the SE3 space, that is, ⁇ i also represents the camera pose when the camera acquires the ith frame image, and T( ⁇ i ) maps the Lie algebra ⁇ i Is the T i in the Euclidean space.
  • Represents the lth row in the rotation matrix R i ; t il represents the lth element in the translation vector t i , l 0, 1, 2.
  • I 3 ⁇ 3 represents a 3 ⁇ 3 unit matrix.
  • the four non-zero 6 ⁇ 6 submatrices are: Below As an example, the other three non-zero sub-matrices are similarly calculated and will not be described again.
  • each correspondence The Jacobian matrix is composed of geometric terms ⁇ i , ⁇ j and structure terms Decide.
  • their corresponding Jacobian matrices share the same geometric item, but have different structural items.
  • calculate The correlation algorithm depends on the number of feature point correspondences in C i,j , and the present embodiment can calculate the complexity with a fixed complexity. It is only necessary to calculate the second-order statistic W of the structure item, without each corresponding relationship to participate in the calculation of the relevant structural item, ie
  • the four non-zero sub-matrices can be calculated by the complexity O(1) instead of the complexity O(
  • the sparse matrices J T J and J T r required in the iterative step of nonlinear Gauss-Newton optimization for ⁇ -(J( ⁇ ) T J( ⁇ )) -1 J( ⁇ ) T r( ⁇ )
  • the complexity O(M) can be efficiently calculated instead of the original computational complexity O(N coor ), N coor represents the total number of correspondences of all feature points of all frame pairs, and M represents the number of frame pairs.
  • O(N coor ) is approximately 300 in sparse matching and approximately 10,000 in dense matching, much larger than the number M of frame pairs.
  • a global consistent optimization update may be performed on the acquired pose. For example, if the current frame image collected by the first tag camera is a key frame, the loop detection is performed according to the current key frame and the historical key frame of the first tag camera; if the loopback is successful, the acquired according to the current key frame pair.
  • the first label camera poses a globally consistent optimized update.
  • the determination of the key frame may be that the image of each frame captured by the first label camera is matched with the previous key frame corresponding to the camera to obtain a transformation relationship matrix between the two frames; if the conversion relationship matrix is greater than or Equal to the preset conversion threshold, it is determined that the current frame image is the current key frame under the camera.
  • the globally consistent optimization update means that during the calibration process, as the camera moves to the point where it has arrived or has a large overlap with the historical perspective, the current frame image is consistent with the captured image, instead of Produce staggering, aliasing and other phenomena.
  • Loopback detection is based on the current observation of the depth camera to determine whether the camera has moved to a place that has been reached or has a large overlap with the historical perspective. If the loopback is successful, the first label camera pose is globally consistently optimized according to the current key frame. Update to reduce the cumulative error.
  • the loopback detection of the current key frame and the historical key frame may be performed by matching the current key frame with the ORB feature point of the historical key frame. If the matching degree is high, the loopback is successful. In an embodiment, one or more historical key frames with high matching degree may be selected to perform global consistent optimization update of the camera pose according to the matching degree between the current key frame and the historical key frame.
  • the camera poses globally consistent optimization update, that is, according to the correspondence between the current key frame and one or more historical key frames with high matching degree, Minimizes the conversion error problem between the current keyframe of the cost function and all historical keyframes with high matching.
  • T i ⁇ SE3,i ⁇ [1,N-1]) represent all pairs of frames (any historical matching key frame and current key frame Conversion error for one frame pair; N indicates the number of historical key frames with high matching degree with the current key frame; E i,j indicates the conversion error between the i-th frame and the j-th frame, and the conversion error is the re-projection error .
  • the relative pose of the non-key frame and its corresponding key frame needs to be kept unchanged.
  • the optimization update algorithm may use the related BA algorithm, or step S503 may be used.
  • the method in order to improve the optimization speed, and will not be described in detail.
  • the algorithm of the embodiment ie, the method in S503 can also be used for the calculation and optimization process of the relative pose between cameras.
  • the linear pose of the second-order statistic of the feature point and the nonlinear component including the pose of the camera are used to iteratively calculate the pose of the camera, and the nonlinear term is performed.
  • Fixed linear part when calculating As a whole W is calculated, which reduces the complexity of camera pose calculation, enhances the real-time performance of camera pose calculation, and has low hardware requirements.
  • the embodiment of the present invention can implement pose estimation and optimization based on the flow and principle of SLAM.
  • the pose estimation is implemented by the front-end visual odometer thread
  • the pose optimization is implemented by the back-end loop detection and optimization algorithm, for example, The associated beam adjustment (BA) algorithm or the algorithm in this embodiment.
  • BA beam adjustment
  • SLAM SLAM
  • pose estimation and optimization of the first label camera and the relative pose between the cameras is calculated by the overlap of the angles of view, and the relative pose is calculated. These operations can be performed simultaneously.
  • the map in SLAM refers to the motion trajectory of the camera in the world coordinate system and the position of the key frames observed in the motion trajectory in the world coordinate system. If the camera system is rigidly deformed due to physical impact, the embodiment only needs to start the calibration procedure for rapid recalibration without re-arranging the calibration object.
  • the embodiment provides a depth camera calibration device, which can be used to perform the depth camera calibration method provided by any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.
  • the device can be implemented by means of hardware and/or software, for example by means of a CPU.
  • the depth camera calibration device and the panoramic depth camera system need to transmit control signals and images, etc.
  • There are many communication methods between the two for example, communication through a serial port, a network cable, or the like, or wireless through Bluetooth, wireless broadband, etc. The way to communicate.
  • the device includes a camera control module 61, a pose acquisition module 62, and a relative pose calculation module 63.
  • the camera control module 61 is configured to control at least two depth cameras in the panoramic depth camera system to synchronously acquire images during motion, wherein each depth camera is provided with a corresponding label.
  • the pose acquisition module 62 is configured to acquire a pose when at least one first label camera captures a plurality of frame images.
  • the relative pose calculation module 63 is configured to: when the second label camera overlaps with the first label camera appearance history angle, overlap the corresponding image according to the historical perspective and the pose of the first label camera when acquiring the plurality of frame images, and determine The second label camera is in a relative position at the same time as the first label camera.
  • the foregoing apparatus may further include: a label modification module and an operation execution module.
  • a label modification module configured to modify the label of the second label camera to the first label after the relative pose calculation module 63 calculates the relative pose of the second label camera and the first label camera at the same time;
  • An operation execution module configured to repeatedly perform acquiring a pose of the at least one first label camera when acquiring a plurality of frame images, determining a relative position of the second label camera having a historical perspective overlap and a corresponding first label camera at the same time;
  • the operation of the label is modified (ie, the operations of the pose acquisition module 62, the relative pose calculation module 63, and the label modification module are repeated) until the second label camera is not included in the at least two depth cameras of the panoramic depth camera system.
  • the apparatus may further include: a key frame determining module configured to respectively set the plurality of depth cameras while the pose acquiring module 62 acquires the pose of the at least one first label camera when acquiring the plurality of frame images
  • the collected current frame image is matched with the feature key of the previous key frame to obtain a conversion relationship matrix between the two frames; if the conversion relationship matrix is greater than or equal to the preset conversion threshold, the current frame image is determined to be the corresponding depth camera.
  • the current key frame and store the key frame, specifically storing the key frame and the depth camera to which it belongs.
  • the apparatus may further include: a viewing angle overlap determining module configured to: before determining the relative pose of the second label camera and the first label camera at the same time, the current frame image acquired by the second label camera and The historical key frame of the at least one first tag camera performs feature point matching; if the historical key frame and the current frame image reach a matching threshold, it is determined that the second tag camera overlaps with the corresponding first tag camera appearance history angle.
  • a viewing angle overlap determining module configured to: before determining the relative pose of the second label camera and the first label camera at the same time, the current frame image acquired by the second label camera and The historical key frame of the at least one first tag camera performs feature point matching; if the historical key frame and the current frame image reach a matching threshold, it is determined that the second tag camera overlaps with the corresponding first tag camera appearance history angle.
  • the relative pose calculation module 63 includes a relative position relationship calculation unit, a transformation relationship calculation unit, and a relative pose calculation unit.
  • the relative position relationship calculation unit is configured to remove the abnormal data in the correspondence between the current frame image acquired by the second tag camera and the feature point of the corresponding historical key frame, and calculate the current frame image and the corresponding historical key frame according to the remaining feature point correspondence relationship Relative positional relationship;
  • the transformation relationship calculation unit is configured to calculate, according to the relative positional relationship, a transformation relationship between a pose when the second label camera acquires the current frame image and a pose of the first label camera when acquiring the corresponding historical key frame;
  • the relative pose calculation unit is configured to determine, according to the transformation relationship and the plurality of poses between the collection of the corresponding historical key frame and the acquisition of the current frame image, the second label camera and the first label camera at the current frame moment. Relative pose.
  • the pose acquisition module 62 includes a feature extraction unit, a feature matching unit, and an iterative computing unit.
  • a feature extraction unit configured to perform feature extraction on each frame image acquired by the first tag camera for each first tag camera to obtain at least one feature point of each frame image
  • the feature matching unit is configured to perform feature point matching on the adjacent two frames of images to obtain a feature point correspondence relationship between adjacent two frames of images;
  • An iterative calculation unit configured to remove anomaly data in a correspondence of feature points, and calculate a linear component of a second-order statistic of the remaining feature points and a nonlinear component including a camera pose to calculate J( ⁇ ) T J( ⁇ )
  • r( ⁇ ) denotes a vector containing all reprojection errors
  • J( ⁇ ) is the Jacobian matrix of r( ⁇ )
  • ⁇ represents the Lie algebra of the camera pose
  • ⁇ represents the increase of r( ⁇ ) at each iteration.
  • R i represents the rotation matrix of the camera when the image of the ith frame is acquired
  • R j represents the rotation matrix of the camera when the image of the jth frame is acquired
  • C i,j represents a set of correspondence points of the feature points of the i-th frame image and the j-th frame image
  • -1 represents the i-th frame image
  • [] ⁇ represents the vector product
  • represents the norm of C i,j .
  • the nonlinear term The expression is:
  • Represents a linear component; And r jl represent nonlinear components, Is the lth row in the rotation matrix R i , and r jl is the transpose of the l th row in the rotation matrix R j , l 0, 1, 2 .
  • the foregoing apparatus may further include: a loopback detection module and an optimization update module.
  • the loop detection module is configured to: after acquiring the pose of the at least one first label camera to capture the plurality of frame images, if the current frame image captured by the first label camera is a key frame, according to the current key frame and the first label camera Historical key frames for loopback detection;
  • the optimization update module is configured to perform globally consistent optimization update on the acquired first label camera pose according to the current key frame in the case of successful loopback.
  • the loopback detection module is further configured to: after the relative pose calculation module 63 calculates the relative pose of the second label camera and the first label camera at the same time, if the depth camera of the relative pose has been calculated If there is a key frame in the current frame image acquired synchronously, loopback detection is performed according to the key frame and the historical key frame under the depth camera in which the relative pose has been calculated; the above optimization update module is also set to be successful in the loopback process.
  • the relative pose between the corresponding depth cameras is updated according to the key frame and the corresponding historical key frame.
  • each unit and module included is divided according to functional logic, but is not limited to the above division, as long as the corresponding function can be implemented;
  • the specific names of the functional units are also for convenience of distinguishing from each other and are not intended to limit the scope of protection of the present application.
  • the embodiment provides an electronic device comprising: one or more processors, a memory and a panoramic depth camera system.
  • the memory is set to store one or more programs.
  • a panoramic depth camera system includes at least two depth cameras that cover a panoramic field of view and are arranged to capture images.
  • the one or more programs are executed by the one or more processors such that the one or more processors implement a depth camera calibration method as described in any of the embodiments herein.
  • FIG. 7 is a schematic structural diagram of an electronic device according to Embodiment 6 of the present invention.
  • FIG. 7 illustrates a block diagram of an exemplary electronic device suitable for use in implementing embodiments of the present invention.
  • the electronic device 712 shown in FIG. 7 is merely an example and should not impose any limitation on the function and scope of use of the embodiments of the present invention.
  • electronic device 712 is embodied in the form of a general purpose computing device.
  • Components of electronic device 712 may include, but are not limited to, one or more processors (or processing unit 716), system memory 728, and a bus 718 that connects different system components, including system memory 728 and processing unit 716.
  • Bus 718 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures.
  • these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA Bus, and the Video Electronics Standards. Association, VESA) Local Bus and Peripheral Component Interconnect (PCI) bus.
  • ISA Industry Standard Architecture
  • MAC Micro Channel Architecture
  • VESA Video Electronics Standards. Association
  • PCI Peripheral Component Interconnect
  • Electronic device 712 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 712, including volatile and non-volatile media, removable and non-removable media.
  • System memory 728 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 730 and/or cache memory 732.
  • Electronic device 712 may further include other removable/non-removable, volatile/non-volatile computer system storage media.
  • storage system 734 can be used to read and write non-removable, non-volatile magnetic media (not shown in Figure 7, commonly referred to as a "hard disk drive”).
  • a disk drive for reading and writing to a removable non-volatile disk such as a "floppy disk”
  • a removable non-volatile disk such as a CD-ROM, DVD-ROM
  • other optical media read and write optical drive.
  • each drive can be coupled to bus 718 via one or more data medium interfaces.
  • System memory 728 can include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present application.
  • Program/utility 740 having a set (at least one) of program modules 742, which may be stored, for example, in system memory 728, including but not limited to an operating system, one or more applications, other program modules, and programs Data, each of these examples or some combination may include an implementation of a network environment.
  • Program module 742 typically performs the functions and/or methods of the embodiments described herein.
  • the electronic device 712 can also be in communication with one or more external devices 714 (eg, a keyboard, pointing device, display 724, etc.), and can also be in communication with one or more devices that enable a user to interact with the electronic device 712, and/or Any device (eg, a network card, modem, etc.) that enables the electronic device 712 to communicate with one or more other computing devices. This communication can take place via an input/output (I/O) interface 722. Moreover, the electronic device 712 can also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through the network adapter 720.
  • networks eg, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet
  • network adapter 720 communicates with other modules of electronic device 712 via bus 718. It should be understood that although not shown in the figures, other hardware and/or software modules may be utilized in conjunction with electronic device 712, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays) Of Independent Disks (RAID) systems, tape drives, and data backup storage systems.
  • RAID Redundant Arrays
  • tape drives and data backup storage systems.
  • the processing unit 716 performs various functional applications and data processing by executing programs stored in the system memory 728, such as implementing the depth camera calibration method provided by embodiments of the present invention.
  • the electronic device 712 can also include a panoramic depth camera system 750 that includes at least two depth cameras that cover the panoramic field of view for acquiring images.
  • the panoramic depth camera system 750 is coupled to the processing unit 716 and the system memory 728.
  • the depth camera included in the panoramic depth camera system 750 can acquire images under the control of the processing unit 716.
  • the panoramic depth camera system 750 can be embedded in an electronic device.
  • the one or more processors are central processors; the electronic devices are portable mobile electronic devices, such as mobile robots, drones, three-dimensional visual interactive devices (such as VR glasses or wearable helmets), or smart terminals. (such as a mobile phone or tablet).
  • portable mobile electronic devices such as mobile robots, drones, three-dimensional visual interactive devices (such as VR glasses or wearable helmets), or smart terminals. (such as a mobile phone or tablet).
  • the present embodiment provides a computer readable storage medium having stored thereon a computer program that, when executed by a processor, implements a depth camera calibration method as described in any of the embodiments of the present application.
  • the computer storage medium of the embodiments of the present invention may employ any combination of one or more computer readable mediums.
  • the computer readable medium can be a computer readable signal medium or a computer readable storage medium.
  • the computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above.
  • a computer readable storage medium can be any tangible medium that can contain or store a program, which can be used by or in connection with an instruction execution system, apparatus or device.
  • a computer readable signal medium may include a data signal that is propagated in the baseband or as part of a carrier, carrying computer readable program code. Such propagated data signals can take a variety of forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer readable signal medium can also be any computer readable medium other than a computer readable storage medium, which can transmit, propagate, or transport a program for use by or in connection with the instruction execution system, apparatus, or device. .
  • Program code embodied on a computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
  • Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including an object oriented programming language such as Java, Smalltalk, C++, and conventional A procedural programming language - such as the "C" language or a similar programming language.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (eg, using an Internet service provider) Internet connection).
  • LAN local area network
  • WAN wide area network
  • Internet service provider Internet service provider
  • modules or operations of the above-described embodiments of the present invention may be implemented by a general-purpose computing device, which may be centralized on a single computing device or distributed over a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computer device so that they may be stored in the storage device by the computing device, or they may be separately fabricated into individual integrated circuit modules, or many of them Modules or operations are made as a single integrated circuit module. Thus, the application is not limited to any specific combination of hardware and software.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)
  • Stereoscopic And Panoramic Photography (AREA)
  • Image Analysis (AREA)

Abstract

公开了一种深度相机标定方法、装置、电子设备及存储介质，其中该方法包括：控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像，每个深度相机均设置对应的标签；获取至少一个第一标签相机采集各帧图像时的位姿；若第二标签相机与第一标签相机出现历史视角重叠，根据历史视角重叠对应的图像及第一标签相机采集各帧图像时的位姿，计算第二标签相机与第一标签相机在同一时刻的相对位姿。

Description

深度相机标定方法以及装置、电子设备及存储介质
本申请要求在2018年03月05日提交中国专利局、申请号为201810179738.7的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本发明实施例涉及机器视觉技术，例如涉及一种深度相机标定方法以及装置、电子设备及存储介质。
背景技术
随着机器人导航、虚拟现实及增强现实技术的发展，RGB-D相机(即深度相机)被广泛应用于机器人导航、静态场景重建及动态人体重建等。RGB-D相机结合传统RGB相机与深度相机，具有精度高、体积小、信息量大、无源性和信息丰富等优点。
目前，由于单个RGB-D相机的视场(Field of View，FoV)有限，使用单个RGB-D相机导航无法同时获取周围全部信息，而小视场对于三维重建也有一定限制，例如动态物体重建中往往无法获取全身或多物体模型，又如无法针对视场内的动态物体与静态物体同时重建等。
若使用多相机系统，则需要对相机进行精准的外参标定。目前一般采用基于特定标定物的相机标定方法，方法相对繁琐，且要求相邻相机间的视场具有较大重叠范围，无法适应视场重叠范围很小的情况。而基于同时定位与建图(Simultaneous Localization And Mapping，SLAM)的标定方法利用相机按照预设轨迹运动以采集图像，然后离线处理图像进行标定，无法快速在线同时标定360°全景RGB-D相机。
发明内容
本发明实施例提供一种深度相机标定方法以及装置、电子设备及存储介质，以实现无需人为干预的、实时在线的全景多相机自标定。
第一方面，本发明实施例提供了一种深度相机标定方法，包括：
控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像，其中每个深度相机均设置标签；
获取至少一个第一标签相机采集各帧图像时的位姿；
若第二标签相机与所述第一标签相机出现历史视角重叠，根据历史视角重叠对应的图像及所述第一标签相机采集多个帧图像时的位姿，确定所述第二标签相机与所述第一标签相机在同一时刻的相对位姿。
第二方面，本发明实施例还提供了一种深度相机标定装置，包括：
相机控制模块，设置为控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像，其中每个深度相机均设置标签；
位姿获取模块，设置为获取至少一个第一标签相机采集多个帧图像时的位姿；
相对位姿计算模块，设置为在第二标签相机与所述第一标签相机出现历史视角重叠的情况下，根据历史视角重叠对应的图像及所述第一标签相机采集各帧图像时的位姿，确定所述第二标签相机与所述第一标签相机在同一时刻的相对位姿。
第三方面，本发明实施例还提供了一种电子设备，包括：
一个或多个处理器；
存储器，设置为存储一个或多个程序；
全景深度相机系统，包括至少两个深度相机，所述至少两个深度相机覆盖全景视场，用于采集图像；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如本申请任意实施例所述的深度相机标定方法。
第四方面，本发明实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本申请任意实施例所述的深度相机标定方法。
附图概述
下面将对实施例或相关技术描述中所需要使用的附图做一简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本发明实施例一提供的深度相机标定方法的流程图；
图2是本发明实施例一提供的全景深度相机系统的示意图；
图3是本发明实施例二提供的深度相机标定方法的流程图；
图4是本发明实施例三提供的深度相机标定方法的流程图；
图5是本发明实施例四提供的获取第一标签相机位姿的流程图；
图6是本发明实施例五提供的深度相机标定装置的结构框图；
图7是本发明实施例六提供的电子设备的结构示意图。
具体实施方式
下面结合附图和实施例对本申请作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本申请，而非对本申请的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本申请相关的部分而非全部结构。
实施例一
图1是本发明实施例一提供的深度相机标定方法的流程图，本实施例可适用于多相机自标定的情况，本实施例中的标定是指计算相机之间的相对位姿。该方法可以由深度相机标定装置或电子设备来执行，该深度相机标定装置可以通过软件和/或硬件实现，例如通过中央处理器(Central Processing Unit，CPU)实现，由CPU完成相机的控制和标定；进一步的，该装置可集成在便携式移动电子设备中。如图1所示，该方法例如可以包括：步骤S101、步骤S102和步骤S103。
在步骤S101中，控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像。
其中，每个深度相机均设置对应的标签。
其中，全景深度相机系统(可简称为相机系统)包括至少两个深度相机(RGB-D相机)，该至少两个深度相机覆盖360度全景视场。在实际应用中，可以根据具体需求选取要使用的相机尺寸和相机个数，根据相机尺寸和相机个数将多个相机固定在平台(例如刚性结构的部件)上，以满足视场覆盖需求，初步完成全景深度相机系统的搭建。在一实施例中，根据单相机的视场和需求视场确定所需相机个数，所有相机的视场之和需要大于需求视场，示例性的，以使用视角相同的单相机为例，相机个数n需满足n×Fov＞α，Fov表示单相机的视场，α表示所搭建的相机系统的需求视场，例如，取α＝360°，单相机的横向视场为65°，纵向视场为80°，可选取n＝5或n＝6，考虑到纵向视场的需求，可以使用6个相机。根据所使用的相机尺寸及相机个数对相机进行合理布局，示例性的，以长为10～15厘米，宽为3～5厘米，高为3～5厘米，分辨率为640×480的RGB-D相机为例，选取底边边长为5厘米的正六棱柱为轴，将相机镜头朝外通过固定件固定在轴上，如图2所示。需要说明的是，搭建相机系统 时，结合具体的系统使用需求，相邻相机间可以不具有视角重叠或者有较小的视角重叠，例如，一两度的视角重叠。
全景深度相机系统中的深度相机同步采集图像，同步方式可以是硬件同步，也可以是软件同步。在一实施例中，硬件同步是指利用一信号(如上升沿信号)同时触发所有相机采集同一时刻的图像；软件方式是指缓存多个相机采集的图像时，为多个图像打上时间戳，时间戳最相近的图像被认为是同一时刻采集的图像，即维持一个缓冲区域，每次输出多个相机最相近时间戳的图像帧。
在线进行相机自标定时，需使全景深度相机系统处于运动状态，以采集图像作为观测信息。例如，利用机器人搭载全景深度相机系统或用户手持全景深度相机系统在常规室内进行自由运动，并保证运动过程中尽可能增加旋转运动，以增加运动过程中相机间的历史视角重叠，得到更多观测信息，便于标定。
每个深度相机均设置对应的属性标签，该标签用于区分相机的角色，例如，第一标签表示相机是基准相机，第二标签表示相机是非基准相机，简单起见，标签具体取值可以是0或1，例如0代表非基准相机，1代表基准相机。
在步骤S102中，获取至少一个第一标签相机采集各帧图像时的位姿。
其中，可以根据相机标签确定相机角色，全景深度相机系统所包含的至少两个深度相机中至少有一个基准相机，例如可以预先设置系统中任意相机作为基准相机，第一标签相机即为基准相机。需要注意的是，若预先设置多个基准相机，则需要准确获知基准相机之间的相对位姿，考虑到标定的准确性，一般可预先设置一个基准相机。针对相机系统中的基准相机，在整个标定过程中，需要实时获取基准相机采集多个帧图像时的位姿，位姿是指基准相机采集当前帧图像时相对于采集上一帧图像时的相对位姿变化，位姿例如可以包括位置(平移矩阵T)和姿态(旋转矩阵R)，涉及六个自由度X、Y、Z、α、β和γ，其中X、Y和Z是相机位置的参数，α、β和γ是相机姿态的参数。
在步骤S103中，若第二标签相机与第一标签相机出现历史视角重叠，根据历史视角重叠对应的图像及第一标签相机采集多个帧图像时的位姿，确定第二标签相机与第一标签相机在同一时刻的相对位姿。
其中，随着全景深度相机系统的运动，每个第二标签相机都会与第一标签相机出现历史视角重叠，由此可以得到每个第二标签相机与第一标签相机在同一时刻的相对位姿，即得到了多个相机之间的相对位姿，完成了实时在线的相机自标定过程。
需要说明的是，在整个标定过程中，全景深度相机系统中待标定的深度相 机均处于运动状态，这些运动的相机实时同步采集图像，并维护多个相机下的关键帧。对于其中的基准相机，根据采集的多个帧图像进行单相机位姿估计，得到采集每个帧图像时的相机位姿；与此同时，对于其中的非基准相机，不进行单相机位姿估计，采集的多个帧图像作为观测信息，以判断是否与任一基准相机出现历史视角重叠，若重叠，则可以计算出该非基准相机与对应的基准相机在同一时刻的相对位姿。
本实施例的深度相机标定方法，基于全景深度相机系统中预先设置的第一标签相机，在全景深度相机系统边运动边采集图像的过程中，利用第二标签相机与第一标签相机的历史视角重叠，确定第二标签相机与第一标签相机的相对位姿，实现了多相机的快速在线自标定。该方法无需标定物，无需相邻相机间具有固定的较大视场重叠范围，搭建相机系统时相邻相机间存在较小的重叠视角或者不具有重叠视角，利用全景深度相机系统的运动使不同相机采集到历史视角重叠的图像，即可依此进行标定。该方法计算量小，可基于CPU进行在线标定。本发明实施例的深度相机标定方法适用于室内机器人导航或三维场景重建等应用背景。
实施例二
本实施例在上述实施例的基础上，进一步对历史视角重叠的确定以及相机间相对位姿的计算进行优化。
针对相机系统中每个待标定的深度相机，采集到一帧图像，确定该帧图像是否为关键帧，以便于确定历史视角重叠，若后续进行回环优化，也需要使用关键帧。在一实施例中，在获取至少一个第一标签相机采集各帧图像时的位姿的同时，还可以包括：分别将多个深度相机采集的当前帧图像与自身上一关键帧进行特征点匹配，得到两帧图像之间的转换关系矩阵；若转换关系矩阵大于或等于预设转换阈值，确定当前帧图像为对应深度相机下的当前关键帧，并存储该关键帧。
其中，每个深度相机采集的第一帧图像均默认是关键帧，对于后续采集的每帧图像，则通过该帧图像与该相机下最近的关键帧进行比较，来确定该帧图像是否为关键帧。预设转换阈值是根据深度相机采集图像时的运动情况提前设定的，例如，若相机拍摄相邻两帧图像时位姿变化较大，则预设转换阈值就设置大一些。特征点匹配可以采用相关的匹配算法，例如，基于快速特征点提取和描述(Oriented FAST and Rotated BRIEF，ORB)算法(稀疏算法)对RGB-D 相机采集的彩色图像进行特征匹配，或者使用直接法进行稠密配准。
图3是本发明实施例二提供的深度相机标定方法的流程图，如图3所示，该方法包括：步骤S301至步骤S306。
在步骤S301中，控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像。
其中，每个深度相机均设置对应的标签。
在步骤S302中，获取至少一个第一标签相机采集多个帧图像时的位姿。
在步骤S303中，将第二标签相机采集的当前帧图像与上述至少一个第一标签相机的历史关键帧进行特征点匹配；若存在历史关键帧与当前帧图像达到匹配阈值，确定第二标签相机与对应的第一标签相机出现历史视角重叠。
其中，第二标签相机每采集到新一帧图像，就与第一标签相机下存储的关键帧(即历史关键帧)进行特征点匹配，若达到匹配阈值，则认为出现历史视角重叠。特征点匹配可以采用相关的匹配算法，例如，基于稀疏的ORB算法对RGB-D相机采集的彩色图像进行特征匹配，或者使用直接法进行稠密配准。匹配阈值可以是预设的匹配特征点的数量。
在步骤S304中，移除第二标签相机采集的当前帧图像与对应历史关键帧的特征点对应关系中的异常数据，根据剩余的特征点对应关系计算当前帧图像与对应历史关键帧的相对位置关系。
在步骤S305中，根据相对位置关系确定第二标签相机采集当前帧图像时的位姿与第一标签相机采集对应历史关键帧时的位姿的变换关系。
在步骤S306中，根据变换关系及第一标签相机从采集对应历史关键帧到采集当前帧图像之间的多个位姿，确定第二标签相机与第一标签相机在当前帧时刻的相对位姿。
其中，可以使用随机抽样一致(Random Sample Consensus，RANSAC)算法移除异常数据，并依据剩余的特征点对应关系计算视角重叠的两帧图像间的相对位置关系；依据图像间的相对位置关系，可以求得对应的相机位姿变换关系。由于一直在估计第一标签相机采集多个帧图像时的位姿，历史视角重叠对应的历史关键帧到当前帧之间的第一标签相机位姿都是已知的，由此可以推导出现历史视角重叠的第二标签相机和第一标签相机在当前帧时刻(即出现历史视角重叠的时刻)的相对位姿。相对位姿同样包括六个自由度，涉及旋转矩阵R和平移矩阵T。
示例性的，全景深度相机系统包括三个深度相机A、B和C，其中相机A 为第一标签相机，相机B和C为第二标签相机。三个相机同步采集图像，采集到第10帧时，相机B的第10帧与相机A的第5帧(历史关键帧)出现历史视角重叠，根据这两帧历史视角重叠的图像之间的特征点匹配，可以算出相机B采集第10帧的位姿与相机A采集第5帧的位姿的变换关系。由于相机A一直在做位姿估计，记录有第5帧到第10帧的位姿，由此一帧帧推导，便可得到相机B与相机A在第10帧的相对位姿。
本实施例的深度相机标定方法，利用历史关键帧确定第二标签相机与第一标签相机出现历史视角重叠，通过视角重叠的两帧图像计算第二标签相机与第一标签相机的位姿变换关系，然后利用第一标签相机下对应的历史关键帧与当前帧之间的第一标签相机位姿，可以推导出第二标签相机与第一标签相机在同一时刻的相对位姿，计算简单，能够保证快速在线标定。
在一实施例中，为了得到更为准确的相机间的相对位姿，提高标定准确度，在计算第二标签相机与第一标签相机在同一时刻的相对位姿之后，可以对已计算出的相对位姿进行全局优化，例如可以包括：若已计算出相对位姿的深度相机同步采集的当前帧图像中有关键帧，则根据该关键帧和已计算出相对位姿的深度相机下的历史关键帧进行回环检测；若回环成功(即找到匹配的历史关键帧)，根据该关键帧及对应的历史关键帧优化更新对应深度相机间的相对位姿。
其中，回环检测是判断深度相机是否运动到曾经达到的地方或与历史视角具有较大重叠的地方。回环检测是针对关键帧率进行的，对于深度相机采集的每帧图像，需要判断该帧图像是否为关键帧，若是，则进行回环检测，若否，则等待下一关键帧到来以进行回环检测。其中，关键帧的判断，可以是将深度相机采集的每帧图像与该相机对应的上一关键帧进行特征点匹配，得到两帧图像之间的转换关系矩阵；若转换关系矩阵大于或等于预设转换阈值，则确定当前帧图像为该相机下的当前关键帧。
由于涉及相机间的相对位姿，因此本实施例中回环检测的比较对象是相机系统中已计算出相对位姿的深度相机下的历史关键帧，而非仅仅比较自身的历史关键帧，若回环成功，根据当前关键帧及对应的历史关键帧优化更新对应深度相机间的相对位姿，以减小累积误差，提高相机间相对位姿的精度。
在一实施例中，将当前关键帧与历史关键帧进行回环检测可以是将当前关键帧与历史关键帧的ORB特征点进行匹配运算，若匹配度高，则说明回环成功，在一实施例中，可以按照当前关键帧与历史关键帧的匹配程度，选出匹配度高的一个或多个历史关键帧进行对应相机间的相对位姿的优化更新。需要说明的 是，若匹配的历史关键帧属于该深度相机自身，则根据当前关键帧和历史关键帧对该深度相机的自身位姿进行优化。上述相对位姿的优化更新过程，在得到相机系统中一对相机的相对位姿之后，即可启动，随着相机运动采集图像的过程对已计算的相机间相对位姿进行更新，如果相机系统中所有相机间的相对位姿都计算完成，且当满足预设条件(例如，多个相机间的相对位姿的优化次数达到预设次数，或者满足预设误差需求)时，停止优化更新，得到最终较为准确的标定参数。
示例性的，仍以全景深度相机系统包括三个深度相机A、B和C为例，计算出相机B与相机A在同一时刻的相对位姿之后，在确定相机C与相机A是否出现历史视角重叠(以确定二者相对位姿)的同时，也根据相机A和/或相机B采集的关键帧对相机B与A之间的相对位姿进行优化更新。若得到了相机C与相机A在同一时刻的相对位姿，则可以进一步推导出相机B与相机C的相对位姿，由此得到多个相机之间的相对位姿，接下来便进行优化更新。对于当前帧时刻，三个相机同步采集对应的图像，若判断出相机A采集的图像a和相机B采集的图像b是关键帧，相机C采集的图像c不是关键帧，则对关键帧a和b进行回环检测，二者均回环成功，分别根据关键帧a和b进行优化更新，优化更新过程中相对位姿的计算与初始的相对位姿计算方法相同，此处不再赘述。
本实施例可以根据相机采集的图像实时进行已计算的相机间相对位姿的优化更新，提高相机标定的准确度。
实施例三
上述各实施例的深度相机标定方法是利全景深度相机系统中预设的第一标签相机作为基准，求取相机间的相对位姿。本实施例在上述各实施例的基础上，提供了另一种深度相机标定方法，以进一步加快在线标定速度。图4是本发明实施例三提供的深度相机标定方法的流程图，如图4所示，该方法包括：步骤S401至步骤S405。
在步骤S401中，控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像。
其中，每个深度相机均设置对应的标签。
在步骤S402中，获取至少一个第一标签相机采集多个帧图像时的位姿。
在步骤S403中，若第二标签相机与第一标签相机出现历史视角重叠，根据历史视角重叠对应的图像及第一标签相机采集多个帧图像时的位姿，确定第二 标签相机与第一标签相机在同一时刻的相对位姿。
在步骤S404中，将第二标签相机的标签修改为第一标签。
也就是说，在计算相对位姿之后，将与第一标签相机出现历史视角重叠的第二标签相机纳入基准范畴，扩大基准范围。
在步骤S405中，重复执行获取至少一个第一标签相机采集多个帧图像时的位姿、确定存在历史视角重叠的第二标签相机与对应的第一标签相机在同一时刻的相对位姿以及修改标签的操作(即重复执行S402至S404)，直到上述至少两个深度相机中不包含第二标签相机。相机系统中深度相机的标签都修改为第一标签，表示每个第二标签相机都计算得到了与其他相机的相对位姿，得到了标定结果。
本实施例中，利用历史视角重叠得到第二标签相机与第一标签相机的相对位姿之后，将该第二标签相机也作为基准，扩大基准范围，增加其他第二标签相机出现历史视角重叠的概率，从而可以进一步加快标定速度。
对于关键帧的确定、历史视角重叠的确定、相对位姿的计算、相对位姿的优化更新，请参考前述实施例的描述，此处不再赘述。
实施例四
本实施例在上述各实施例的基础上，进一步对“获取至少一个第一标签相机采集多个帧图像时的位姿”进行了优化，以提高计算速度。图5是本发明实施例四提供的获取第一标签相机位姿的流程图，如图5所示，该方法包括：步骤S501、步骤S502和步骤S503。
在步骤S501中，针对每个第一标签相机，对第一标签相机采集的每帧图像进行特征提取，得到每帧图像的至少一个特征点。
其中，对图像进行特征提取是为了找到该帧图像中一些具有标志性特征的像素点(即特征点)，例如，可以是一帧图像中的角点、纹理、边缘处的像素点。对每帧图像进行特征提取可以采用ORB算法，找到该帧图像中的至少一个特征点。
在步骤S502中，对相邻两帧图像进行特征点匹配，得到相邻两帧图像间的特征点对应关系。
考虑到相机在运动过程中采集图像的频率较快，同一相机采集的相邻两帧图像的部分内容是一样的，因此两帧图像对应的特征点之间也存在着一定的对应关系。可以采用稀疏的ORB特征配准或者直接法稠密配准，得到相邻两帧图 像间的特征点对应关系，即得到相邻两帧图像间的特征点对应关系。
在一实施例中，以相邻两帧图像间的一个特征点为例，假设两帧图像中表示同一个纹理特征的特征点X1，X2分别位于两帧图像的不同位置，以H(X1，X2)表示两个特征点X1，X2之间的汉明距离，对两特征点进行异或运算，并统计结果为1的个数，作为相邻两帧图像间的一个特征点的汉明距离(即特征点对应关系)。
在步骤S503中，移除特征点对应关系中的异常数据，通过包含剩余特征点二阶统计量的线性成分以及包含相机位姿的非线性成分计算J(ξ) TJ(ξ)中的非线性项
Figure PCTCN2019085515-appb-000001
对δ＝-(J(ξ) TJ(ξ)) -1J(ξ) Tr(ξ)进行多次迭代计算，求解重投影误差r(ξ)小于预设阈值时的位姿。具体可使用高斯牛顿法进行迭代计算。优选可以是计算重投影误差最小化时的位姿。
其中，r(ξ)表示包含所有重投影误差的向量，J(ξ)为r(ξ)的雅克比矩阵，ξ表示相机位姿的李代数，δ表示每次迭代时r(ξ)的增量值；R i表示采集第i帧图像时相机的旋转矩阵；R j表示采集第j帧图像时相机的旋转矩阵；
Figure PCTCN2019085515-appb-000002
表示第i帧图像上的第k个特征点；
Figure PCTCN2019085515-appb-000003
表示第j帧图像上的第k个特征点；C i,j表示第i帧图像与第j帧图像的特征点对应关系的集合；||C i,j||-1表示第i帧图像与第j帧图像的特征点对应关系的数量；[] ×表示向量积；||C i,j||表示取C i,j的范数。
在一实施例中，非线性项
Figure PCTCN2019085515-appb-000004
的表达式为：
Figure PCTCN2019085515-appb-000005
其中，
Figure PCTCN2019085515-appb-000006
表示线性成分；
Figure PCTCN2019085515-appb-000007
和r jl表示非线性成分，
Figure PCTCN2019085515-appb-000008
是旋转矩阵R i中的第l行，r jl是旋转矩阵R j中的第l行的转置，l＝0,1,2(本实施例基于编程思想从0开始计数，l＝0即表示通常所说的矩阵第1行，依此类推)。
在一实施例中，步骤S502中得到的相邻两帧图像间的每个特征点对应关系中会存在不合格的异常数据，例如，相邻的两帧图像中，每帧图像中一定存在 另一帧图像所没有的特征点，将它们进行步骤S502的匹配运算，就会出现异常的对应关系。在一实施例中，可以使用RANSAC算法对异常数据进行移除，剩余的特征点对应关系表示为
Figure PCTCN2019085515-appb-000009
其中，
Figure PCTCN2019085515-appb-000010
表示第i帧图像与第j帧图像间第k个特征点之间的对应关系；j＝i-1。
计算相机位姿就是求解以下式为代价函数的两帧图像之间的非线性最小二乘问题：
Figure PCTCN2019085515-appb-000011
其中，E表示欧氏空间中第i帧图像相比于第j帧图像(本实施例中指上一帧图像)的重投影误差；T i表示相机采集第i帧图像时的位姿(根据前述对相机位姿的解释可知，实际是指采集第i帧图像相对于上一帧图像的位姿变化)，T j表示相机采集第j帧图像时的位姿；N表示相机采集到的总帧数；
Figure PCTCN2019085515-appb-000012
表示第i帧图像上的第k个特征点
Figure PCTCN2019085515-appb-000013
的齐次坐标，
Figure PCTCN2019085515-appb-000014
表示第j帧图像上的第k个特征点
Figure PCTCN2019085515-appb-000015
的齐次坐标。需要说明的是，当i和k取值相同时，
Figure PCTCN2019085515-appb-000016
和
Figure PCTCN2019085515-appb-000017
表示同一个点，区别在于
Figure PCTCN2019085515-appb-000018
是本地坐标，
Figure PCTCN2019085515-appb-000019
是齐次坐标。
为了加快计算速率，本实施例并不对式(2)的代价函数进行直接计算，而是通过包含剩余特征点二阶统计量对应关系的线性成分以及包含相机位姿的非线性成分计算J(ξ) TJ(ξ)中的非线性项
Figure PCTCN2019085515-appb-000020
对δ＝-(J(ξ) TJ(ξ)) -1J(ξ) Tr(ξ)进行多次迭代计算，求解重投影误差小于预设阈值时的位姿。由非线性项
Figure PCTCN2019085515-appb-000021
的表达式可知，在进行非线性项
Figure PCTCN2019085515-appb-000022
计算时，将两帧图像间固定的线性部分
Figure PCTCN2019085515-appb-000023
看成一个整体W来进行计算，不需要按照特征点对应关系的数量进行计算，降低了相机位姿计算的复杂度，增强了相机位姿计算的实时性。
下面对式(1)的推导过程进行说明，并结合推导过程分析降低算法复杂度的原理。
欧氏空间中相机采集第i帧图像时的相机位姿T i＝[R i/t i]，实际上T i是指相机 采集第i帧图像时相对于采集第j帧图像(本实施例中指上一帧图像)时的位姿变换矩阵，包括旋转矩阵R i和平移矩阵t i。将欧氏空间中的刚性变换T i用SE3空间上的李代数ξ i来表示，即ξ i也表示相机采集第i帧图像时的相机位姿，T(ξ i)将李代数ξ i映射为欧氏空间中的T i。
对于每个特征点对应关系
Figure PCTCN2019085515-appb-000024
其重投影误差为：
Figure PCTCN2019085515-appb-000025
式(1)中欧氏空间的重投影误差可表示为E(ξ)＝||r(ξ)||，r(ξ)表示包含所有重投影误差的向量，即：
Figure PCTCN2019085515-appb-000026
Figure PCTCN2019085515-appb-000027
可以表示为(为表示简便，以下省去ξ i)：
Figure PCTCN2019085515-appb-000028
其中，
Figure PCTCN2019085515-appb-000029
表示旋转矩阵R i中的第l行；t il表示平移向量t i中的第l个元素，l＝0,1,2。
Figure PCTCN2019085515-appb-000030
其中，
Figure PCTCN2019085515-appb-000031
表示第i帧图像与第j帧图像间特征点对应关系相应的雅克比矩阵；m表示第m个特征点对应关系。
Figure PCTCN2019085515-appb-000032
Figure PCTCN2019085515-appb-000033
是一个6×6方阵，
Figure PCTCN2019085515-appb-000034
表示矩阵
Figure PCTCN2019085515-appb-000035
的转置，
Figure PCTCN2019085515-appb-000036
表达式如下：
Figure PCTCN2019085515-appb-000037
其中，I 3×3表示3×3的单位矩阵。根据式(6)和式(7)，
Figure PCTCN2019085515-appb-000038
中四个非零的6×6子矩阵为：
Figure PCTCN2019085515-appb-000039
下面以
Figure PCTCN2019085515-appb-000040
为例进行说明，其他三个非零子矩阵也类似计算，不再赘述。
Figure PCTCN2019085515-appb-000041
其中，结合式(5)可以得到：
Figure PCTCN2019085515-appb-000042
将
Figure PCTCN2019085515-appb-000043
表示为W，结合式(5)，则可将式(10)中的非线性项
Figure PCTCN2019085515-appb-000044
简化为式(1)，该非线性项中的结构项
Figure PCTCN2019085515-appb-000045
被线性为W。虽然对结构项
Figure PCTCN2019085515-appb-000046
而言，
Figure PCTCN2019085515-appb-000047
是非线性的，但经过上述分析，
Figure PCTCN2019085515-appb-000048
中的所有非零元素与C i,j中结构项的二阶统计量成线性关系，结构项的二阶统计量为
Figure PCTCN2019085515-appb-000049
和
Figure PCTCN2019085515-appb-000050
也就是说，稀疏矩阵
Figure PCTCN2019085515-appb-000051
对C i,j中结构项的二阶统计量是元素线性的。
需要说明的是，每个对应关系
Figure PCTCN2019085515-appb-000052
的雅克比矩阵均由几何项ξ i，ξ j和结构项
Figure PCTCN2019085515-appb-000053
Figure PCTCN2019085515-appb-000054
决定。对于同一帧对C i,j中的所有对应关系，其对应的雅可比矩阵共享相同的几何项，但具有不同的结构项。对于一个帧对C i,j，计算
Figure PCTCN2019085515-appb-000055
时，相关算法依赖于C i,j中特征点对应关系的数量，而本实施例可以固定的复杂度高效计算
Figure PCTCN2019085515-appb-000056
只需计算结构项的二阶统计量W，而不需要每个对应关系都将相关的 结构项去参与计算，即
Figure PCTCN2019085515-appb-000057
中四个非零子矩阵可以用复杂度O(1)代替复杂度O(||C i,j||)来计算。
因此，在δ＝-(J(ξ) TJ(ξ)) -1J(ξ) Tr(ξ)的非线性高斯牛顿最优化的迭代步骤中需要的稀疏矩阵J TJ和J Tr可以复杂度O(M)高效计算，代替原来的计算复杂度O(N coor)，N coor表示所有帧对的全部特征点对应关系的总数，M表示帧对的个数。一般的，O(N coor)在稀疏匹配中大约为300，而在稠密匹配中大约为10000，远大于帧对个数M。
经过上述推导，在相机位姿计算过程中，对于每个帧对，计算W，然后计算式(1)、(10、(9)、(8)和(6)，求取
Figure PCTCN2019085515-appb-000058
进而可以通过迭代计算，求取r(ξ)最小时的ξ。
在一实施例中，为了得到更为准确的第一标签相机位姿，在获取至少一个第一标签相机采集多个帧图像时的位姿之后，可以对获取的位姿进行全局一致的优化更新，例如可以包括：若第一标签相机采集的当前帧图像为关键帧，则根据当前关键帧和该第一标签相机的历史关键帧进行回环检测；若回环成功，根据当前关键帧对已获取的第一标签相机位姿进行全局一致的优化更新。
也就是说，对于第一标签相机采集的每帧图像，需要判断该帧图像是否为关键帧，若是，则进行回环检测，若否，则等待下一关键帧到来以进行回环检测。其中，关键帧的判断，可以是将第一标签相机采集的每帧图像与该相机对应的上一关键帧进行特征点匹配，得到两帧图像之间的转换关系矩阵；若转换关系矩阵大于或等于预设转换阈值，则确定当前帧图像为该相机下的当前关键帧。
全局一致的优化更新是指在标定过程中，随着相机的运动，当深度相机运动到曾经到达的地方或与历史视角具有较大重叠时，当前帧图像和已采集过的图像一致，而非产生交错、混叠等现象。回环检测则是依据深度相机当前观测判断该相机是否运动到曾经达到的地方或与历史视角具有较大重叠的地方，若回环成功，根据当前关键帧对第一标签相机位姿进行全局一致的优化更新，减小累积误差。
在一实施例中，将当前关键帧与历史关键帧进行回环检测可以是将当前关键帧与历史关键帧的ORB特征点进行匹配运算，若匹配度高，则说明回环成功。在一实施例中，可以按照当前关键帧与历史关键帧的匹配程度，选出匹配度高的一个或多个历史关键帧进行相机位姿的全局一致的优化更新。
在一实施例中，相机位姿全局一致的优化更新，即依据当前关键帧和匹配 度高的一个或多个历史关键帧之间的对应关系，求解以
Figure PCTCN2019085515-appb-000059
为代价函数的当前关键帧与所有匹配度高的历史关键帧间的最小化转换误差问题。其中，E(T 1,T 2,···,T N-1|T i∈SE3,i∈[1,N-1])表示所有帧对(任意一个历史匹配关键帧与当前关键帧即为一个帧对)的转换误差；N表示与当前关键帧匹配度高的历史关键帧的个数；E i,j表示第i帧与第j帧之间的转换误差，转换误差即重投影误差。
在一实施例中，在相机位姿优化更新的过程中，需要保持非关键帧和其对应的关键帧的相对位姿不变，例如优化更新算法可以使用相关的BA算法，也可以使用步骤S503中的方法以提高优化速度，具体不再赘述。同样的，对于相机间相对位姿的计算和优化过程，也可使用本实施例的算法(即S503中的方法)。
本实施例通过包含特征点二阶统计量的线性成分以及包含相机位姿的非线性成分迭代计算相机位姿，在进行非线性项
Figure PCTCN2019085515-appb-000060
计算时，将固定的线性部分
Figure PCTCN2019085515-appb-000061
看成一个整体W来进行计算，降低了相机位姿计算的复杂度，增强了相机位姿计算的实时性，对硬件要求低。将上述算法应用于求解位姿及后端优化过程中，能够得到快速并且全局一致的标定参数。
需要说明的是，本发明实施例可以基于SLAM的流程及原理实现位姿估计与优化，位姿估计由前端的视觉里程计线程实现，位姿优化由后端的回环检测及优化算法实现，例如使用相关的光束平差(BundleAdjustment，BA)算法或本实施例中的算法。
在SLAM的过程中，根据采集的图像执行以下操作：第一标签相机的位姿估计与优化，利用视角重叠计算相机间的相对位姿，对已计算出相对位姿进行优化。这些操作可以同时执行的。通过全局一致的SLAM，优化每个相机的姿态并不断更新已计算的相机之间的相对位姿，同时可以维护局部地图和全局一致的全局地图，以适应常规RGB-D相机进行室内机器人导航或三维场景重建的应用背景。SLAM中的地图是指相机在世界坐标系中的运动轨迹以及在运动轨 迹中所观测的关键帧在世界坐标系中的位置。若因物理撞击导致相机系统出现刚性形变，本实施例只需启动标定程序进行快速的重新标定，无需重新布置标定物。
实施例五
本实施例提供了一种深度相机标定装置，可以用于执行本申请任意实施例所提供的深度相机标定方法，具备执行方法相应的功能模块和有益效果。该装置可以通过硬件和/或软件的方式实现，例如通过CPU实现。未在本实施例中详尽描述的技术细节，可参见本申请任意实施例提供的深度相机标定方法。深度相机标定装置与全景深度相机系统之间需要传输控制信号和图像等，二者之间的通信方式很多，例如，通过串口、网线等有线的方式进行通信，也可以通过蓝牙、无线宽带等无线的方式进行通信。如图6所示，该装置包括：相机控制模块61、位姿获取模块62和相对位姿计算模块63。
相机控制模块61，设置为控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像，其中每个深度相机均设置对应的标签。
位姿获取模块62，设置为获取至少一个第一标签相机采集多个帧图像时的位姿。
相对位姿计算模块63，设置为在第二标签相机与第一标签相机出现历史视角重叠的情况下，根据历史视角重叠对应的图像及第一标签相机采集多个帧图像时的位姿，确定第二标签相机与第一标签相机在同一时刻的相对位姿。
在一实施例中，上述装置还可以包括：标签修改模块和操作执行模块。
标签修改模块，设置为在相对位姿计算模块63计算第二标签相机与第一标签相机在同一时刻的相对位姿之后，将第二标签相机的标签修改为第一标签；
操作执行模块，设置为重复执行获取至少一个第一标签相机采集多个帧图像时的位姿、确定存在历史视角重叠的第二标签相机与对应的第一标签相机在同一时刻的相对位姿以及修改标签的操作(即重复执行位姿获取模块62、相对位姿计算模块63和标签修改模块的操作)，直到全景深度相机系统的至少两个深度相机中不包含第二标签相机。
在一实施例中，上述装置还可以包括：关键帧确定模块，设置为在位姿获取模块62获取至少一个第一标签相机采集多个帧图像时的位姿的同时，分别将多个深度相机采集的当前帧图像与自身上一关键帧进行特征点匹配，得到两帧图像之间的转换关系矩阵；若该转换关系矩阵大于或等于预设转换阈值，确定 当前帧图像为对应深度相机下的当前关键帧，并存储该关键帧，具体是存储该关键帧及其所属的深度相机。
在一实施例中，上述装置还可以包括：视角重叠确定模块，设置为在确定第二标签相机与第一标签相机在同一时刻的相对位姿之前，将第二标签相机采集的当前帧图像与上述至少一个第一标签相机的历史关键帧进行特征点匹配；若存在历史关键帧与当前帧图像达到匹配阈值，确定第二标签相机与对应的第一标签相机出现历史视角重叠。
在一实施例中，相对位姿计算模块63包括：相对位置关系计算单元、变换关系计算单元和相对位姿计算单元。
相对位置关系计算单元，设置为移除第二标签相机采集的当前帧图像与对应历史关键帧的特征点对应关系中的异常数据，根据剩余的特征点对应关系计算当前帧图像与对应历史关键帧的相对位置关系；
变换关系计算单元，设置为根据上述相对位置关系计算第二标签相机采集当前帧图像时的位姿与第一标签相机采集对应历史关键帧时的位姿的变换关系；
相对位姿计算单元，设置为根据上述变换关系及第一标签相机从采集对应历史关键帧到采集当前帧图像之间的多个位姿，确定第二标签相机与第一标签相机在当前帧时刻的相对位姿。
在一实施例中，位姿获取模块62包括：特征提取单元、特征匹配单元和迭代计算单元。
特征提取单元，设置为针对每个第一标签相机，对该第一标签相机采集的每帧图像进行特征提取，得到每帧图像的至少一个特征点；
特征匹配单元，设置为对相邻两帧图像进行特征点匹配，得到相邻两帧图像间的特征点对应关系；
迭代计算单元，设置为移除特征点对应关系中的异常数据，通过包含剩余特征点二阶统计量的线性成分以及包含相机位姿的非线性成分计算J(ξ) TJ(ξ)中的非线性项
Figure PCTCN2019085515-appb-000062
对δ＝-(J(ξ) TJ(ξ)) -1J(ξ) Tr(ξ)进行多次迭代计算，求解重投影误差小于预设阈值时的位姿；
其中，r(ξ)表示包含所有重投影误差的向量，J(ξ)为r(ξ)的雅克比矩阵，ξ表示相机位姿的李代数，δ表示每次迭代时r(ξ)的增量值；R i表示采集第i帧图像时相机的旋转矩阵；R j表示采集第j帧图像时相机的旋转矩阵；
Figure PCTCN2019085515-appb-000063
表示第i帧图像 上的第k个特征点；
Figure PCTCN2019085515-appb-000064
表示第j帧图像上的第k个特征点；C i,j表示第i帧图像与第j帧图像的特征点对应关系的集合；||C i,j||-1表示第i帧图像与第j帧图像的特征点对应关系的数量；[] ×表示向量积；||C i,j||表示取C i,j的范数。
在一实施例中，非线性项
Figure PCTCN2019085515-appb-000065
的表达式为：
Figure PCTCN2019085515-appb-000066
其中，
Figure PCTCN2019085515-appb-000067
表示线性成分；
Figure PCTCN2019085515-appb-000068
和r jl表示非线性成分，
Figure PCTCN2019085515-appb-000069
是旋转矩阵R i中的第l行，r jl是旋转矩阵R j中的第l行的转置，l＝0,1,2。
在一实施例中，上述装置还可以包括：回环检测模块与优化更新模块。
回环检测模块，设置为在获取至少一个第一标签相机采集多个帧图像时的位姿之后，若第一标签相机采集的当前帧图像为关键帧，则根据当前关键帧和该第一标签相机的历史关键帧进行回环检测；
优化更新模块，设置为在回环成功的情况下，根据当前关键帧对已获取的第一标签相机位姿进行全局一致的优化更新。
在一实施例中，上述回环检测模块，还设置为在相对位姿计算模块63计算第二标签相机与第一标签相机在同一时刻的相对位姿之后，若已计算出相对位姿的深度相机同步采集的当前帧图像中有关键帧，则根据该关键帧和已计算出相对位姿的深度相机下的历史关键帧进行回环检测；上述优化更新模块，还设置为在回环成功的情况下，根据该关键帧及对应的历史关键帧优化更新对应深度相机间的相对位姿。
值得注意的是，上述深度相机标定装置的实施例中，所包括的各个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分，并不用于限制本申请的保护范围。
实施例六
本实施例提供了一种电子设备，包括：一个或多个处理器，存储器和全景深度相机系统。其中，存储器，设置为存储一个或多个程序。全景深度相机系 统，包括至少两个深度相机，该至少两个深度相机覆盖全景视场，设置为采集图像。当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如本申请任意实施例所述的深度相机标定方法。
图7是本发明实施例六提供的电子设备的结构示意图。图7示出了适于用来实现本发明实施方式的示例性电子设备的框图。图7显示的电子设备712仅仅是一个示例，不应对本发明实施例的功能和使用范围带来任何限制。
如图7所示，电子设备712以通用计算设备的形式表现。电子设备712的组件可以包括但不限于：一个或多个处理器(或称为处理单元716)，系统存储器728，连接不同系统组件(包括系统存储器728和处理单元716)的总线718。
总线718表示几类总线结构中的一种或多种，包括存储器总线或者存储器控制器，外围总线，图形加速端口，处理器或者使用多种总线结构中的任意总线结构的局域总线。举例来说，这些体系结构包括但不限于工业标准体系结构(Industry Standard Architecture，ISA)总线，微通道体系结构(Micro Channel Architecture，MAC)总线，增强型ISA总线、视频电子标准协会(Video Electronics Standards Association，VESA)局域总线以及外围组件互连(Peripheral Component Interconnect，PCI)总线。
电子设备712典型地包括多种计算机系统可读介质。这些介质可以是任何能够被电子设备712访问的可用介质，包括易失性和非易失性介质，可移动的和不可移动的介质。
系统存储器728可以包括易失性存储器形式的计算机系统可读介质，例如随机存取存储器(Random-Access Memory，RAM)730和/或高速缓存存储器732。电子设备712可以进一步包括其它可移动/不可移动的、易失性/非易失性计算机系统存储介质。仅作为举例，存储系统734可以用于读写不可移动的、非易失性磁介质(图7未显示，通常称为“硬盘驱动器”)。尽管图7中未示出，可以提供用于对可移动非易失性磁盘(例如“软盘”)读写的磁盘驱动器，以及对可移动非易失性光盘(例如CD-ROM,DVD-ROM或者其它光介质)读写的光盘驱动器。在这些情况下，每个驱动器可以通过一个或者多个数据介质接口与总线718相连。系统存储器728可以包括至少一个程序产品，该程序产品具有一组(例如至少一个)程序模块，这些程序模块被配置以执行本申请各实施例的功能。
具有一组(至少一个)程序模块742的程序/实用工具740，可以存储在例如系统存储器728中，这样的程序模块742包括但不限于操作系统、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合 中可能包括网络环境的实现。程序模块742通常执行本申请所描述的实施例中的功能和/或方法。
电子设备712也可以与一个或多个外部设备714(例如键盘、指向设备、显示器724等)通信，还可与一个或者多个使得用户能与该电子设备712交互的设备通信，和/或与使得该电子设备712能与一个或多个其它计算设备进行通信的任何设备(例如网卡，调制解调器等等)通信。这种通信可以通过输入/输出(I/O)接口722进行。并且，电子设备712还可以通过网络适配器720与一个或者多个网络(例如局域网(Local Area Network，LAN)，广域网(Wide Area Network，WAN)和/或公共网络，例如因特网)通信。如图所示，网络适配器720通过总线718与电子设备712的其它模块通信。应当明白，尽管图中未示出，可以结合电子设备712使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、磁盘阵列(Redundant Arrays of Independent Disks，RAID)系统、磁带驱动器以及数据备份存储系统等。
处理单元716通过运行存储在系统存储器728中的程序，从而执行各种功能应用以及数据处理，例如实现本发明实施例所提供的深度相机标定方法。
电子设备712还可以包括：全景深度相机系统750，包含至少两个深度相机，该至少两个深度相机覆盖全景视场，用于采集图像。全景深度相机系统750与处理单元716及系统存储器728连接。全景深度相机系统750所包含的深度相机可在处理单元716的控制下采集图像。在一实施例中，全景深度相机系统750可嵌入式安装在电子设备中。
在一实施例中，一个或多个处理器为中央处理器；电子设备为便携式移动电子设备，例如移动机器人、无人机、三维视觉交互设备(如VR眼镜或可戴式头盔)或智能终端(如手机或平板电脑)等。
实施例七
本实施例提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本申请任意实施例所述的深度相机标定方法。
本发明实施例的计算机存储介质，可以采用一个或多个计算机可读的介质的任意组合。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机存取存储器(RAM)、只读存储器(Read-Only  Memory，ROM)、可擦式可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)或闪存、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本文件中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。
计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括——但不限于无线、电线、光缆、RF等等，或者上述的任意合适的组合。
可以以一种或多种程序设计语言或其组合来编写用于执行本申请操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如”C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
上述实施例序号仅仅为了描述，不代表实施例的优劣。
本领域普通技术人员应该明白，上述的本发明实施例的各模块或各操作可以用通用的计算装置来实现，它们可以集中在单个计算装置上，或者分布在多个计算装置所组成的网络上，可选地，他们可以用计算机装置可执行的程序代码来实现，从而可以将它们存储在存储装置中由计算装置来执行，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或操作制作成单个集成电路模块来实现。这样，本申请不限制于任何特定的硬件和软件的结合。
本说明书中的各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间的相同或相似的部分互相参见即可。

Claims (15)

  1. 一种深度相机标定方法，包括：
    控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像，其中每个深度相机均设置有标签；
    获取所述至少两个深度相机中的至少一个第一标签相机采集多个帧图像时的位姿；
    在所述至少两个深度相机中的第二标签相机与所述第一标签相机出现历史视角重叠的情况下，根据历史视角重叠对应的图像及所述第一标签相机采集多个帧图像时的位姿，确定所述第二标签相机与所述第一标签相机在同一时刻的相对位姿。
  2. 根据权利要求1所述的方法，在确定所述第二标签相机与所述第一标签相机在同一时刻的相对位姿之后，还包括：
    将所述第二标签相机的标签修改为第一标签；
    重复执行获取至少一个第一标签相机采集多个帧图像时的位姿、确定存在历史视角重叠的第二标签相机与第一标签相机在同一时刻的相对位姿以及修改标签的操作，直到所述至少两个深度相机中不包含第二标签相机。
  3. 根据权利要求1所述的方法，其中，在获取所述至少两个深度相机中的至少一个第一标签相机采集多个帧图像时的位姿的同时，还包括：
    将每个深度相机采集的当前帧图像与所述每个深度相机的上一关键帧进行特征点匹配，得到所述当前帧图像和所述上一关键帧之间的转换关系矩阵；
    在所述转换关系矩阵大于或等于预设转换阈值的情况下，确定所述当前帧图像为所述每个深度相机下的当前关键帧并存储所述当前关键帧。
  4. 根据权利要求1所述的方法，在确定所述第二标签相机与所述第一标签相机在同一时刻的相对位姿之前，还包括：
    将所述第二标签相机采集的当前帧图像与所述至少一个第一标签相机的历史关键帧进行特征点匹配；
    在存在所述至少一个第一标签相机的历史关键帧与所述第二标签相机采集的当前帧图像达到匹配阈值的情况下，确定所述第二标签相机和与所述历史关键帧对应的第一标签相机出现历史视角重叠。
  5. 根据权利要求4所述的方法，其中，根据历史视角重叠对应的图像及所述第一标签相机采集多个帧图像时的位姿，确定所述第二标签相机与所述第一标签相机在同一时刻的相对位姿，包括：
    移除所述第二标签相机采集的当前帧图像与历史关键帧的特征点对应关系中的异常数据，根据剩余的特征点对应关系计算所述当前帧图像与历史关键帧的相对位置关系；
    根据所述当前帧图像与历史关键帧的相对位置关系，确定所述第二标签相机采集所述当前帧图像时的位姿与所述第一标签相机采集所述历史关键帧时的位姿的变换关系；
    根据所述变换关系及所述第一标签相机从采集所述历史关键帧到采集当前帧图像之间的多个位姿，确定所述第二标签相机与所述第一标签相机在当前帧时刻的相对位姿。
  6. 根据权利要求1所述的方法，其中，获取所述至少两个深度相机中的至少一个第一标签相机采集多个帧图像时的位姿，包括：
    针对每个第一标签相机，对所述第一标签相机采集的多帧图像分别进行特征提取，得到每帧图像的至少一个特征点；
    对所述多帧图像中的相邻两帧图像进行特征点匹配，得到所述相邻两帧图像间的特征点对应关系；
    移除所述特征点对应关系中的异常数据，通过包含剩余特征点二阶统计量的线性成分以及包含相机位姿的非线性成分计算J(ξ) TJ(ξ)中的非线性项
    Figure PCTCN2019085515-appb-100001
    对δ＝-(J(ξ) T J(ξ)) -1J(ξ) T r(ξ)进行多次迭代计算，求解重投影误差小于预设阈值时的位姿；
    其中，r(ξ)表示包含所有重投影误差的向量，J(ξ)为r(ξ)的雅克比矩阵，ξ表示相机位姿的李代数，δ表示每次迭代时r(ξ)的增量值；R i表示采集第i帧图像时相机的旋转矩阵；R j表示采集第j帧图像时相机的旋转矩阵；
    Figure PCTCN2019085515-appb-100002
    表示第i帧图像上的第k个特征点；
    Figure PCTCN2019085515-appb-100003
    表示第j帧图像上的第k个特征点；C i,j表示第i帧图像与第j帧图像的特征点对应关系的集合；||C i,j||-1表示第i帧图像与第j帧图像的特征点对应关系的数量；[] ×表示向量积；||C i,j||表示取C i,j的范数。
  7. 根据权利要求6所述的方法，其中，所述非线性项
    Figure PCTCN2019085515-appb-100004
    的表达式为：
    Figure PCTCN2019085515-appb-100005
    其中，
    Figure PCTCN2019085515-appb-100006
    表示线性成分；r il T和r jl表示非线性成分，r il T是旋转矩阵R i中的第l行，r jl是旋转矩阵R j中的第l行的转置，l＝0,1,2。
  8. 根据权利要求1或6所述的方法，在获取所述至少两个深度相机中的至少一个第一标签相机采集多个帧图像时的位姿之后，还包括：
    在所述第一标签相机采集的当前帧图像为关键帧的情况下，根据当前关键帧和所述第一标签相机的历史关键帧进行回环检测；
    在当前关键帧和所述第一标签相机的历史关键帧进行回环检测成功的情况下，根据所述当前关键帧对已获取的第一标签相机位姿进行全局一致的优化更新。
  9. 根据权利要求1或2所述的方法，在确定所述第二标签相机与所述第一标签相机在同一时刻的相对位姿之后，还包括：
    在已确定出相对位姿的深度相机同步采集的当前帧图像中有当前关键帧的情况下，根据所述当前关键帧和已确定出相对位姿的深度相机下的历史关键帧进行回环检测；
    在所述当前关键帧和已计算出相对位姿的深度相机下的历史关键帧进行回环检测成功的情况下，根据所述当前关键帧及对应的历史关键帧优化更新对应深度相机间的相对位姿。
  10. 一种深度相机标定装置，包括：
    相机控制模块，设置为控制全景深度相机系统中的至少两个深度相机在运动过程中同步采集图像，其中每个深度相机均设置标签；
    位姿获取模块，设置为获取所述至少两个深度相机中的至少一个第一标签相机采集多个帧图像时的位姿；
    相对位姿计算模块，设置为在所述至少两个深度相机中的第二标签相机与所述第一标签相机出现历史视角重叠的情况下，根据历史视角重叠对应的图像及所述第一标签相机采集各帧图像时的位姿，确定所述第二标签相机与所述第一标签相机在同一时刻的相对位姿。
  11. 根据权利要求10所述的装置，还包括：
    标签修改模块，设置为将第二标签相机的标签修改为第一标签；
    操作执行模块，设置为重复执行获取至少一个第一标签相机采集多个帧图像时的位姿、确定存在历史视角重叠的第二标签相机与第一标签相机在同一时刻的相对位姿以及修改标签的操作，直到所述至少两个深度相机中不包含第二标签相机。
  12. 根据权利要求10所述的装置，还包括：
    关键帧确定模块，设置为将每个深度相机采集的当前帧图像与所述每个相机的上一关键帧进行特征点匹配，得到所述当前帧图像和所述上一关键帧之间的转换关系矩阵；在所述转换关系矩阵大于或等于预设转换阈值的情况下，确定所述当前帧图像为所述每个深度相机下的当前关键帧，并存储所述当前关键帧。
  13. 一种电子设备，包括：
    至少一个处理器；
    存储器，设置为存储至少一个程序；
    全景深度相机系统，包括至少两个深度相机，所述至少两个深度相机覆盖全景视场，用于采集图像；
    当所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如权利要求1-9中任一项所述的深度相机标定方法。
  14. 根据权利要求13所述的电子设备，其中，所述至少一个处理器为中央处理器；所述电子设备为便携式移动电子设备。
  15. 一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如权利要求1-9中任一项所述的深度相机标定方法。
PCT/CN2019/085515 2018-03-05 2019-05-05 深度相机标定方法以及装置、电子设备及存储介质 Ceased WO2019170166A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810179738.7 2018-03-05
CN201810179738.7A CN108447097B (zh) 2018-03-05 2018-03-05 深度相机标定方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2019170166A1 true WO2019170166A1 (zh) 2019-09-12

Family

ID=63193477

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/085515 Ceased WO2019170166A1 (zh) 2018-03-05 2019-05-05 深度相机标定方法以及装置、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN108447097B (zh)
WO (1) WO2019170166A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110689585A (zh) * 2019-10-09 2020-01-14 北京百度网讯科技有限公司 多相机外参的联合标定方法、装置、设备和介质

Families Citing this family (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108447097B (zh) * 2018-03-05 2021-04-27 清华-伯克利深圳学院筹备办公室 深度相机标定方法、装置、电子设备及存储介质
CN109218562B (zh) 2018-09-07 2021-04-27 百度在线网络技术（北京）有限公司 时钟同步方法、装置、设备、存储介质和车辆
CN109242913B (zh) 2018-09-07 2020-11-10 百度在线网络技术（北京）有限公司 采集器相对参数的标定方法、装置、设备和介质
CN109360243B (zh) * 2018-09-28 2022-08-19 安徽爱观视觉科技有限公司 一种多自由度可动视觉系统的标定方法
CN109584302B (zh) * 2018-11-27 2023-12-01 北京旷视科技有限公司 相机位姿优化方法、装置、电子设备和计算机可读介质
CN111563840B (zh) * 2019-01-28 2023-09-05 北京魔门塔科技有限公司 分割模型的训练方法、装置、位姿检测方法及车载终端
CN109946680B (zh) * 2019-02-28 2021-07-09 北京旷视科技有限公司 探测系统的外参数标定方法、装置、存储介质及标定系统
CN110166714A (zh) * 2019-04-11 2019-08-23 深圳市朗驰欣创科技股份有限公司 双光融合调整方法、双光融合调整装置及双光融合设备
CN110232715B (zh) * 2019-05-08 2021-11-19 奥比中光科技集团股份有限公司 一种多深度相机自校准的方法、装置及系统
CN110132306B (zh) * 2019-05-20 2021-02-19 广州小鹏汽车科技有限公司 车辆定位误差的纠正方法及系统
CN110349249B (zh) * 2019-06-26 2021-04-06 华中科技大学 基于rgb-d数据的实时稠密重建方法及系统
CN110363821B (zh) * 2019-07-12 2021-09-28 顺丰科技有限公司 单目相机安装偏差角的获取方法、装置、相机及存储介质
CN110415286B (zh) * 2019-09-24 2020-01-17 杭州蓝芯科技有限公司 一种多飞行时间深度相机系统的外参标定方法
CN110794828A (zh) * 2019-10-08 2020-02-14 福瑞泰克智能系统有限公司 一种融合语义信息的路标定位方法
CN110866953B (zh) * 2019-10-31 2023-12-29 Oppo广东移动通信有限公司 地图构建方法及装置、定位方法及装置
CN113330487B (zh) * 2019-12-30 2025-03-28 深圳引望智能技术有限公司 参数标定方法及装置
CN114089901B (zh) 2020-07-29 2023-11-24 华为技术有限公司 一种跨设备的对象拖拽方法及设备
CN113781548B (zh) * 2020-06-10 2024-06-14 华为技术有限公司 多设备的位姿测量方法、电子设备及系统
CN114115629B (zh) 2020-08-26 2025-01-10 华为技术有限公司 一种界面显示方法及设备
CN114764298B (zh) 2020-07-29 2023-03-03 华为技术有限公司 一种跨设备的对象拖拽方法及设备
CN112115980A (zh) * 2020-08-25 2020-12-22 西北工业大学 基于光流跟踪和点线特征匹配的双目视觉里程计设计方法
CN113256731B (zh) * 2021-04-01 2025-09-02 深圳市宁创云软件技术有限公司 基于单目视觉的目标检测方法及装置
CN112802112B (zh) * 2021-04-12 2021-07-16 北京三快在线科技有限公司 视觉定位方法、装置、服务器及存储介质
CN113269876B (zh) * 2021-05-10 2024-06-21 Oppo广东移动通信有限公司 地图点坐标优化方法及装置、电子设备及存储介质
CN113870358B (zh) * 2021-09-17 2024-05-24 聚好看科技股份有限公司 一种多个3d相机联合标定的方法及设备
CN116311488A (zh) * 2021-12-21 2023-06-23 北京字跳网络技术有限公司 多视角手势识别数据采集系统、方法、装置、设备及介质
CN116310958A (zh) * 2023-02-03 2023-06-23 深圳云天励飞技术股份有限公司 标签处理方法、装置、电子设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016118352A1 (en) * 2015-01-19 2016-07-28 Aquifi, Inc. Multiple camera system with auto recalibration
CN105844624A (zh) * 2016-03-18 2016-08-10 上海欧菲智能车联科技有限公司 动态标定系统、动态标定系统中的联合优化方法及装置
CN106097300A (zh) * 2016-05-27 2016-11-09 西安交通大学 一种基于高精度运动平台的多相机标定方法
CN108447097A (zh) * 2018-03-05 2018-08-24 清华-伯克利深圳学院筹备办公室 深度相机标定方法、装置、电子设备及存储介质

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9247238B2 (en) * 2011-01-31 2016-01-26 Microsoft Technology Licensing, Llc Reducing interference between multiple infra-red depth cameras
US9338440B2 (en) * 2013-06-17 2016-05-10 Microsoft Technology Licensing, Llc User interface for three-dimensional modeling
WO2015102637A1 (en) * 2014-01-03 2015-07-09 Intel Corporation Real-time 3d reconstruction with a depth camera
US20190000564A1 (en) * 2015-12-30 2019-01-03 The Johns Hopkins University System and method for medical imaging
CN106204443A (zh) * 2016-07-01 2016-12-07 成都通甲优博科技有限责任公司 一种基于多目复用的全景无人机系统
CN106157304A (zh) * 2016-07-01 2016-11-23 成都通甲优博科技有限责任公司 一种基于多个相机的全景图拼接方法及系统
CN107025668B (zh) * 2017-03-30 2020-08-18 华南理工大学 一种基于深度相机的视觉里程计的设计方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016118352A1 (en) * 2015-01-19 2016-07-28 Aquifi, Inc. Multiple camera system with auto recalibration
CN105844624A (zh) * 2016-03-18 2016-08-10 上海欧菲智能车联科技有限公司 动态标定系统、动态标定系统中的联合优化方法及装置
CN106097300A (zh) * 2016-05-27 2016-11-09 西安交通大学 一种基于高精度运动平台的多相机标定方法
CN108447097A (zh) * 2018-03-05 2018-08-24 清华-伯克利深圳学院筹备办公室 深度相机标定方法、装置、电子设备及存储介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110689585A (zh) * 2019-10-09 2020-01-14 北京百度网讯科技有限公司 多相机外参的联合标定方法、装置、设备和介质
CN110689585B (zh) * 2019-10-09 2022-06-21 阿波罗智能技术(北京)有限公司 多相机外参的联合标定方法、装置、设备和介质
US11394872B2 (en) 2019-10-09 2022-07-19 Apollo Intelligent Driving Technology (Beijing) Co., Ltd. Method and apparatus for jointly calibrating external parameters of multiple cameras, device and medium

Also Published As

Publication number Publication date
CN108447097B (zh) 2021-04-27
CN108447097A (zh) 2018-08-24

Similar Documents

Publication Publication Date Title
CN108447097B (zh) 深度相机标定方法、装置、电子设备及存储介质
CN110322500B (zh) 即时定位与地图构建的优化方法及装置、介质和电子设备
CN110866496B (zh) 基于深度图像的机器人定位与建图方法和装置
CN109084746B (zh) 用于具有辅助传感器的自主平台引导系统的单目模式
CN108537876B (zh) 三维重建方法、装置、设备及存储介质
CN107747941B (zh) 一种双目视觉定位方法、装置及系统
CN111709973B (zh) 目标跟踪方法、装置、设备及存储介质
CN107888828B (zh) 空间定位方法及装置、电子设备、以及存储介质
CN108898630A (zh) 一种三维重建方法、装置、设备和存储介质
CN108805979B (zh) 一种动态模型三维重建方法、装置、设备和存储介质
CN110111388B (zh) 三维物体位姿参数估计方法及视觉设备
WO2019205852A1 (zh) 确定图像捕捉设备的位姿的方法、装置及其存储介质
CN113034652A (zh) 虚拟形象驱动方法、装置、设备及存储介质
CN109461208B (zh) 三维地图处理方法、装置、介质和计算设备
CN110246147A (zh) 视觉惯性里程计方法、视觉惯性里程计装置及移动设备
CN111899276A (zh) 一种基于双目事件相机的slam方法及系统
WO2021004416A1 (zh) 一种基于视觉信标建立信标地图的方法、装置
WO2022156755A1 (zh) 一种室内定位方法、装置、设备和计算机可读存储介质
CN110349212B (zh) 即时定位与地图构建的优化方法及装置、介质和电子设备
WO2021093679A1 (zh) 视觉定位方法和装置
CN114659523B (zh) 一种大范围高精度姿态测量方法及装置
TW202238449A (zh) 室內定位系統及室內定位方法
CN112446347B (zh) 人脸方向确定方法和装置、存储介质、电子设备
CN112184766B (zh) 一种对象的跟踪方法、装置、计算机设备和存储介质
WO2023101662A1 (en) Methods and systems for implementing visual-inertial odometry based on parallel simd processing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19763796

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19763796

Country of ref document: EP

Kind code of ref document: A1