CN116945166A - An optimization method for robot re-grasp based on tactile element slip feature feedback - Google Patents
An optimization method for robot re-grasp based on tactile element slip feature feedback Download PDFInfo
- Publication number
- CN116945166A CN116945166A CN202310774480.6A CN202310774480A CN116945166A CN 116945166 A CN116945166 A CN 116945166A CN 202310774480 A CN202310774480 A CN 202310774480A CN 116945166 A CN116945166 A CN 116945166A
- Authority
- CN
- China
- Prior art keywords
- tactile
- contact
- grasping
- ellipse
- slip
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/02—Sensing devices
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1602—Programme controls characterised by the control system, structure, architecture
- B25J9/161—Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1628—Programme controls characterised by the control loop
- B25J9/1633—Programme controls characterised by the control loop compliant, force, torque control, e.g. combined with position control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1661—Programme controls characterised by programming, planning systems for manipulators characterised by task planning, object-oriented languages
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Manipulator (AREA)
Abstract
Description
技术领域Technical field
本发明属于机器人智能抓取技术领域,涉及一种基于触觉图元滑移特征反馈的机器人再抓取优化方法。The invention belongs to the field of robot intelligent grasping technology, and relates to a robot re-grasp optimization method based on tactile element slip feature feedback.
背景技术Background technique
基于视觉的抓取检测方法通常依赖物体的尺寸、纹理、颜色等信息来获取抓取位姿,但无法感知被抓取物体的接触表面特性、质心位姿变化信息以及接触力的变化,以及抓取过程中物体与夹爪相对运动状态等信息,无法对现有的抓取配置做出评估及反馈。因此,当发生不稳定抓取现象时,需要对视觉推理出的抓取位姿进行进一步优化,以满足稳定性、安全性和效率性等方面的要求。Vision-based grasping detection methods usually rely on the size, texture, color and other information of the object to obtain the grasping pose, but cannot perceive the contact surface characteristics, center of mass pose change information and contact force changes of the grasped object, as well as the grasping Information such as the relative motion status of the object and the gripper during the picking process makes it impossible to evaluate and provide feedback on the existing grabbing configuration. Therefore, when unstable grasping occurs, the grasping pose derived from visual reasoning needs to be further optimized to meet the requirements of stability, safety, and efficiency.
发明内容Contents of the invention
为了解决现有技术中存在的上述技术问题,本发明提出了一种基于触觉图元滑移特征反馈的机器人再抓取优化方法,基于Canny边缘检测和最小二乘的椭圆拟合,实现了光学式触觉传感器感知的触觉图元的提取,并进一步确定了触觉图元滑移特征,依照物体在试抓取中发生滑移的剧烈程度将其分为三种类型:初期滑移、局部滑移和全面滑移,通过结合的视觉抓取检测地图,触觉反馈特征和视触融合质量分析模型方法制定对应的自适应优化调整策略,以稳定重构抓取位姿,其具体技术方案如下:In order to solve the above technical problems existing in the prior art, the present invention proposes a robot re-grasp optimization method based on tactile element slip feature feedback, based on Canny edge detection and least squares ellipse fitting, to achieve optical Extract the tactile primitives sensed by the tactile sensor, and further determine the slip characteristics of the tactile primitives. According to the severity of the slip of the object during the trial grasp, it is divided into three types: initial slip and local slip. and comprehensive slipping. By combining the visual grasping detection map, tactile feedback features and visual-touch fusion quality analysis model method, a corresponding adaptive optimization adjustment strategy is developed to stably reconstruct the grasping pose. The specific technical solution is as follows:
一种基于触觉图元滑移特征反馈的机器人再抓取优化方法,包括以下步骤:A robot re-grasp optimization method based on tactile element slip feature feedback, including the following steps:
步骤1,利用设置在机械臂上的光学式触觉传感器获取触觉模态信息,即感知获取到接触区域的触觉图元;Step 1: Use the optical tactile sensor installed on the robotic arm to obtain tactile modal information, that is, sense and acquire the tactile primitives in the contact area;
步骤2:基于获取的触觉模态信息,进行椭圆参数拟合;Step 2: Perform ellipse parameter fitting based on the acquired tactile modal information;
步骤3:构建机械臂的试抓取坐标系;Step 3: Construct the trial grasping coordinate system of the robotic arm;
步骤4:对物体进行试抓取使物体发生滑移,解算物体的触觉感知滑动矢量,推测物体在滑移期间的位姿变化;Step 4: Trial grasp the object to cause the object to slip, calculate the tactile sensing sliding vector of the object, and infer the posture changes of the object during the sliding period;
步骤5:对物体在试抓取过程中发生的不同剧烈程度的滑移进行对应的优化调整,重构机械臂的再抓取位姿。Step 5: Make corresponding optimization adjustments to the different degrees of slippage that occur during the trial grasping process of the object, and reconstruct the re-grasping posture of the robotic arm.
进一步的,所述步骤1,具体为:利用基于视觉的触觉传感器通过接触图像的变化,判断机械臂夹爪是否与物体发生接触,若是则获取到传感器接触图像,若否则获取到无接触触觉初始图像;获取物体接触的面积、位置、方向;通过连续的视频图像获取接触部分在传感器表面运动状况;Further, the step 1 is specifically: using a vision-based tactile sensor to determine whether the gripper of the robotic arm is in contact with the object through changes in the contact image. If so, obtain the sensor contact image; if not, obtain the non-contact tactile initial state. Image; obtain the area, position, and direction of object contact; obtain the movement of the contact part on the sensor surface through continuous video images;
其中,所述触觉传感器通过单点和多点特征估计物体的接触区域,并通过几何特征的提取和匹配来实现对接触区域的感知,从而获取到与初始图像的差分图;Wherein, the tactile sensor estimates the contact area of the object through single-point and multi-point features, and realizes the perception of the contact area through the extraction and matching of geometric features, thereby obtaining a difference map from the initial image;
利用弹性体材料和表面微凸的构型在成像图像上表现为椭圆形接触区域或者圆形接触区域的特性,通过触觉感知模态椭圆图元提取描述接触区域,并将接触区域统一表达成A(p,a,b,θ)的格式;其中,p(x,y)为触觉传感器图像坐标系下接触椭圆的中心坐标,a为接触椭圆的半长轴长,b为接触椭圆的半短轴长,θ为接触椭圆的偏转角度,以椭圆的长轴为参考,其值为长轴偏转水平方向的角度,值域为[-π/2,π/2];当接触区域的形状为圆形时,长轴长与短轴长相等即a=b,θ为[-π/2,π/2]内任意值。The characteristics of the elastomer material and the slightly convex surface are used to show the characteristics of an elliptical contact area or a circular contact area on the imaging image. The contact area is described through the tactile sensing modal elliptical primitive extraction, and the contact area is uniformly expressed as A The format of (p, a, b, θ); where, p (x, y) is the center coordinate of the contact ellipse in the tactile sensor image coordinate system, a is the semi-major axis length of the contact ellipse, and b is the semi-minor axis of the contact ellipse. Axis length, θ is the deflection angle of the contact ellipse, taking the long axis of the ellipse as a reference, its value is the angle of the long axis deflection in the horizontal direction, the value range is [-π/2, π/2]; when the shape of the contact area is In the case of a circle, the length of the major axis and the length of the minor axis are equal, that is, a=b, and θ is any value within [-π/2, π/2].
进一步的,所述步骤2,具体包括:Further, step 2 specifically includes:
步骤2.1:对差分图进行灰度和形态学处理后使用Canny边缘检测算子对接触区域进行边缘检测提取;Step 2.1: After performing grayscale and morphological processing on the difference image, use the Canny edge detection operator to perform edge detection and extraction on the contact area;
步骤2.2:使用基于最小二乘法的几何椭圆拟合方式对检测提取到的边缘图像进行拟合。Step 2.2: Use the geometric ellipse fitting method based on the least squares method to fit the detected edge image.
进一步的,所述步骤3,具体包括:Further, step 3 specifically includes:
步骤3.1:以当前时刻由抓取检测算法推理出的抓取配置地图G(x,y,z,rx,ry,rz,W)为基准,以抓取中心坐标为原点建立抓取坐标系o-xyz;Step 3.1: Based on the current grab configuration map G (x, y, z, r x , r y , r z , W) deduced by the grab detection algorithm, establish a grab with the grab center coordinates as the origin. Coordinate system o-xyz;
步骤3.2:所述机械臂夹爪的工作表面安装两个面对面的触觉传感器,则对通过触觉传感器获取的两幅接触图像的坐标系重新定义,以右侧传感器成像为基准,以320×240成像区域中心为坐标系中心建立右图像坐标系o-xz,左侧传感器成像按照o-xyz坐标系的zox平面为镜像,以图像中心为中心建立左图像坐标系o-xz,左、右图像产生的接触区域分别标记为A1、A2,对应的中心分别为p1(x1,z1)、p2(x2,z2);其中,p(x,z)的坐标值为触觉图像空间到真实接触表面尺寸的映射值。Step 3.2: Install two face-to-face tactile sensors on the working surface of the manipulator's gripper. Then redefine the coordinate system of the two contact images obtained through the tactile sensors. Take the right sensor imaging as the benchmark and 320×240 imaging. The center of the area is the center of the coordinate system to establish the right image coordinate system o-xz. The left sensor imaging is mirrored according to the zox plane of the o-xyz coordinate system. The left image coordinate system o-xz is established with the image center as the center. The left and right images are generated. The contact areas are marked A 1 and A 2 respectively, and the corresponding centers are p 1 (x 1 , z 1 ) and p 2 (x 2 , z 2 ) respectively; among them, the coordinate value of p (x, z) is the tactile Mapping value from image space to real contact surface dimensions.
进一步的,所述步骤4,具体包括:Further, step 4 specifically includes:
步骤4.1:将初始状态统一描述成偏差的形式;物体的初始偏差包括初始位置偏差(Δx,Δy,Δz)和初始角度偏差(Δrx,Δry,Δrz),其中Δy由相机视野中获得,其大小表示物体偏离抓取坐标系o-xyz的zox平面距离;初始时刻的位姿偏差如下式所示:Step 4.1: Unify the initial state into the form of deviation; the initial deviation of the object includes the initial position deviation (Δx, Δy, Δz) and the initial angle deviation (Δr x , Δr y , Δr z ), where Δy is obtained from the camera field of view. , its size represents the zox plane distance of the object from the grasping coordinate system o-xyz; the pose deviation at the initial moment is as follows:
步骤4.2:通过读取触觉图像的视频流,获取接触椭圆的运动特征;Step 4.2: Obtain the motion characteristics of the contact ellipse by reading the video stream of the tactile image;
步骤4.3:对椭圆参数进行滤波,再描述椭圆状态矢量,最终确定物体在滑移期间的位姿变化:Step 4.3: Filter the ellipse parameters, then describe the ellipse state vector, and finally determine the posture change of the object during the sliding period:
进一步的,所述步骤5,具体包括:Further, step 5 specifically includes:
步骤5.1:对物体在试抓取过程中发生的初期滑移进行优化调整;Step 5.1: Optimize and adjust the initial slip of the object during the trial grasping process;
步骤5.2:对物体在试抓取过程中发生的局部滑移进行优化调整;Step 5.2: Optimize and adjust the local slip that occurs during the grasping process of the object;
步骤5.3:对物体在试抓取过程中发生的全面滑移进行优化调整。Step 5.3: Optimize and adjust the overall slippage of the object during the trial grasping process.
进一步的,所述步骤5.1,具体为:按照初期滑移收敛且有界的特点,进行以下优化调整:Further, step 5.1 is specifically as follows: According to the characteristics of initial slip convergence and bounding, the following optimization adjustments are made:
(1)期望接触区域的中心位置在传感器的感受范围之内,若在刚接触时接触检测的拟合椭圆中心位于传感器的感受范围之外,则为发生全面滑移,在调整时使接触区域中心沿着靠近传感器感受范围中心方向进行;(1) It is expected that the center position of the contact area is within the sensing range of the sensor. If the center of the fitting ellipse for contact detection is outside the sensing range of the sensor at the time of initial contact, full slip has occurred, and the contact area will be adjusted during adjustment. The center is along the direction close to the center of the sensor's sensing range;
(2)对于小幅度的初期滑移运动即拟合椭圆中心位于传感器的感受范围内,只需使末端执行器的位姿沿物体位姿变化的反方向进行调整即可,特别地,由于重力的影响,物体天然具有沿z轴负方向运动的趋势,对于z轴方向上的变动采用缩放处理,采用的缩放因子为0.6,即将抓取配置地图G(x,y,z,rx,ry,rz,W)调整为:(2) For a small initial sliding motion, that is, the center of the fitting ellipse is within the sensing range of the sensor, it is only necessary to adjust the posture of the end effector in the opposite direction of the object's posture change. In particular, due to gravity Influence of y , r z , W) are adjusted to:
G′(x-dx,y-dy,z-dz×0.6,rx-drx,ry-dry,rz-drz,W)。G′(x-dx,y-dy,z-dz×0.6,r x -dr x ,r y -dr y ,r z -dr z ,W).
进一步的,所述步骤5.2,具体为:对照当前物体的抓取位置地图,在抓取位置地图非零位置上沿当前抓取坐标系的x轴靠近物体重心的方向搜索,根据该区域附近置信度最高点的点重新构建抓取构型,通过空间映射函数变化f获取新的抓取配置,搜索的方向由两个触觉传感器确定,若物体的dry为正值,则沿x轴负方向搜索,若dry为负值,则沿x轴正方向搜索。Further, the step 5.2 is specifically as follows: compare with the grasping position map of the current object, search at the non-zero position of the grasping position map along the x-axis of the current grasping coordinate system in the direction close to the center of gravity of the object, and based on the confidence near the area The grasping configuration is reconstructed from the point with the highest degree, and the new grasping configuration is obtained by changing the spatial mapping function f. The direction of the search is determined by the two tactile sensors. If the dr y of the object is a positive value, then along the negative direction of the x-axis Search, if dr y is a negative value, search along the positive direction of the x-axis.
进一步的,所述步骤5.3,具体为:若当前的抓取配置发生全面滑移,则舍弃当前抓取区域,更换抓取配置地图上的抓取点,舍弃当前抓取位置地图上置信度的峰值,寻找另一个dry超过阈值的局部峰值,然后从抓取地图上生成一个新的坐标,并以此推断新的抓取构型,得到更新后的抓取配置;最后通过多次抓取质量分析和局部调整,将全面滑移问题转化为局部滑移或者初期滑移问题,从而在新的抓取配置附近调整抓取姿态直至达到稳定抓取的状态。Further, step 5.3 is specifically as follows: if the current crawling configuration completely slips, the current crawling area is discarded, the crawling point on the crawling configuration map is replaced, and the confidence level on the current crawling location map is discarded. Peak, find another local peak where dr y exceeds the threshold, then generate a new coordinate from the crawl map, and use this to infer the new crawl configuration to obtain the updated crawl configuration; finally, through multiple crawls Quality analysis and local adjustment convert the overall slip problem into a local slip or initial slip problem, thereby adjusting the grasping posture near the new grasping configuration until a stable grasping state is achieved.
有益效果:Beneficial effects:
本发明针对光学式传感器的感知机理和成像特点,通过形态学方法描述触觉接触图元特征,基于边缘检测和几何特征完成接触椭圆检测;依照连续动作的视觉输入图像和触觉感应图像对被抓取物体的稳定性做出判断的同时,利用该连续动作信息评估该物体在机器人夹臂的不稳定运动趋势;Aiming at the sensing mechanism and imaging characteristics of optical sensors, this invention describes the characteristics of tactile contact elements through morphological methods, and completes contact ellipse detection based on edge detection and geometric features; the pairs of visual input images and tactile sensing images are captured according to continuous actions. While making judgments on the stability of the object, the continuous motion information is used to evaluate the unstable movement trend of the object in the robot clamping arm;
本发明针对不同程度和种类滑动趋势,结合基于视觉的抓取地图的全局信息和基于触觉的局部位姿感知信息,使该不稳定的趋势得到有效的抑制,从而再抓取时可以获得更加有效的抓取配置,进而使机器人实现更加稳定、安全的抓取任务;This invention aims at different degrees and types of sliding trends, and combines the global information of the vision-based grasping map and the local posture perception information based on touch, so that the unstable trend can be effectively suppressed, so that the grab can be more effective again. The grabbing configuration enables the robot to achieve more stable and safer grabbing tasks;
本发明解决了现有抓取调整方法依赖于试错,无法充分利用触觉模态信息的问题。The present invention solves the problem that existing grasping adjustment methods rely on trial and error and cannot fully utilize tactile modal information.
附图说明Description of the drawings
图1为本发明的一种基于触觉图元滑移特征反馈的机器人再抓取优化方法的流程图;Figure 1 is a flow chart of a robot re-grasp optimization method based on tactile primitive slip feature feedback according to the present invention;
图2为接触区域图元拟合图;Figure 2 is a fitting diagram of contact area primitives;
图3为抓取坐标系及触觉图像坐标系图;Figure 3 shows the grasping coordinate system and the tactile image coordinate system;
图4为视频流两帧图像之间接触椭圆的滑动特征示意图;Figure 4 is a schematic diagram of the sliding characteristics of the contact ellipse between two frames of the video stream;
图5为本发明的对物体在试抓取中发生的不同剧烈程度的滑移进行对应的优化调整的具体流程示意图;Figure 5 is a schematic diagram of the specific process of optimizing and adjusting the different degrees of slippage of an object during trial grasping according to the present invention;
图6为是机械臂抓取杆状物体的受力分析示意图;Figure 6 is a schematic diagram of the force analysis of the robotic arm grabbing a rod-shaped object;
图7为本发明实施例进行抓取实验时所用的物体模型示意图;Figure 7 is a schematic diagram of the object model used in grasping experiments according to the embodiment of the present invention;
图8为本发明实施例的再抓取仿真实验流程图;Figure 8 is a flow chart of the re-grasp simulation experiment according to the embodiment of the present invention;
图9为本发明实施了的抓取预测及优化后的抓取位姿图;Figure 9 is a grasping prediction and optimized grasping posture diagram implemented in the present invention;
具体实施方式Detailed ways
为了使本发明的目的、技术方案和技术效果更加清楚明白,以下结合说明书附图和实施例,对本发明作进一步详细说明。In order to make the purpose, technical solution and technical effect of the present invention more clear, the present invention will be further described in detail below with reference to the drawings and examples of the description.
如图1所示,本发明提出的一种基于触觉图元滑移特征反馈的机器人再抓取优化方法,通过对触觉图元的提取、分析实现抓取状态的感知,进而实现在抓取策略的优化,可应用于常规的六轴工业机械臂,机械臂上设置光学式触觉传感器,该方法具体实施步骤如下:As shown in Figure 1, the invention proposes a robot re-grasp optimization method based on the feedback of tactile primitives' slip characteristics, which realizes the perception of the grasping state through the extraction and analysis of tactile primitives, and then realizes the grasping strategy. The optimization can be applied to a conventional six-axis industrial robot arm. An optical tactile sensor is installed on the robot arm. The specific implementation steps of this method are as follows:
步骤1:使用机械臂进行物体抓取时,利用光学式触觉传感器获取触觉模态信息,即感知获取到接触区域的触觉图元。Step 1: When using a robotic arm to grab an object, use an optical tactile sensor to obtain tactile modal information, that is, to sense the tactile primitives in the contact area.
具体的,利用基于视觉的触觉传感器通过接触图像的变化,获取触觉模态信息并进行以下推断:Specifically, a vision-based tactile sensor is used to obtain tactile modal information through changes in the contact image and make the following inferences:
(1)是否与物体发生接触,若是则获取到传感器接触图像,若否则获取到无接触触觉初始图像;(1) Whether there is contact with the object, if so, the sensor contact image is obtained, if not, the non-contact tactile initial image is obtained;
(2)物体接触的面积、位置、方向;(2) The area, position and direction of object contact;
(3)通过连续的视频图像获取接触部分在传感器表面运动状况;(3) Obtain the movement of the contact part on the sensor surface through continuous video images;
其中,在使用触觉传感器时,通过单点和多点特征估计物体的接触区域,并通过几何特征的提取和匹配来实现对接触区域的感知,从而获取到与初始图像的差分图;Among them, when using a tactile sensor, the contact area of the object is estimated through single-point and multi-point features, and the perception of the contact area is realized through the extraction and matching of geometric features, thereby obtaining a difference map from the initial image;
利用弹性体材料和表面微凸的构型大多在成像图像上表现为椭圆形接触区域或者圆形接触区域的特性,通过触觉感知模态椭圆图元提取描述接触区域,并将接触区域统一表达成A(p,a,b,θ)的格式;其中,p(x,y)为触觉传感器图像坐标系下接触椭圆的中心坐标,a为接触椭圆的半长轴长,b为接触椭圆的半短轴长,θ为接触椭圆的偏转角度,以椭圆的长轴为参考,其值为长轴偏转水平方向的角度,值域为[-π/2,π/2]。特别地,当接触区域的形状为圆形时,长轴长与短轴长相等即a=b,θ为[-π/2,π/2]内任意值。Most of the configurations using elastomer materials and surface micro-convexities appear as elliptical contact areas or circular contact areas on imaging images. The contact area is described through tactile sensing modal elliptical primitive extraction, and the contact area is uniformly expressed as The format of A(p,a,b,θ); where, p(x,y) is the center coordinate of the contact ellipse in the tactile sensor image coordinate system, a is the semi-major axis length of the contact ellipse, and b is the semi-major axis of the contact ellipse. The length of the minor axis, θ is the deflection angle of the contact ellipse, taking the major axis of the ellipse as a reference, its value is the angle of the major axis deflection in the horizontal direction, and the value range is [-π/2,π/2]. In particular, when the shape of the contact area is circular, the length of the major axis is equal to the length of the minor axis, that is, a=b, and θ is any value within [-π/2, π/2].
步骤2:基于获取的触觉模态信息,进行椭圆参数拟合,具体包括:Step 2: Based on the acquired tactile modal information, perform ellipse parameter fitting, specifically including:
步骤2.1:对差分图进行灰度和形态学处理后使用Canny边缘检测算子对接触区域进行边缘检测提取;Step 2.1: After performing grayscale and morphological processing on the difference image, use the Canny edge detection operator to perform edge detection and extraction on the contact area;
步骤2.2:使用基于最小二乘法的几何椭圆拟合方式对检测提取到的边缘图像进行拟合,如图2所示,在提高检测速度的同时保证椭圆参数拟合的精度。Step 2.2: Use the geometric ellipse fitting method based on the least squares method to fit the edge image extracted by the detection, as shown in Figure 2, which improves the detection speed while ensuring the accuracy of the ellipse parameter fitting.
步骤3:构建机械臂的试抓取坐标系,具体包括:Step 3: Construct the trial grasping coordinate system of the robotic arm, including:
步骤3.1:以当前时刻由抓取检测算法推理出的抓取配置地图G(x,y,z,rx,ry,rz,W)为基准,以抓取中心坐标为原点建立抓取坐标系o-xyz,用于方便描述机械臂的夹爪稳定闭合之后的视触感知环节的物体的运动特征;Step 3.1: Based on the current grab configuration map G (x, y, z, r x , r y , r z , W) deduced by the grab detection algorithm, establish a grab with the grab center coordinates as the origin. The coordinate system o-xyz is used to conveniently describe the motion characteristics of objects in the visual and tactile sensing link after the gripper of the robotic arm is stably closed;
步骤3.2:如图3所示,本实施例的机械臂夹爪的工作表面安装两个面对面的触觉传感器,为方便同一物体的接触信息的描述,对两幅接触图像的坐标系重新定义,以右侧传感器成像为基准,以320×240成像区域中心为坐标系中心建立右图像坐标系o-xz,左侧传感器成像按照o-xyz坐标系的zox平面为镜像,以图像中心为中心建立左图像坐标系o-xz,左、右图像产生的接触区域分别标记为A1、A2,对应的中心分别为p1(x1,z1)、p2(x2,z2);其中,p(x,z)的坐标值为触觉图像空间到真实接触表面尺寸的映射值。Step 3.2: As shown in Figure 3, two face-to-face tactile sensors are installed on the working surface of the manipulator gripper in this embodiment. In order to facilitate the description of the contact information of the same object, the coordinate system of the two contact images is redefined to The right sensor imaging is used as a benchmark, and the right image coordinate system o-xz is established with the center of the 320×240 imaging area as the coordinate system center. The left sensor imaging is mirrored according to the zox plane of the o-xyz coordinate system, and the left image coordinate system is established with the image center as the center. In the image coordinate system o-xz, the contact areas generated by the left and right images are marked A 1 and A 2 respectively, and the corresponding centers are p 1 (x 1 ,z 1 ) and p 2 (x 2 ,z 2 ) respectively; where , the coordinate value of p(x,z) is the mapping value from the tactile image space to the real contact surface size.
步骤4:对物体进行试抓取使物体发生滑移,解算物体的触觉感知滑动矢量,推测物体在滑移期间的位姿变化,具体包括:Step 4: Trial grasp the object to cause the object to slip, calculate the tactile sensing sliding vector of the object, and infer the posture changes of the object during the sliding period, including:
步骤4.1:本发明将该初始状态统一描述成偏差的形式;物体的初始偏差包括初始位置偏差(Δx,Δy,Δz)和初始角度偏差(Δrx,Δry,Δrz),其中Δy由相机视野中获得,其大小表示物体偏离抓取坐标系o-xyz的zox平面距离;初始时刻的位姿偏差如下式所示:Step 4.1: The present invention uniformly describes the initial state in the form of deviation; the initial deviation of the object includes the initial position deviation (Δx, Δy, Δz) and the initial angle deviation (Δr x , Δr y , Δr z ), where Δy is determined by the camera Obtained from the field of view, its size represents the zox plane distance of the object from the grasping coordinate system o-xyz; the pose deviation at the initial moment is as follows:
步骤4.2:通过读取触觉图像的视频流,获取接触椭圆的运动特征,两帧图像之间的滑动特征,如图4所示;Step 4.2: By reading the video stream of the tactile image, obtain the motion characteristics of the contact ellipse and the sliding characteristics between the two frame images, as shown in Figure 4;
步骤4.3:由于椭圆拟合方法存在误差,直接通过视频的第一帧和最后一帧确定椭圆的状态矢量会导致较大的误差,为了减少误差,先对椭圆参数进行滤波,再描述椭圆状态矢量,最终确定物体在滑移期间的位姿变化:Step 4.3: Due to errors in the ellipse fitting method, determining the state vector of the ellipse directly from the first and last frames of the video will lead to large errors. In order to reduce the error, filter the ellipse parameters first, and then describe the ellipse state vector. , and finally determine the pose change of the object during the sliding period:
步骤5:如图5所示,对物体在试抓取过程中发生的不同剧烈程度的滑移进行对应的优化调整,重构机械臂的再抓取位姿,具体包括:Step 5: As shown in Figure 5, make corresponding optimization adjustments to the different degrees of slippage that occur during the trial grasping process of the object, and reconstruct the re-grasping posture of the robotic arm, specifically including:
步骤5.1:对物体在试抓取过程中发生的初期滑移进行优化调整,具体为:Step 5.1: Optimize and adjust the initial slip of the object during the trial grasping process, specifically as follows:
按照初期滑移的收敛且有界的特点,提出如下两条调整策略:According to the convergent and bounded characteristics of initial slip, the following two adjustment strategies are proposed:
(1)期望接触区域的中心位置在传感器的感受范围之内,若在刚接触时接触检测的拟合椭圆中心位于传感器的感受范围之外,则极有可能发生全局滑移,调整时应使接触区域中心沿着靠近传感器感受范围中心方向进行。(1) It is expected that the center position of the contact area is within the sensing range of the sensor. If the center of the fitting ellipse for contact detection is outside the sensing range of the sensor at the time of initial contact, global slip is very likely to occur, and adjustments should be made so that The center of the contact area is along a direction close to the center of the sensor's sensing range.
(2)对于小幅度的初期滑移运动,调整策略只需使末端执行器的位姿沿物体位姿变化的反方向进行调整即可。特别地,由于重力的影响,物体天然具有沿z轴负方向运动的趋势,对于z轴方向上的变动采用缩放处理,本发明采用的缩放因子为0.6,即将抓取配置地图G(x,y,z,rx,ry,rz,W)调整为:(2) For small initial sliding movements, the adjustment strategy only needs to adjust the pose of the end effector in the opposite direction of the object pose change. In particular, due to the influence of gravity, objects naturally have a tendency to move in the negative direction of the z-axis. For changes in the z-axis direction, scaling is used. The scaling factor used in the present invention is 0.6, which means that the configuration map G(x, y) will be captured. ,z,r x ,r y ,r z ,W) adjusted to:
G′(x-dx,y-dy,z-dz×0.6,rx-drx,ry-dry,rz-drz,W)。G′(x-dx,y-dy,z-dz×0.6,r x -dr x ,r y -dr y ,r z -dr z ,W).
步骤5.2:对物体在试抓取过程中发生的局部滑移进行优化调整,具体为:Step 5.2: Optimize and adjust the local slip that occurs during the trial grasping process of the object, specifically as follows:
对照当前物体的抓取位置地图,在抓取地图非零位置上沿当前抓取坐标系的x轴靠近物体重心的方向搜索,根据该区域附近置信度最高点的点重新构建抓取构型,通过所述的空间映射函数变化f获取新的抓取配置。搜索的方向由两个触觉传感器确定,若物体的dry为正值,则沿x轴负方向搜索,若dry为负值,则沿x轴正方向搜索。Compare the grab position map of the current object, search at the non-zero position of the grab map in the direction close to the center of gravity of the object along the x-axis of the current grab coordinate system, and reconstruct the grab configuration based on the point with the highest confidence near the area. A new crawling configuration is obtained by changing f through the spatial mapping function. The direction of the search is determined by two tactile sensors. If the dr y of the object is a positive value, the search is along the negative direction of the x-axis. If the dr y is a negative value, the search is along the positive direction of the x-axis.
在三维空间中以杆状物体为例,如图6所示,通过受力分析可知,稳定抓取的临界条件是机械臂末端手指的摩擦扭矩等于重力乘以力臂,其中,力臂的距离可由式所得:Taking a rod-shaped object in the three-dimensional space as an example, as shown in Figure 6, through force analysis, it can be seen that the critical condition for stable grasping is that the friction torque of the fingers at the end of the mechanical arm is equal to gravity times the force arm, where the distance of the force arm It can be obtained from the formula:
其中,L为重力力臂,沿当前抓取坐标下x轴方向移动该距离即可将局部滑移矫正为稳定的抓取动作;T为摩擦副的力矩;θ是与水平方向的夹角。Among them, L is the gravity arm. Moving this distance along the x-axis direction under the current grasping coordinate can correct the local slip into a stable grasping action; T is the torque of the friction pair; θ is the angle with the horizontal direction.
由于抓取物体的种类、形状、质量及其分布特点、表面接触特性等属性均未知,因此需要对上式进行估值处理。在本实施例中抓取物体的质量不超过1kg,因此可设定G=9.8,摩擦力矩T可由下式获得。Since the type, shape, quality, distribution characteristics, surface contact characteristics and other attributes of the grasped object are unknown, the above equation needs to be evaluated. In this embodiment, the mass of the grabbed object does not exceed 1kg, so G=9.8 can be set, and the friction torque T can be obtained by the following formula.
T=μF∫A rdrT=μF∫ A rdr
其中,μ为手指与物体的摩擦系数,本实施例近似取值为0.7;F为夹持力,其大小可由夹爪状态读取;A为物体与触觉传感器的接触区域,因为物体的表面曲率分布不一致,为方便后续处理,本实施例假设弹性体的变形区域的力分布满足均匀分布的要求。综上所述,可估计摩擦力矩T的大小。最后,以估值的L为探索的距离,并将其映射到图像空间中获得探索的距离Li,矢量即为图像空间中搜索的迭代方向和步长。每次迭代时更换抓取点,新的抓取坐标表示为下式:Among them, μ is the friction coefficient between the finger and the object, and the approximate value in this embodiment is 0.7; F is the clamping force, the size of which can be read from the state of the clamp; A is the contact area between the object and the tactile sensor, because the surface curvature of the object The distribution is inconsistent. To facilitate subsequent processing, this embodiment assumes that the force distribution in the deformation area of the elastic body meets the requirements of uniform distribution. In summary, the friction torque T can be estimated. Finally, take the estimated L as the explored distance and map it to the image space to obtain the explored distance L i , vector That is the iteration direction and step size of the search in the image space. The grab point is replaced in each iteration, and the new grab coordinates are expressed as follows:
步骤5.3:对物体在试抓取过程中发生的全面滑移进行优化调整,具体为:Step 5.3: Optimize and adjust the overall slippage of the object during the trial grasping process, specifically as follows:
若当前的抓取配置发生全面滑移,说明当前的抓取姿态不适合当前的抓取任务;调整策略应当舍弃当前抓取区域,更换抓取配置地图上的抓取点,舍弃当前抓取位置地图上置信度的峰值,寻找另一个超过阈值的局部峰值,然后从抓取地图上生成一个新的坐标,并以此推断新的抓取构型,得到更新后的抓取配置;最后通过多次抓取质量分析和局部调整,可将全面滑移问题转化为局部滑移或者初期滑移问题,从而在新的抓取配置附近按照上述策略调整抓取姿态直至达到稳定抓取的状态。If the current grabbing configuration completely slips, it means that the current grabbing posture is not suitable for the current grabbing task; the adjustment strategy should discard the current grabbing area, replace the grabbing points on the grabbing configuration map, and discard the current grabbing position. The peak confidence level on the map is used to find another local peak value that exceeds the threshold, and then a new coordinate is generated from the capture map, and the new capture configuration is inferred from this to obtain the updated capture configuration; finally, through multiple Secondary grasping quality analysis and local adjustment can convert the overall slip problem into a local slip or initial slip problem, so that the grasping posture can be adjusted according to the above strategy near the new grasping configuration until a stable grasping state is achieved.
下面根据实施例描述本发明,本发明的目的和效果将变得更加明显。The purpose and effects of the present invention will become more apparent when the present invention is described below based on the embodiments.
本发明性能测定:本实施例所用物体来自EGAD,在进行抓取实验前,首先构建其碰撞模型并将其放缩至适合夹取的尺寸,通过随机改变其重心位置来模拟抓取对象性质的不确定性,对于每个抓取对象进行10次随机重心改变,每个变换过重心的物体分别进行10次抓取实验,实验所用物体模型如图7所示。本实施例基于Pybullet展开仿真实验,其中选用的机器人型号为Rethink sawyer、平行夹爪型号为WSG-50、触觉传感器为DIGIT,实验过程如图8所示,经过调整前后的抓取配置结果如图9所示。Performance measurement of the present invention: The object used in this example comes from EGAD. Before conducting the grasping experiment, first construct its collision model and scale it to a size suitable for clamping, and simulate the properties of the grasped object by randomly changing the position of its center of gravity. Uncertainty: 10 random center-of-gravity changes were performed for each grasped object, and 10 grasping experiments were conducted for each object with a changed center of gravity. The object model used in the experiment is shown in Figure 7. In this embodiment, a simulation experiment is carried out based on Pybullet. The selected robot model is Rethink sawyer, the parallel gripper model is WSG-50, and the tactile sensor is DIGIT. The experimental process is shown in Figure 8. The grabbing configuration results before and after adjustment are shown in Figure 8. 9 shown.
实施例1:Example 1:
为了模拟抓取对象位姿的不确定性,本实施例随机采样了下式所示的抓取对象位姿不确定空间;为了模拟机器人执行误差以及其他不确定性环境的影响,在执行抓取时对预测抓取姿态增加随机的扰动。In order to simulate the uncertainty of the grasping object's pose, this embodiment randomly samples the grasping object's pose uncertainty space shown in the following formula; in order to simulate the robot execution error and the influence of other uncertain environments, when executing the grasp When adding random perturbations to the predicted grasping posture.
S={<x,y,θ>∣x∈[-20,20],y∈[-20,20],θ∈[0,2π]}S={<x,y,θ>∣x∈[-20,20],y∈[-20,20],θ∈[0,2π]}
在实施例中,抓取物体的质量为1kg,夹爪夹持力为20N,调整上限次数为五,通过Pybullet的getBasePositionAndOrientation方法获取抓取前后物体的位姿变化,若其变化量达到理想值的80%~100%,则视为成功的抓取,否则视为失败的抓取,其中,若其变化量达到理想值的90%~100%,则视为稳定的抓取。In the embodiment, the mass of the grabbed object is 1kg, the clamping force is 20N, and the upper limit of adjustment times is five. The change in posture of the object before and after grabbing is obtained through Pybullet's getBasePositionAndOrientation method. If the change reaches the ideal value If the change reaches 80% to 100%, it is considered a successful capture, otherwise it is considered a failed capture. Among them, if the change reaches 90% to 100% of the ideal value, it is considered a stable capture.
实验结果如下表所示:The experimental results are shown in the following table:
由表可知,随着再抓取姿态调整的应用,平均抓取成功率从87.6%提升至92.2%,其中稳定抓取的比率为88.4%,该结果表明反应式再抓取优化策略提高了在物体及其姿态不确定下抓取过程的稳定性。As can be seen from the table, with the application of re-grasp posture adjustment, the average grasp success rate increased from 87.6% to 92.2%, of which the stable grasp rate was 88.4%. This result shows that the reactive re-grasp optimization strategy improves the performance of Stability of grasping process under uncertainty of object and its pose.
以上所述,仅为本发明的优选实施案例,并非对本发明做任何形式上的限制。虽然前文对本发明的实施过程进行了详细说明,对于熟悉本领域的人员来说,其依然可以对前述各实例记载的技术方案进行修改,或者对其中部分技术特征进行同等替换。凡在本发明精神和原则之内所做修改、同等替换等,均应包含在本发明的保护范围之内。The above are only preferred implementation examples of the present invention, and do not impose any formal restrictions on the present invention. Although the implementation process of the present invention has been described in detail above, those familiar with the art can still modify the technical solutions recorded in the foregoing examples, or make equivalent substitutions for some of the technical features. All modifications, equivalent substitutions, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.
Claims (9)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310774480.6A CN116945166A (en) | 2023-06-28 | 2023-06-28 | An optimization method for robot re-grasp based on tactile element slip feature feedback |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310774480.6A CN116945166A (en) | 2023-06-28 | 2023-06-28 | An optimization method for robot re-grasp based on tactile element slip feature feedback |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116945166A true CN116945166A (en) | 2023-10-27 |
Family
ID=88453944
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310774480.6A Pending CN116945166A (en) | 2023-06-28 | 2023-06-28 | An optimization method for robot re-grasp based on tactile element slip feature feedback |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116945166A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120816502A (en) * | 2025-09-17 | 2025-10-21 | 途见科技(北京)有限公司 | Tactile perception data processing method, object recognition method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080202202A1 (en) * | 2004-09-14 | 2008-08-28 | Jun Ueda | Tactile Sensor and Use Thereof |
| US20210125052A1 (en) * | 2019-10-24 | 2021-04-29 | Nvidia Corporation | Reinforcement learning of tactile grasp policies |
| CN113510700A (en) * | 2021-05-19 | 2021-10-19 | 哈尔滨理工大学 | Touch perception method for robot grabbing task |
| US20220126453A1 (en) * | 2020-10-22 | 2022-04-28 | Mitsubishi Electric Research Laboratories, Inc. | Interactive Tactile Perception Method for Classification and Recognition of Object Instances |
-
2023
- 2023-06-28 CN CN202310774480.6A patent/CN116945166A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080202202A1 (en) * | 2004-09-14 | 2008-08-28 | Jun Ueda | Tactile Sensor and Use Thereof |
| US20210125052A1 (en) * | 2019-10-24 | 2021-04-29 | Nvidia Corporation | Reinforcement learning of tactile grasp policies |
| US20220126453A1 (en) * | 2020-10-22 | 2022-04-28 | Mitsubishi Electric Research Laboratories, Inc. | Interactive Tactile Perception Method for Classification and Recognition of Object Instances |
| CN113510700A (en) * | 2021-05-19 | 2021-10-19 | 哈尔滨理工大学 | Touch perception method for robot grabbing task |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120816502A (en) * | 2025-09-17 | 2025-10-21 | 途见科技(北京)有限公司 | Tactile perception data processing method, object recognition method and device |
| CN120816502B (en) * | 2025-09-17 | 2025-11-18 | 途见科技(北京)有限公司 | Haptic perception data processing method, object cognition method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113500017B (en) | An intelligent system and method for material sorting in unstructured scenarios | |
| D’Avella et al. | A study on picking objects in cluttered environments: Exploiting depth features for a custom low-cost universal jamming gripper | |
| EP4037878B1 (en) | Systems and methods for determining pose of objects held by flexible end effectors | |
| Eppner et al. | Grasping unknown objects by exploiting shape adaptability and environmental constraints | |
| Platt et al. | Null-space grasp control: Theory and experiments | |
| Sommer et al. | Bimanual compliant tactile exploration for grasping unknown objects | |
| CN114494426B (en) | Devices and methods for controlling a robot to pick up objects from different orientations. | |
| CN113284179B (en) | Robot multi-object sorting method based on deep learning | |
| CN111098323B (en) | A five-finger dexterous hand based on force and displacement fuzzy hybrid control and its control method | |
| CN115401698B (en) | Method and system for planning smart grabbing of manipulator based on grabbing gesture detection | |
| CN207231476U (en) | A kind of courier packages' grabbing device based on binocular vision | |
| CN117754558A (en) | Grasping teaching through human demonstration | |
| Ford et al. | Tactile-driven gentle grasping for human-robot collaborative tasks | |
| CN116652962A (en) | An Object Grasping Method Based on Visual Feedback Control | |
| Su et al. | Robust grasping for an under-actuated anthropomorphic hand under object position uncertainty | |
| CN114952836A (en) | Multi-fingered hand robot grasping method, device and robot system | |
| CN116945166A (en) | An optimization method for robot re-grasp based on tactile element slip feature feedback | |
| Schiebener et al. | Discovery, segmentation and reactive grasping of unknown objects | |
| CN117381799A (en) | Machine vision-based gripping device control method, gripping device and storage medium | |
| CN116460845A (en) | A 6-DOF Grasping Method for Unknown Objects Considering Point Cloud Skeleton Features | |
| Gil et al. | Visual/Tactile sensing to monitor grasps with robot-hand for planar elastic objects | |
| Vatsal et al. | Augmenting vision-based grasp plans for soft robotic grippers using reinforcement learning | |
| Kicki et al. | Robotic manipulation of elongated and elastic objects | |
| Zhu et al. | A robotic semantic grasping method for pick-and-place tasks | |
| Ranjan et al. | Identification and control of NAO humanoid robot to grasp an object using monocular vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |