[go: up one dir, main page]

WO2025155278A1 - Système et procédé d'estimation de dimensions planes d'objet pour une manipulation autonome d'objets - Google Patents

Système et procédé d'estimation de dimensions planes d'objet pour une manipulation autonome d'objets

Info

Publication number
WO2025155278A1
WO2025155278A1 PCT/US2024/011534 US2024011534W WO2025155278A1 WO 2025155278 A1 WO2025155278 A1 WO 2025155278A1 US 2024011534 W US2024011534 W US 2024011534W WO 2025155278 A1 WO2025155278 A1 WO 2025155278A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
image
points
main plane
segmented
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/US2024/011534
Other languages
English (en)
Inventor
Ines UGALDE DIAZ
Kyle COELHO
Eugen SOLOWJOW
Husnu Melih ERDOGAN
Paul Andreas Batsii
Brian ZHU
Christopher SCHÜTTE
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens AG
Siemens Corp
Original Assignee
Siemens AG
Siemens Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens AG, Siemens Corp filed Critical Siemens AG
Priority to PCT/US2024/011534 priority Critical patent/WO2025155278A1/fr
Publication of WO2025155278A1 publication Critical patent/WO2025155278A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40053—Pick 3-D object from pile of objects
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40298—Manipulator on vehicle, wheels, mobile
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40584—Camera, non-contact sensor mounted on wrist, indep from gripper
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/45—Nc applications
    • G05B2219/45063—Pick and place manipulator
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • the fourth industrial revolution aims to drive mass customization to the cost of mass production.
  • This can be enabled by autonomous machines that no longer have to be programmed with detailed instructions, such as way points or manually taught paths, but automatically define their tasks using design information of the product to be produced.
  • Robotic bin picking is one such skill that enables autonomous machines.
  • Traditional robotic picking relies on a teach-based approach, such that operators can predefine the robot poses for pick and drop positions.
  • advances in computer vision and deep learning have enabled flexible robotic bin picking, where pre-teaching pick and drop positions is no longer required.
  • Camera systems such as RGB-D cameras, collect both color pictures and depth maps or point-clouds of bins with objects in random configurations. The camera input is then transferred to computer vision algorithms or deep neural networks that have been trained to compute grasping points or “pick points” on said input.
  • autonomous systems infer other high-level information of the objects they handle.
  • an autonomous system may need to estimate an object’s footprint, particularly the planar dimensions of the surface on which it is to be eventually placed or dropped, which may allow downstream processes to be performed in an informed manner.
  • the method further comprises projecting a set of points in the segmented point cloud, which define an extent of the main plane, with respect to a normal of the main plane, to create a 2D image.
  • the method further comprises processing the 2D image to generate a contour defining an outline of the placement surface and therefrom estimating planar dimensions of the placement surface.
  • the estimated planar dimensions of the placement surface are used to determine a placement operation executable by the autonomous system for placing the first object at the second location.
  • the computing system 104 may comprise an industrial PC, or any other computing device, such as a desktop or a laptop, or an embedded system, among others.
  • the computing system 104 can include one or more processors configured to process information and/or control various operations associated with the robot 102.
  • the one or more processors may be configured to execute an application program, such as an engineering tool, for operating the robot 102.
  • the application program may generate controller code that defines a task at a high level, for example, using skill functions as described above, which may be deployed to a robot controller 108. From the high-level controller code, the robot controller 108 may generate low-level control signals for one or more motors for controlling the movement of the robot 102, such as angular position of the robot arms, swivel angle of the robot base, and so on, to execute the specified task.
  • the controller code generated by the application program may be deployed to intermediate control equipment, such as programmable logic controllers (PLC), which may then generate low-level control commands for the robot 102 to be controlled.
  • PLC programmable logic controllers
  • the application program may be configured to directly integrate sensor data from physical environment 106 in which the robot 102 operates.
  • the computing system 104 may comprise a network interface to facilitate transfer of live data between the application program and the physical environment 106. An example of a computing system suitable for the present application is described hereinafter in connection with FIG. 5.
  • the robot 102 can include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110.
  • the base 112 can include wheels 114 or can otherwise be configured to move within the physical environment 106.
  • the robot 102 can further include an end effector 116 attached to the robotic manipulator 110.
  • the end effector 116 can include one or more tools configured to grasp and move an object 118 from a first location to a second location.
  • the end effector 116 may include a vacuum-based gripper (suction cup), among other types.
  • objects 118 to be transported are placed in a receiver or bin 120 at the first location.
  • the robotic manipulator 110 can be configured to move so as to change the position of the end effector 116, for example, so as to place or move objects 118 within the physical environment 106.
  • a bin picking application involves grasping objects 118, in a singulated manner, from the bin 120, by the robotic manipulator 110, using the end effectors 116.
  • the objects 118 may be arranged in arbitrary poses within the bin 120.
  • the objects 118 can be of assorted types or of the same type.
  • the physical environment 106 which includes the objects 118 placed in the bin 120, is perceived via sensor data generated by one or more sensors 122.
  • the sensor(s) 122 may include, one or more single or multi-modal sensors, for example, RGB sensors, depth sensors, infrared cameras, 3D scanners, among others.
  • the one or more sensors includes at least one camera 122 mounted to the robotic manipulator 110.
  • the sensor data may be fed to a grasp detection algorithm deployed on the computing system 104 to determine an optimal grasp to be executed by the robot 102 based on the specified objective and imposed constraints (e.g., dimensions and location of the bin).
  • the grasp detection algorithm may comprise a grasp neural network to compute the grasp for a robot to pick up an object given an input depth image (3D map) of the scene.
  • Grasp neural networks are often convolutional, such that the networks can label each location (e.g., pixel) of the input image with some type of grasp affordance metric, referred to as grasp score.
  • the output of the grasp detection algorithm may include an optimal grasp location, referred to as a “pick point.”
  • a grasp detection algorithm by itself may not have any notion regarding the presence of an object of interest in the scene from the sensor data. Accordingly, it may be often necessary to additionally deploy an object detection algorithm on the computing system 104 to perform a semantic recognition of the objects 118 in the scene, to be able to select an object 118 to be transported by the robot 102.
  • the object detection algorithm may comprise, among others, a segmentation neural network (e.g., a Mask Recurrent Convolutional Neural Network or Mask R-CNN) that can provide a pixel-wise object recognition output given an input two-dimensional color intensity image, such as a RGB color image.
  • a segmentation neural network e.g., a Mask Recurrent Convolutional Neural Network or Mask R-CNN
  • a bin picking application may involve a combination of an object detection algorithm, to localize an object of interest among the objects 118 in the bin 120 given a color intensity image of the scene, and a grasp detection algorithm, to compute optimal grasp locations given a depth image of the scene.
  • FIG. 2 illustrates a bin and object configuration in a robotic bin picking application. As shown, objects 202 in a bin 200 can be detected based on an object detection algorithm and respective pick points 204 for individual objects 202 can be computed using a grasp detection algorithm.
  • the object detection and grasp detection algorithms may utilize off-the-shelf neural networks that have been validated and tested extensively in similar applications.
  • the above-described approach can enable a robot to pick an object from a scene without any preprogramming or prior knowledge of how the object looks like.
  • the disclosed methodology is directed to this objective.
  • FIG. 3 illustrates an example of a computer-implemented method 300 for estimating planar dimensions of an object for effecting a placement operation by an autonomous system.
  • the method 300 may be implemented in a computing environment in various ways, for example, as hardware and programming.
  • the programming of the blocks 302-310 of the method 300 may take the form of processor-executable instructions stored on non-transitory machine-readable storage mediums and the hardware may include processors to execute those instructions.
  • the programs may run on an industrial PC or on a smaller device (e.g., a controller) of an autonomous system.
  • the processing capability may be distributed among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems or cloud/network elements.
  • outlier points with respect to the main plane that contribute to noisy measurements may be removed, for example, using statistical outlier filters.
  • This step may be performed depending on the object geometry. For example, if the object is deformable (e.g., a bag of chips, garment wrapped in a plastic bag, etc.), this step can be less critical, since there is already significant noise in the point cloud on account of the deformable object geometry. However, in case of rigid objects, this step may be suitably implemented to produce a clean surface.
  • the contour 410 generated on the 2D image 408 is missing a comer. This may occur, for example, due to faulty depth imaging and/or insufficient points in the point cloud.
  • the missing corner is recovered by fitting a minimum area rectangle 412 that contains all the points in the contour 410.
  • the dimensions of the main plane may be obtained by simply measuring the length and width of the rectangle 412.
  • the estimated planar dimensions of the object placement surface may be used to compute a placement plan.
  • the placement plan may include, among others, the position (coordinates) and/or orientation (rotation) of the transported object at the placement location.
  • the placement plan may be outputted to a controller of a physical device (e.g., robot) for executing a placement of the object at the second location.
  • a controller of a physical device e.g., robot
  • the placement operation may include placing the object in another bin, placing the object in a stack or grid, dropping the object in a chute, etc.
  • FIG. 5 illustrates an exemplary computing environment comprising a computing system 502, within which aspects of the present disclosure may be implemented.
  • the computing system 502 may be embodied, for example and without limitation, as an industrial PC with a Linux operating system, for executing real-time control of a physical device, such as a robot.
  • the computing system 502 may include a communication mechanism such as a system bus 504 or other communication mechanism for communicating information within the computing system 502.
  • the computing system 502 further includes one or more processors 506 coupled with the system bus 504 for processing the information.
  • the processors 506 may include one or more central processing units (CPUs), graphical processing units (GPUs), Al accelerators, or any other processor known in the ait.
  • system memory 508 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 506.
  • a basic input/output system 514 (BIOS) containing the basic routines that help to transfer information between elements within computing system 502, such as during start-up, may be stored in system memory ROM 510.
  • System memory RAM 512 may contain data and/or program modules that are immediately accessible to and/or presently being operated on by the processors 506.
  • System memory 508 may additionally include, for example, operating system 516, application programs 518, other program modules 520 and program data 522.
  • the computing system 502 also includes a disk controller 524 coupled to the system bus 504 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 526 and a removable media drive 528 (e.g., floppy disk drive, compact disc drive, tape drive, and/or solid state drive).
  • the storage devices may be added to the computing system 502 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire).
  • SCSI small computer system interface
  • IDE integrated device electronics
  • USB Universal Serial Bus
  • FireWire FireWire
  • the computing system 502 may also include a display controller 530 coupled to the system bus 504 to control a display 532, such as a cathode ray tube (CRT) or liquid crystal display (LCD), among other, for displaying information to a computer user.
  • the computing system 502 includes a user input interface 534 and one or more input devices, such as a keyboard 536 and a pointing device 538, for interacting with a computer user and providing information to the one or more processors 506.
  • the pointing device 538 for example, may be a mouse, a light pen, a trackball, or a pointing stick for communicating direction information and command selections to the one or more processors 506 and for controlling cursor movement on the display 532.
  • the display 532 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 538.
  • the computing system 502 also includes an I/O adapter 546 coupled to the system bus 504 to connect the computing system 502 to a controllable physical device, such as a robot.
  • a controllable physical device such as a robot.
  • the VO adapter 546 is connected to robot controller 548.
  • the robot controller 548 includes, for example, one or more motors for controlling linear and/or angular positions of various pails (e.g., arm, base, etc.) of a robot.
  • the computing system 502 may perform a portion or all of the processing steps of embodiments of the disclosure in response to the one or more processors 506 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 508. Such instructions may be read into the system memory 508 from another computer readable storage medium, such as a magnetic hard disk 526 or a removable media drive 528.
  • the magnetic hard disk 526 may contain one or more datastores and data files used by embodiments of the present disclosure. Datastore contents and data files may be encrypted to improve security.
  • the processors 506 may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory 508.
  • hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.
  • the computing system 502 may include at least one computer readable storage medium or memory for holding instructions programmed according to embodiments of the disclosure and for containing data structures, tables, records, or other data described herein.
  • the term “computer readable storage medium” as used herein refers to any medium that participates in providing instructions to the one or more processors 506 for execution.
  • a computer readable storage medium may take many forms including, but not limited to, non-transitory, non-volatile media, volatile media, and transmission media.
  • Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 526 or removable media drive 528.
  • Non-limiting examples of volatile media include dynamic memory, such as system memory 508.
  • Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 504. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
  • the computing environment 500 may further include the computing system 502 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 544.
  • Remote computing device 544 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computing system 502.
  • computing system 502 may include a modem 542 for establishing communications over a network 540, such as the Internet. Modem 542 may be connected to system bus 504 via network interface 545, or via another appropriate mechanism.
  • Network 540 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computing system 502 and other computers (e.g., remote computing device 544).
  • the network 540 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art.
  • Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 540.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Geometry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

Un procédé de gestion autonome d'un objet consiste à acquérir des données de capteur pour produire un nuage de points d'une scène comprenant un ou plusieurs objets à un premier emplacement. Une instance d'objet est segmentée dans le nuage de points à l'aide d'un masque de segmentation d'instance. L'instance d'objet représente un objet individuel à transporter par un dispositif physique pouvant être commandé. À l'aide de points du nuage de points segmenté, une direction d'un plan principal est déterminée, qui représente une surface de placement de l'objet. Un ensemble de points du nuage de points segmenté délimitant une étendue du plan principal est projeté par rapport à une normale du plan principal, pour créer une image 2D. L'image 2D est traitée pour générer un contour délimitant un aperçu de la surface de placement, à partir duquel des dimensions planes de la surface de placement sont estimées pour informer un placement de l'objet par le dispositif physique.
PCT/US2024/011534 2024-01-15 2024-01-15 Système et procédé d'estimation de dimensions planes d'objet pour une manipulation autonome d'objets Pending WO2025155278A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/US2024/011534 WO2025155278A1 (fr) 2024-01-15 2024-01-15 Système et procédé d'estimation de dimensions planes d'objet pour une manipulation autonome d'objets

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2024/011534 WO2025155278A1 (fr) 2024-01-15 2024-01-15 Système et procédé d'estimation de dimensions planes d'objet pour une manipulation autonome d'objets

Publications (1)

Publication Number Publication Date
WO2025155278A1 true WO2025155278A1 (fr) 2025-07-24

Family

ID=90053761

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2024/011534 Pending WO2025155278A1 (fr) 2024-01-15 2024-01-15 Système et procédé d'estimation de dimensions planes d'objet pour une manipulation autonome d'objets

Country Status (1)

Country Link
WO (1) WO2025155278A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170302905A1 (en) * 2016-04-13 2017-10-19 Sick Inc. Method and system for measuring dimensions of a target object
CN112605986A (zh) * 2020-11-09 2021-04-06 深圳先进技术研究院 自动取货的方法、装置、设备及计算机可读存储介质
US11341350B2 (en) * 2018-01-05 2022-05-24 Packsize Llc Systems and methods for volumetric sizing
CN116255912A (zh) * 2022-12-29 2023-06-13 江苏中科贯微自动化科技有限公司 包裹的静态体积测量方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170302905A1 (en) * 2016-04-13 2017-10-19 Sick Inc. Method and system for measuring dimensions of a target object
US11341350B2 (en) * 2018-01-05 2022-05-24 Packsize Llc Systems and methods for volumetric sizing
CN112605986A (zh) * 2020-11-09 2021-04-06 深圳先进技术研究院 自动取货的方法、装置、设备及计算机可读存储介质
CN116255912A (zh) * 2022-12-29 2023-06-13 江苏中科贯微自动化科技有限公司 包裹的静态体积测量方法及系统

Similar Documents

Publication Publication Date Title
JP6692107B1 (ja) 物体識別のための方法および計算システム
US12456287B2 (en) Synthetic dataset creation for object detection and classification with deep learning
TW202034215A (zh) 利用視訊資料映射物件實體
US12380671B2 (en) Systems and methods for generating an object recognition template
US12236609B2 (en) Method and computing system for processing candidate edges
US20240198530A1 (en) High-level sensor fusion and multi-criteria decision making for autonomous bin picking
US20240066723A1 (en) Automatic bin detection for robotic applications
US20240198515A1 (en) Transformation for covariate shift of grasp neural networks
WO2025049074A1 (fr) Préhension robotique à l'aide d'un transformateur de vision efficace
WO2025155278A1 (fr) Système et procédé d'estimation de dimensions planes d'objet pour une manipulation autonome d'objets
Khafagy et al. Automated complex path extraction and optimization based on scalable vector graphics and genetic algorithm for industrial robot applications
US20250242498A1 (en) System and method for pick pose estimation for robotic picking with arbitrarily sized end effectors
Wiedholz et al. Semantic 3d scene segmentation for robotic assembly process execution
WO2025250119A1 (fr) Profilage de bac basé sur la vision pour prélèvement de bac robotique
US20250262772A1 (en) Place conditioned pick for robotic pick and place operations
WO2025085127A1 (fr) Système et procédé d'apprentissage et d'exécution de dispositifs de préhension robotiques
US20240253234A1 (en) Adaptive region of interest (roi) for vision guided robotic bin picking
Schwarz et al. Tracking planes with time of flight cameras and j-linkage
Hayat et al. Vision System and Calibration
Gomes Trajectory Generation for a Robot Manipulator using data from a 2D Industrial Laser

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24707366

Country of ref document: EP

Kind code of ref document: A1