WO2025004380A1 - Program, information processing method, and information processing device - Google Patents

Abstract

Disclosed is a program that causes a computer to execute: first output processing for outputting an input screen including an image in a prescribed area; input acquisition processing for acquiring a user input from a user; image processing for image-recognizing a target having a boundary corresponding to the position or an area pertaining to a designated input by performing image recognition processing on the image on the basis of the user input including the designated input of the position or the area with respect to the prescribed area or the image; second output processing for generating a prescribed display representing an image recognition result of the image processing, and outputting the prescribed display in association with the target; and setting processing for generating setting information pertaining to the target pertaining to the prescribed display on the basis of a user input including a setting instruction from the user.

Description

PROGRAM, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING APPARATUS

This disclosure relates to a program, an information processing method, and an information processing device.

A technology is known in which an image recognition unit performs image recognition to recognize parts in the parts storage unit using images acquired by a camera that captures images of the parts in the parts storage unit, and when the presence of a part in the picking section is recognized by image recognition, the robot control unit controls the robot to pick up the recognized part (see, for example, Patent Document 1).

JP 2018-202599 A

Incidentally, when performing various processes by image recognition on various objects, not limited to parts in a parts storage section, settings for preparation are required.
Therefore, the present disclosure makes it possible to efficiently perform image-based settings via a user interface.

According to an aspect of the present disclosure, a first output process outputs an input screen including an image in a predetermined area;
an input acquisition process for acquiring a user input from a user;
performing image recognition processing on the image based on the user input including a designation input of a position or area for the predetermined area or the image, thereby performing image recognition of an object having a boundary corresponding to the position or area related to the designation input;
a second output process of generating a predetermined display representing an image recognition result of the image processing and outputting the predetermined display in association with the target;
A setting process for generating setting information related to the target related to the predetermined display based on the user input including a setting instruction from a user;
A program for execution by a computer is provided.

According to the present disclosure, image-based settings can be efficiently performed via a user interface.

1 is a configuration diagram illustrating an example of an information processing device according to an embodiment of the present invention. 2 is a diagram illustrating an example of a hardware configuration of a processing unit of the information processing device illustrated in FIG. 1 . FIG. 2 is a block diagram for explaining functions of the information processing device according to the embodiment. 11A to 11C are diagrams showing examples (part 1) of input screens output by an input screen output processing unit when a user selects a teaching mode. 13A to 13C are diagrams showing examples (part 2) of input screens output by the input screen output processing unit when a user selects the teaching mode. 13A to 13C are diagrams showing examples (part 3) of input screens output by the input screen output processing unit when the user selects the teaching mode. 13A to 13C are diagrams showing examples (part 4) of input screens output by the input screen output processing unit when the user selects the teaching mode. 10 is a schematic flowchart for illustrating a flow of a process executed when an input screen for a teaching mode is selected. 13 is a schematic flowchart for explaining the flow of a process that is executed in a state where the input screen for the picking execution mode is selected. FIG. 10 is an explanatory diagram of FIG. 9 and shows an example (part 1) of an input screen for a picking execution mode. FIG. 10 is an explanatory diagram of FIG. 9 and shows an example of an input screen for the picking execution mode (part 2). FIG. 10 is an explanatory diagram of FIG. 9 and shows an example of an input screen for the picking execution mode (part 3). 10 is a timing chart showing an example of information exchange between components during picking. FIG. 13 is a diagram showing an input screen output by an input screen output processing unit according to another embodiment.

Each embodiment will be described in detail below with reference to the attached drawings.

FIG. 1 is a configuration diagram showing an example of a system 1 including an information processing device 20 according to this embodiment.

The system 1 includes an information processing device 20, a robot arm controller 40, a robot arm 42, and a camera 50.

The information processing device 20 is, for example, a PC (Personal Computer), but may also be realized by a mobile phone, a smartphone, a tablet terminal, etc. The information processing device 20 may also be realized by a combination of multiple PCs that are communicatively connected via a network. The information processing device 20 may also be realized by a combination of PCs and server computers that are communicatively connected via a network.

The information processing device 20 realizes the setting assistance function according to this embodiment. The setting assistance function is a function that assists the user in setting various settings based on images, and details thereof will be described later with reference to FIG. 3 and subsequent figures.

In the example shown in FIG. 1, the information processing device 20 includes a communication unit 21, a memory unit 22, a display unit 23, an input unit 24, and a control unit 25.

The communication unit 21 includes an interface that communicates with external devices wirelessly or via a wired connection to send and receive information. The communication unit 21 can communicate with the robot arm controller 40 and the camera 50 via a network.

The storage unit 22 includes, for example, a primary storage device and a secondary storage device. For example, the storage unit 22 may include a semiconductor memory, a magnetic memory, an optical memory, or the like. The storage unit 22 stores various information and programs related to the setting assistance function. The information and programs related to the setting assistance function may be obtained from an external device via an external memory or the communication unit 21. For example, a setting assistance application program may be obtained from a specified application distribution server. Hereinafter, the application program may also be simply referred to as an application. The setting assistance application may be in the form of a web application.

The display unit 23 includes a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 23 is capable of displaying a variety of images. The display unit 23 may be configured as a touch panel, for example. The display unit 23 may be built into a head-mounted display.

The input unit 24 may include physical keys, and may further include any input interface, including a pointing device such as a mouse.

The control unit 25 controls the operation of the entire information processing device 20. The control unit 25 executes a setting assistance application to execute processing related to the setting assistance function. For example, the control unit 25 outputs a GUI (Graphical User Interface) (described later with reference to FIG. 4) that detects operation input on the screen of the display unit 23. The control unit 25 receives various inputs by touch operations, input operations, etc. via the input unit 24, and executes processing related to the setting assistance function in response to the various inputs.

The robot arm controller 40 is communicatively connected to the information processing device 20. The robot arm controller 40 controls a robot arm 42. Note that some or all of the functions of the robot arm controller 40 may be realized by the information processing device 20.
The robot arm 42 may be, for example, an articulated arm. The robot arm 42 may be used for any purpose, but in this embodiment, as an example, the robot arm 42 is used for picking up objects. In this case, the robot arm 42 may have a gripping portion (hand) at the tip, and may be capable of gripping objects by opening and closing the gripping portion.

The camera 50 is positioned so as to be able to capture images of various objects, including objects to be picked by the robot arm 42 (hereinafter also referred to as "picking objects"). For example, the camera 50 may be positioned in association with a conveyor in a factory line, and capture images of various objects moving on the conveyor in real time. The image data captured by the camera 50 may be supplied to the information processing device 20. The camera 50 may include a camera that captures RGB color images (hereinafter also referred to as "RGB images") and/or a depth camera that captures depth images. The camera 50 may also include a camera that utilizes electromagnetic waves having wavelengths outside the wavelength range of visible light, such as infrared rays and ultraviolet rays (for example, a camera equipped with an element that forms an image on a flat surface).

FIG. 2 is a diagram showing an example of the hardware configuration of the processing unit 10 of the information processing device 20 shown in FIG. 1.

In the example shown in FIG. 2, the processing unit 10 includes a CPU (Central Processing Unit) 11, a GPU (Graphics Processing Unit) 11A, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an auxiliary storage device 14, a drive device 15, and a communication interface 17, all connected by a bus 19, as well as a wired transceiver unit 18A and a wireless transceiver unit 18B connected to the communication interface 17.

GPU 11A performs image processing such as drawing various images, which will be described later.

The auxiliary storage device 14 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and is a storage device that stores data related to application software, etc.

The wired transceiver 18A includes a transceiver capable of communicating using a wired network. Peripheral devices forming the input unit 24 and display unit 23 shown in FIG. 1 are connected to the wired transceiver 18A. However, some or all of the peripheral devices may be connected to the bus 19 or may be connected to the wireless transceiver 18B.

The wireless transceiver 18B is a transceiver capable of communicating using a wireless network. The wireless network may include a wireless communication network for mobile phones, the Internet, a VPN (Virtual Private Network), a WAN (Wide Area Network), etc. The wireless transceiver 18B may also include a near field communication (NFC) unit, a Bluetooth (registered trademark) communication unit, a Wi-Fi (Wireless-Fidelity (registered trademark)) transceiver unit, an infrared transceiver unit, etc.

The wired transceiver unit 18A and the wireless transceiver unit 18B form the communication unit 21 shown in FIG. 1, for example.

The processing unit 10 may be connectable to a recording medium 16. The recording medium 16 stores a specific program. The program stored in the recording medium 16 is installed in the auxiliary storage device 14 of the processing unit 10 via the drive device 15. The installed specific program can be executed by the CPU 11 of the processing unit 10. For example, the recording medium 16 may be a recording medium that records information optically, electrically or magnetically, such as a CD (Compact Disc)-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor memory that records information electrically, such as a ROM or a flash memory. The recording medium 16 does not include a carrier wave.

FIG. 3 is a block diagram for explaining the functions of the processing unit 10 of the information processing device 20 of this embodiment. In the following, a user is defined as a person who provides various operational inputs to the input unit 24 of the information processing device 20.

As shown in FIG. 3, the information processing device 20 includes an image acquisition processing unit 210, an input screen output processing unit 212, an input acquisition processing unit 214, an image recognition processing unit 216, a recognition result output unit 218, and a setting processing unit 220. Each unit from the image acquisition processing unit 210 to the setting processing unit 220 can be realized by the CPU 11 shown in FIG. 2 executing one or more programs stored in a storage device (e.g., ROM 13).

The image acquisition processing unit 210 acquires an image including the object to be picked. As described above, the object to be picked can be any object, but in this embodiment, it is a specific type of tool stored in a tray (case), and the image including the object to be picked is also referred to as a "tray image G10." Note that the object to be picked is not limited to being in a container such as a tray, but may be placed directly on a conveyor or the like, or may be stored in another container such as a bag.

The tray image G10 to be acquired by the image acquisition processing unit 210 may be specified by the user. The tray image G10 acquired by the image acquisition processing unit 210 may be an image stored in an external database or the like, or may be an image stored in the recording medium 16 described above.

The input screen output processing unit 212 outputs an input screen for performing various settings, etc.

FIGS. 4 to 7 are diagrams showing examples (transition states) of input screens output by the input screen output processing unit 212 when the user selects the teaching mode. FIG. 4 shows one state of the input screen G400, and FIG. 5 shows another state of the input screen G400 formed by user input, etc. FIG. 6 shows yet another state of the input screen G400 formed by user input, etc. FIG. 7 shows yet another state of the input screen G400 formed by user input, etc.

The input screen G400 forms a GUI (Graphical User Interface). The input screens G400 (FIG. 4, etc.), G1000 (FIG. 10, etc., described below), etc. may be switchable by the user selecting (switching input) tabs Tb1 and Tb2. When tab Tb1 is selected, settings related to the picking execution mode are possible, and when tab Tb2 is selected, settings related to the teaching mode are possible. Information that the user can input via the input screen G400, etc. will be described later.

The input screen G400 has an image display area G310 that displays the tray image G10 acquired by the image acquisition processing unit 210. In this embodiment, as an example, the tray image G10 is an image of a planar view (top view) of the tray containing the items to be picked, as described above. Note that in modified examples, the tray image G10 may be another view (e.g., an oblique view), or a combination of two or more views may be used.

The input screen G400 becomes available for input when tab Tb2 is selected. The input screen G400 includes a user interface for making various settings in the teaching mode.

The input screen G400 includes buttons BT1 to BT6 that allow the user to input various commands. Buttons BT1 to BT6 are buttons that cause the information processing device 20 to perform image recognition and extract (cut out) the object to be picked from within the tray image G10.

Button BT1 is for "adding a target object point" and is a button that allows the user to associate a target object point P400 (an example of a point with a first attribute) with the image display area G310 or the tray image G10. Three target object points P400 are shown in FIG. 4. In this case, the user can associate the target object point P400 in FIG. 4 with the image display area G310 or the tray image G10 by specifying one or more desired positions while button BT1 is active. In a preferred method of specification, the user specifies the target object point P400 so that it is located within the image portion of the desired picking target.

Button BT2 is for "adding an exclusion point" and is a button that allows the user to associate an exclusion point P402 (an example of a point with a second attribute) with the image display area G310 or the tray image G10. Three exclusion points P402 are shown in FIG. 4. In this case, the user can associate the exclusion point P402 in FIG. 4 with the image display area G310 or the tray image G10 by specifying one or more desired positions while button BT2 is active. In a preferred method of specification, the user specifies the exclusion point P402 so that it is located around the image portion of the desired picking target.

Button BT3 is used to issue a "Cut Out" command, and is a button for extracting (cutting out) an image portion bounded by the contour display G700, which will be described later. Hereinafter, button BT3 is also referred to as cut out button BT3.

Button BT4 is for "remove all points" and is a button for clearing all points (target object points P400 and excluded points P402) associated with the image display area G310 or the tray image G10. For example, the user can redo all associations by operating button BT4 just once.

Button BT5 is used to "go back one step" and clear the most recently matched point (target object point P400 or excluded point P402). For example, the user can operate button BT5 one or more times to redo the previous match.

Button BT6 is used to "go forward one step" and to cancel the operation of button BT5. For example, if the user mistakenly operates button BT5 one or more times, the user can restore the desired association state by operating button BT6 the same number of times.

Note that one or more of the operations related to buttons BT1 to BT6 may be operable using shortcut keys.

The input screen G400 further includes an image load button BT440A, an image capture button BT440B, a work type display area G430, a cut-out work output area G450, a similar product extraction (search) button BT460, a picking location estimation button BT470, a settings load button BT475 (indicated as "Load settings" in the figure), and a settings save button BT480 (indicated as "Save settings" in the figure).

The image load button BT440A is a button for loading a tray image G10, and when operated, it may be possible to specify an image file (the user may specify a desired image file). The loaded tray image G10 is output to the image display area G310.

The image capture button BT440B is typically used when there is no image to load or when teaching is to be performed using a new image. For example, the image capture button BT440B may be a button for capturing an image on the spot using the camera 50 and loading it, and when operated, the image captured by the camera 50 is loaded as a tray image G10. The loaded tray image G10 is output to the image display area G310.

The work type display area G430 is an area that displays the type of work related to the picking target. The type of work related to the picking target may be specified by the user, or may be selectable from a pull-down list. In the latter case, the pull-down list may be automatically generated based on the image recognition results. In this embodiment, as an example, the items that can be selected from the pull-down are already registered work types and items for adding new work. When an item for adding new work is selected, a new name is entered and registered. The "Delete item" button can also be selected, and the item can be deleted.

The cut-out work output area G450 is an area where the cut-out image, described below, is output. By visually checking the image in the cut-out work output area G450, the user can confirm whether the desired image portion has been cut out as the cut-out image.

The similar item extraction button BT460 is a button for extracting an object that is the same as or similar to the target object related to the cut-out image from the tray image G10. For example, the tray image G10 may have multiple (e.g., many) image portions of the same type of tool. In this case, the user can efficiently cut out multiple desired image portions by operating the similar item extraction button BT460 without repeating the same operation on the above-mentioned buttons BT1 and BT2. FIG. 5 shows the state after the similar item extraction button BT460 is operated in the state shown in FIG. 4. In this case, an image portion (cut-out image) related to an object similar to the target object related to the contour display G700 in the state shown in FIG. 4 is extracted in the cut-out work output area G450. At this time, the image portion is preferably output as it is, and the posture in the tray image G10 is maintained (see area Q50). Such maintenance of the posture is suitable when it is desired to set an object in a specific posture as a picking target.

In the example shown in FIG. 5, an edit/delete button BT500 (see area Q51) appears, and the user can edit or delete the image portion (cut-out image) in the cut-out work output area G450 by operating the edit/delete button BT500.

The degree of similarity used to extract similar objects when the similar item extraction button BT460 is operated may be changeable (customizable) by the user. For example, the degree of similarity may be appropriately set to be completely identical, identical in type, etc. Also, factors other than shape, such as color and posture, may be taken into consideration.

The picking location estimation button BT470 is a button for performing (attempting to perform) a picking location estimation process for the current tray image G10 or a newly loaded tray image G10. The picking location estimation process may include a process of recognizing the picking target and estimating the picking location for the picking target.

The save settings button BT480 is a button for saving the current setting state. When the save settings button BT480 is operated in the state shown in FIG. 4 or FIG. 5, the target object related to the image portion extracted in the cut-out work output area G450 at that time is set as the object to be picked.

Here, in this embodiment, in addition to setting (registering) the object to be picked, it may also be possible to set (register) an object that is not to be picked (or an object that should not be touched by the robot arm 42). Hereinafter, the object that is not to be picked or the object that should not be touched by the robot arm 42 will be collectively referred to as a "non-contact object". In this embodiment, as shown in FIG. 6, by selecting the notation "Don't Touch" Box in the work type display area G430, it is possible to set a non-contact object (see Q61 in FIG. 6). Note that the notation is arbitrary. The method of specifying a non-contact object may be the same as the method of specifying an object to be picked described above. In this case, the user can specify a non-contact object by appropriately operating buttons BT1 to BT6. FIG. 6 shows a state in which a target object point P400 is associated with an image portion related to the tray in the image display area G310, and an excluded point P402 is associated with its periphery (see Q60 in FIG. 6). In this state, the user can specify the tray as a non-contact object by operating the cut-out button BT3 for the "cut-out" command.

Note that the image portion of the object that can be the picking object or the non-contact object may be automatically extracted. For example, in the input screen G400A of the example shown in FIG. 7, when the tray image G10 is read, the image portion G720 of each object that can be the picking object or the non-contact object is cut out and output to the unregistered display area G452. The user may be able to edit or delete, as in the case described above with reference to FIG. 5. For example, by dragging and dropping a desired image portion of each image portion output to the unregistered display area G452 to the area G454 related to the picking object, the object related to the image portion can be set as the picking object. Also, by dragging and dropping a desired image portion of each image portion output to the unregistered display area G452 to the area G456 related to the non-contact object, the object related to the image portion can be set as the non-contact object. In this case, a new item may be added by operating the new item addition BT458. A new item is a new item among the items that represent the contents of a process (in this embodiment, the contents of a process related to the robot arm 42), such as a picking target. It may also be possible to delete the item by pressing a cross button BT459 near the item name.

The input acquisition processing unit 214 acquires various inputs via the above-mentioned input screen G400. Note that the input information to be acquired is not limited to input information directly entered by the user, and may include input information that is automatically entered. Automatically entered input information may be acquired definitively when approved by the user (for example, by operating the save settings button BT480).

The image recognition processing unit 216 detects the contour (segment) of an object by performing image recognition processing on the tray image G10 loaded into the image display area G310. Any algorithm can be used to detect the contour of an object, and for example, the Segment Anything Model (SAM) developed by Meta, Inc. or similar may be used.

When the tray image G10 is read, the image recognition processing unit 216 may execute image recognition processing. For example, each time a target object point P400 or a non-contact object is added, the image recognition processing unit 216 may update the image recognition processing result in a manner that reflects the addition.

In addition, in response to the user's operation of the similar object extraction button BT460, the image recognition processing unit 216 recognizes (searches for) an object similar to the object whose contour has been detected (hereinafter also referred to as a "similar object").

In addition, in response to the user's operation of the cut-out button BT3, the image recognition processing unit 216 cuts out (extracts) the object whose contour is detected at that time (the image portion surrounded by the contour display G700 described below).

The recognition result output unit 218 outputs the image recognition result by the image recognition processing unit 216. Specifically, the recognition result output unit 218 generates and outputs a contour display G700 (an example of a predetermined display). The contour display G700 is, for example, a display that highlights (emphasizes) the contour (boundary) detected (recognized) by the image recognition processing unit 216. The contour display G700 represents the image recognition result of the contour (boundary) of the object recognized by the image recognition processing unit 216 based on the above-mentioned target object point P400 and/or the non-contact object. As described above, the image portion surrounded by the contour display G700 can be set (registered) as the picking object and/or the non-contact object. The contour display G700 is preferably displayed superimposed on the tray image G10, but may also be displayed in another location (for example, the cut-out work output area G450) together with the image portion within the contour display G700. In this embodiment, the contour display G700 is displayed superimposed on the tray image G10 according to the detection result of the contour of the corresponding target object.

The recognition result output unit 218 may update the contour display G700 in real time when the image recognition result by the image recognition processing unit 216 is updated. For example, the recognition result output unit 218 may update the contour display G700 in a manner that reflects the update of the image recognition result by the image recognition processing unit 216 resulting from the addition of a target object point P400 or a non-contact object each time the target object point P400 or a non-contact object is added. This allows the user to end the operation of button BT1 or button BT2 and proceed to the operation of the cut-out button BT3 when the desired contour display G700 is obtained.

In place of or in addition to the contour display G700, the recognition result output unit 218 may output a display that highlights (emphasizes) the image portion that should be surrounded by the contour display G700.

In addition, in response to the user's operation of the similar object extraction button BT460, if the image recognition processing unit 216 is able to recognize a similar object, the recognition result output unit 218 generates a contour display G700 of the similar object based on the image recognition result. Note that, if multiple similar objects are recognized, the recognition result output unit 218 may output multiple contour displays G700.

In addition, when the image recognition processing unit 216 cuts out an image portion in response to a user's operation of the cut-out button BT3, the recognition result output unit 218 may output the cut-out image portion to the cut-out work output area G450. At this time, the tray image G10 may be displayed with the cut-out image portion missing, or may be left as is.

The setting processing unit 220 generates setting information for various settings based on the input information obtained up to that point via the input screen G400, in response to the user's operation of the setting save button BT480. At this time, the setting information includes setting information for objects related to the contour display G700. For example, from the input screen G400 shown in FIG. 4, an object related to an image portion that will be output to the cut-out work output area G450 by the subsequent operation of the cut-out button BT3 is set as an object to be picked. Also, from the input screen G400 shown in FIG. 6, an object related to an image portion that will be output to the cut-out work output area G450 by the subsequent operation of the cut-out button BT3 is set as an object to be picked.

In this way, according to this embodiment, various settings based on the tray image G10 can be efficiently performed via the user interface of the input screen G400.

For example, when picking or inspecting for defects, annotation is required to find the detection location, CAD (Computer Aided Design) data (e.g., 3D data) must be read, and rules and procedures must be set. These types of processes and tasks are a burden for users, and are difficult to execute efficiently and accurately.

In this regard, according to this embodiment, the user can efficiently perform these processes and tasks via the teaching mode input screen G400, significantly reducing the burden on the user in picking, defect inspection, etc.

Next, the operation of system 1 of this embodiment will be further explained with reference to Figure 8 onwards.

FIG. 8 is a schematic flowchart for explaining the flow of processing executed when the teaching mode input screen G400 is selected. In the subsequent processing flow diagrams (flowcharts), the processing order of each step may be changed as long as the relationship between the input and output of each step is not lost.

When the user operates the setting read button BT475 ("Y" in step S800), the corresponding setting file is read (step S801). In this case, the user can edit the setting file. When creating a new setting file, it is not necessary to read the setting file using the setting read button BT475. Also, when the user operates the image capture button BT440B ("Y" in step S802), the image captured by the camera 50 is acquired as the tray image G10 as described above (steps S803, S804). Next, the selection state of the work type display area G430 is updated according to the user's selection (step S806), and image recognition processing and output processing of the outline display G700 are executed (step S808). When the user operates the cut-out button BT3, the image portion in the outline display G700 at that time is cut out and displayed in the cut-out work output area G450 (step S810), and if there is further cutting out, the processing from step S806 is repeated. Next, if the user operates the similar item extraction button BT460 ("Y" in step S812), the image portion related to the similar object is cut out as described above and displayed in the cut-out work output area G450 (step S814). If the user operates the picking location estimation button BT470 ("Y" in step S816), the above-mentioned picking location estimation process is executed for the current tray image G10 (step S818). If the user operates the setting save button BT480 ("Y" in step S820), setting information is generated (saved) based on the input information at that time (step S822). The setting information may be saved as a setting file in a form in which a desired file name specified by the user is associated with the setting information. In this case, the user can easily load the desired file in the picking execution mode described later.

FIG. 9 is a schematic flowchart for explaining the flow of processing that is executed when the picking execution mode input screen G1000 (see FIG. 10, etc.) is selected. FIG. 10 and FIG. 11B are explanatory diagrams of FIG. 9, and show an example of the picking execution mode input screen G1000.

The selection button B1010 (automatic picking or setting read button) is operated from the input screen G1000 as shown in FIG. 10 (step S900). At this time, if the setting read button is checked, it is used as the setting information set in the teaching mode described above. When picking is started (step S902), the picking target (and the picking position accordingly) is estimated based on the image from the camera 50 and the setting information during picking (step S904), and it is determined whether or not there is a picking target (a pickable point) (step S906). If there is no pickable point ("N" in step S906), the processing for this tray is terminated. If there is a picking target (a pickable point) ("Y" in step S906), it is determined whether or not the machine can determine whether picking is necessary (step S908). If the machine is capable of making the determination ("Y" in step S908), picking is performed by the robot arm 42 (step S912); if the machine is unable to make the determination ("N" in step S908), the system waits for an instruction from the user as to whether picking should be performed via the input screen G1000 as shown in FIG. 11B. Specifically, if the machine is unable to make the determination ("N" in step S908), an interface (see Q1000 in FIG. 11B) is output and the system waits for a selection from the user (step S910). If an input to perform picking is received from the user ("Y" in step S910), picking is performed by the robot arm 42 (step S912); if an input to perform picking is not received ("N" in step S910), the process from step S904 is repeated (i.e., automatic picking continues).

In this embodiment, after the picking operation mode is started, an input screen G1000 having a stop button BT1100 as shown in FIG. 11A may be output. In this case, the user can stop automatic picking by operating the stop button BT1100. In other words, when the stop button BT1100 is operated ("Y" in step S903), automatic picking is stopped.

Furthermore, in this embodiment, when the object to be picked (and therefore the picking position) is estimated, the outline of the estimated object to be picked may be highlighted as shown by reference symbol G111 in FIG. 11B. This makes it easier for the user to determine whether or not to stop automatic picking (i.e., whether or not to operate the stop button BT1100) and to make a decision (selection) in a situation where the machine cannot make a decision as described above.

FIG. 12 is a timing chart that shows an example of information exchange between each component (information processing device 20, camera 50, and robot arm controller 40) during picking.

The camera 50 acquires an RGB image (camera image) and a depth image of the tray containing the object to be picked (S1200), and transmits these images to the information processing device 20 (step S1202). The image transmission timing may be, for example, the timing when the tray on the conveyor is positioned upstream of the robot arm 42.

When the information processing device 20 receives the RGB image and the depth image from the camera 50 (step S1204), it identifies the object to be picked based on these images and calculates the picking position (step S1206). Then, the information processing device 20 transmits the calculated picking position to the robot arm controller 40 (step S1208).

When the robot arm controller 40 receives the picking position (step S1210), it performs the picking operation, and when the picking is completed, it generates picking completion information indicating that fact (step S1212) and transmits this to the information processing device 20 (step S1214).

Then, the information processing device 20 receives picking completion information (step S1216).

Although each embodiment has been described in detail above, it is not limited to a specific embodiment, and various modifications and changes are possible within the scope of the claims. It is also possible to combine all or some of the components of the above-mentioned embodiments.

For example, in the above-described embodiment, the processing method related to the setting assistance function was picking, but this is not limited to this. For example, the setting can be applied to any processing, such as quality inspection processing based on image recognition, cleaning processing, various cooking processing, etc.

In addition, in the above-described embodiment, there was only one processing method related to the setting assistance function, but it may include a combination of two or more. For example, in the above-described embodiment, the only processing method that could be set was picking, but it may also be possible to set the placement location after picking (the location to which the item to be picked is moved), etc.

In the above-described embodiment, the tabs Tb1 and Tb2 are provided to select either the teaching mode or the picking execution mode, but this is not limited to this. Dedicated software may be prepared for each of the teaching mode and the picking execution mode. In this case, as shown in FIG. 13, the input screen G400A can have a relatively large image display area G310 that does not require the tabs Tb1 and Tb2. Note that while FIG. 13 shows the input screen G400A for the teaching mode, the same applies to the input screen for the picking execution mode (not shown).

REFERENCE SIGNS LIST 1 System 10 Processing section 14 Auxiliary storage device 15 Drive device 16 Recording medium 17 Communication interface 18A Wired transmission/reception section 18B Wireless transmission/reception section 19 Bus 20 Information processing device 21 Communication section 22 Storage section 23 Display section 24 Input section 25 Control section 40 Robot arm controller 42 Robot arm 50 Camera 210 Image acquisition processing section 212 Input screen output processing section 214 Input acquisition processing section 216 Image recognition processing section 218 Recognition result output section 220 Setting processing section

Claims (14)

a first output process for outputting an input screen including an image in a predetermined area;
an input acquisition process for acquiring a user input from a user;
performing image recognition processing on the image based on the user input including a designation input of a position or area for the predetermined area or the image, thereby performing image recognition of an object having a boundary corresponding to the position or area related to the designation input;
a second output process of generating a predetermined display representing an image recognition result of the image processing and outputting the predetermined display in association with the target;
A setting process for generating setting information regarding the target related to the predetermined display based on the user input including a setting instruction from a user;
A program that a computer runs. The program according to claim 1, wherein the predetermined display includes at least one of a display that highlights the contour of the object and a display that highlights the inside of the contour of the object, and is superimposed on the image in accordance with the position of the object in the image. The program according to claim 2, wherein the image processing includes a process of extracting or cutting out an image portion related to the specified display from the image. the input screen has an input area for inputting specific information,
The program according to claim 2 , wherein the setting process includes a process of associating the specific information with the target based on input information in the input area. The program according to claim 4, wherein the specific information includes at least one of information representing a processing method for the object, information representing attributes of the object, and information representing a name of the object. The program according to claim 5, wherein the information representing the processing method includes information representing whether or not to execute a predetermined process on an object that is image-recognized as being the same as or similar to the target. The program according to claim 6, wherein the predetermined process includes a process of picking a specific type of object from among multiple types of objects using a robot arm. The designation input includes an entire designation input that designates an entire image,
The program according to claim 2 , wherein the second output process causes the predetermined display to be superimposed on each of the targets based on the overall designation input. the input screen has an input area for inputting specific information,
the setting process includes a process of associating the specific information with the target based on input information in the input area,
The program according to claim 8 , further causing the computer to execute a third output process of cutting out and outputting, within the input area, image portions of the image relating to each of the targets. the designation input includes an input of a first attribute designating one or more first attribute points or regions;
The program according to claim 2 , wherein the second output process superimposes the predetermined display based on an input of the first attribute so that the contour includes a point or area of the first attribute. the designation input includes an input of a second attribute designating one or more points or regions of a second attribute;
The program according to claim 10 , wherein the second output process superimposes the predetermined display based on the input of the second attribute such that the contour does not include points or areas of the second attribute. The first output process updates the image from one image to another image based on the user input including a change instruction from a user;
The program according to claim 1 , wherein the setting process generates a setting file related to the setting information based on a result of the image recognition process for one or more of the images. a first output processing unit that outputs an input screen including an image in a predetermined area;
an input acquisition processing unit that acquires a user input from a user;
an image processing unit that performs image recognition processing on the image based on the user input including a designation input of a position or area for the predetermined area or the image, thereby performing image recognition of an object having a boundary corresponding to the position or area related to the designation input;
a second output processing unit that generates a predetermined display representing an image recognition result of the image processing unit and outputs the predetermined display in association with the target;
a setting processing unit that generates setting information regarding the target related to the predetermined display based on the user input including a setting instruction from a user. a first output processing step of outputting an input screen including an image in a predetermined area;
an input acquisition processing step for acquiring a user input from a user;
an image processing step of performing image recognition processing on the image based on the user input including a designation input of a position or area for the predetermined area or the image, thereby image-recognizing an object having a boundary corresponding to the position or area related to the designation input;
a second output processing step of generating a predetermined display representing the image recognition result of the image processing step, and outputting the predetermined display in association with the target;
and generating setting information relating to the target related to the predetermined display based on the user input including a setting instruction from a user.