WO2021197195A1 - Picking/placing behavior recognition method and apparatus, and electronic device - Google Patents

Abstract

A picking/placing behavior recognition method and apparatus, and an electronic device. The method comprises: on the basis of multiple frames of acquired images to be recognized, obtaining feature point coordinates corresponding to a target object in each frame of image to be recognized (2000); on the basis of multiple frames of the feature point coordinates, obtaining a recognition result of an action category of the target object, and calculating the distance between the target object and a shelf (2200); and obtaining a recognition result of a picking/placing behavior according to the action category and the distance between the target object and the shelf (2400). The method can reduce the costs of recognizing picking/placing behaviors in a hypermarket, and can achieve the simultaneous recognition of picking/placing behaviors for multiple target objects.

Description

Recognition method, device and electronic equipment of pick-and-place behavior

This application claims the priority of a Chinese patent application filed on March 31, 2020, with the application number 202010244985.8 and the invention title "A method, device and electronic equipment for picking and placing behavior", the entire content of which is incorporated herein by reference Applying.

Technical field

The embodiments of the present specification relate to the technical field of action recognition, and more specifically, to a method for recognizing pick-and-place behavior, a device for recognizing pick-and-place behavior, an electronic device, and a computer-readable storage medium.

Background technique

Smart retail systems can not only provide users with a better shopping experience, but also provide merchants with more convenient management solutions. The intelligent retail system prevents the loss of goods by identifying and recording the user's pick-up or release behavior in the store, and positive applications, such as automated commodity inventory and automated checkout functions.

There are three schemes commonly used in existing smart retail systems. The first is the binocular recognition scheme, that is, a large number of cameras are installed on the shelf or the ceiling, and the position of the user's hand touching the shelf and touching the shelf is measured by the method of binocular three-dimensional reconstruction. The second is to install a camera on the top of the shelf to perform motion recognition and product pick-and-place behavior recognition. The third is the multi-sensor fusion scheme, in which a weight sensor is built into the shelf, and the weight information is combined with the camera to realize the identification of the picking and placing behavior of the goods.

In the above-mentioned existing solutions, the shelves need to be customized, and a large number of cameras need to be installed, and the cost is relatively high. Therefore, it is necessary to propose a new recognition method of pick-and-place behavior.

Summary of the invention

One purpose of the embodiments of this specification is to provide a new technical solution for identifying pick-and-place behavior.

According to the first aspect of the embodiments of this specification, there is provided a method for identifying pick-and-place behavior, including:

Obtain the feature point coordinates corresponding to each frame of the to-be-recognized image based on the collected multiple frames of the to-be-recognized image;

Obtain the recognition result of the action category of the target based on the coordinates of the feature points in multiple frames, and calculate the distance between the target and the shelf;

According to the action category and the distance between the target and the shelf, the recognition result of the pick and place behavior is obtained.

Optionally, wherein the step of obtaining the feature point coordinates corresponding to the target object in each frame of the image to be identified based on the acquired multiple frames of images to be identified includes:

Each frame of the to-be-recognized image is used as the input of the detection algorithm, and the corresponding target image is output;

The target image is used as an input of a feature point extraction algorithm, and the feature point coordinates are output.

Optionally, wherein the step of obtaining the recognition result of the action category of the target object based on the coordinate of the feature points in multiple frames includes:

Encode the feature point coordinates of multiple frames respectively, and use the encoded feature point coordinates as the input of the action recognition algorithm to calculate the corresponding action category, as well as the start time and end time of each category of action, As the recognition result.

Optionally, wherein the step of calculating the distance between the target object and the shelf based on the coordinate of the characteristic points in multiple frames includes:

Based on the coordinates of the first designated point and the coordinate of the second designated point in the coordinate of the characteristic point, and the predetermined shelf elevation label and ground label, the distance between the target and the shelf is calculated.

Optionally, before the step of obtaining the recognition result of the pick and place behavior according to the action category and the distance between the target and the shelf, the method further includes:

Obtaining a determination result of whether the target object holds a commodity based on the target object image;

The step of obtaining the recognition result of the pick-and-place behavior according to the action category and the distance between the target object and the shelf includes:

According to the action type, the distance between the target object and the shelf, and the determination result of whether the target object holds a commodity, the recognition result of the pick-and-place behavior is obtained.

Optionally, wherein the step of obtaining a determination result of whether the target object holds a commodity based on the target object image includes:

The two-classification model is used to calculate the target image to obtain a determination result of whether the target holds a commodity.

Optionally, wherein the multiple frames of images to be recognized are acquired by a monocular camera.

Optionally, wherein the target image includes at least: a human body image, a robot image, or an unmanned aerial vehicle image.

According to the second aspect of the embodiments of the present specification, there is also provided a pick-and-place behavior identification device, including:

An acquisition module, configured to obtain the coordinates of the feature points corresponding to the target in each frame of the image to be identified based on the collected multiple frames of images to be identified;

A calculation module, configured to obtain the recognition result of the action category of the target object based on the coordinate of the feature points in multiple frames, and calculate the distance between the target object and the shelf;

The recognition module is used to obtain the recognition result of the pick-and-place behavior according to the action category and the distance between the target object and the shelf.

According to the third aspect of the embodiments of the present specification, there is provided an electronic device, including the pick-and-place behavior identification device as described in the second aspect of the embodiments of the present specification, or the electronic device includes:

Memory, used to store executable commands;

The processor is configured to, under the control of the executable command, execute the pick-and-place behavior identification method according to any one of the first aspects of the embodiments of this specification.

According to the fourth aspect of the embodiments of the present specification, there is also provided a computer-readable storage medium storing executable instructions, and when the executable instructions are executed by a processor, the fetch as described in the first aspect of the embodiments of the present specification is executed. Recognition method of release behavior.

One beneficial effect of the embodiment of this specification is that the method of this embodiment obtains the feature point coordinates corresponding to the target in each frame of the image to be identified based on the collected multiple frames of images to be identified; based on the multiple frames of the feature point coordinates , Obtain the recognition result of the action category of the target, and calculate the distance between the target and the shelf; according to the action category and the distance between the target and the shelf, obtain the recognition of the pick-and-place behavior result. According to the embodiments of the present specification, the cost of identifying the pick-and-place behavior in the store can be reduced, and the purpose of simultaneously identifying the pick-and-place behavior of multiple targets can be achieved.

Through the following detailed description of exemplary embodiments of the embodiments of the present specification with reference to the accompanying drawings, other features and advantages of the embodiments of the present specification will become clear.

Description of the drawings

The drawings incorporated in the specification and constituting a part of the specification illustrate the embodiments of the embodiments of the specification, and together with the descriptions are used to explain the principles of the embodiments of the specification.

FIG. 1 is a block diagram showing the hardware configuration of a server 1000 that can implement an embodiment of the present invention;

Fig. 2 is a flowchart of a method for recognizing pick-and-place behavior according to an embodiment of the present specification;

Figure 3 is a flowchart of an example according to an embodiment of the present specification;

Figure 4 is a schematic diagram of the recognition process according to an embodiment of the present specification;

Figure 5 is a schematic diagram of calculating the distance between a hand and a shelf according to an embodiment of the present specification;

Fig. 6 is a functional block diagram of a pick-and-place behavior identification device according to an embodiment of the present specification;

Fig. 7 shows a functional block diagram of an electronic device according to an embodiment of the present specification.

Detailed ways

Various exemplary embodiments of the embodiments of the present specification will now be described in detail with reference to the accompanying drawings. It should be noted that unless specifically stated otherwise, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the embodiments of this specification.

The following description of at least one exemplary embodiment is actually only illustrative, and in no way serves as any limitation to the embodiment of the specification and its application or use.

The technologies, methods, and equipment known to persons of ordinary skill in the relevant fields may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be regarded as part of the specification.

In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary, rather than as a limitation. Therefore, other examples of the exemplary embodiment may have different values.

It should be noted that similar reference numerals and letters indicate similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further discussed in the subsequent drawings.

<Hardware Configuration>

FIG. 1 is a block diagram showing a hardware configuration of a server 1000 that can implement an embodiment of the present invention.

The server 1000 may be, for example, a blade server or the like.

In an example, the server 1000 may be a computer.

In another example, as shown in FIG. 2, the server 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, and an input device 1600. Although the server may also include speakers, microphones, etc., these components are not related to the present invention and are therefore omitted here.

Wherein, the processor 1100 may be, for example, a central processing unit CPU, a microprocessor MCU, or the like. The memory 1200 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a serial interface, and the like. The communication device 1400 can perform wired or wireless communication, for example. The display device 1500 is, for example, a liquid crystal display. The input device 1600 may include, for example, a touch screen, a keyboard, and the like.

In this embodiment, the server 1000 is configured to obtain the feature point coordinates corresponding to the target object in each frame of the image to be recognized based on the collected multiple frames of images to be recognized; to obtain the target object based on the feature point coordinates of the multiple frames And calculate the distance between the target object and the shelf; according to the action type and the distance between the target object and the shelf, the recognition result of the pick-and-place behavior is obtained.

The server shown in FIG. 1 is only illustrative and does not imply any restriction on the present invention, its application or use. In an embodiment of the present invention, the memory 1200 of the server 1000 is used to store instructions, and the instructions are used to control the processor 1100 to operate to execute any one of the pick-and-place behavior identification methods provided in the embodiments of the present invention.

Those skilled in the art should understand that although multiple devices are shown for the server 1000 in FIG. 1, the present invention may only involve some of the devices. For example, the server 1000 only involves the processor 1100 and the storage device 1200. Technical personnel can design instructions according to the disclosed scheme of the present invention. How the instruction controls the processor to operate is well known in the art, so it will not be described in detail here.

<Method Example>

This embodiment provides a method for recognizing pick-and-place behavior, and the method may be executed by, for example, the server 1000 as shown in FIG. 1.

As shown in Figure 2, the method includes the following steps 2000-2400:

Step 2000: Obtain the coordinates of the feature points corresponding to the target in each frame of the image to be identified based on the acquired multiple frames of images to be identified.

Wherein, the multi-frame image to be recognized is acquired by the monocular camera. In this embodiment, the space information is given in advance for the labeling of the shelf facade, so that only the multi-frame to-be-identified images collected by the monocular camera can be used to realize the three-dimensional space distance measurement between the target and the shelf.

In this step, the server 1000 may specifically use the detection algorithm and the feature point extraction algorithm to calculate the feature point coordinates. Specifically, the server 1000 may use each frame of the image to be recognized as the input of the detection algorithm and output the corresponding target image; and use the target image as the input of the feature point extraction algorithm to output the coordinate of the feature point.

In practical applications, the object recognized by the server 1000 may be a human body, a robot, or a drone. Correspondingly, the target image may be a human body image, a robot image, or an unmanned aerial vehicle image.

In an example, it is assumed that the multiple frames of images to be recognized collected by the monocular camera are user images. Accordingly, the server 1000 may use the human body detection algorithm and the human skeleton point extraction algorithm to calculate the coordinates of the human skeleton point. That is, the server 1000 uses each frame of user image as the input of the human body detection algorithm, outputs the corresponding human body image, and inputs the human body image into the human skeleton point extraction algorithm to obtain the human skeleton point coordinates corresponding to the human body image.

Wherein, the human skeleton point extraction algorithm may be, for example, a human pose recognition algorithm (OpenPose), a human pose estimation algorithm (AlphaPose), etc., which are not specifically limited herein.

Step 2200: Obtain the recognition result of the action category of the target based on the coordinate of the feature point in multiple frames, and calculate the distance between the target and the shelf.

In this step, when the server 1000 obtains the recognition result of the action category of the target object based on the coordinate of the feature point in multiple frames, the server 1000 may respectively encode the coordinate of the feature point in the multiple frames, and use the coded feature point coordinates as the action The input of the recognition algorithm, the corresponding action category, and the start time and end time of each category of action are calculated as the recognition result.

As shown in Figure 3, taking the target image as a human body image as an example, actions such as extending a hand, touching a shelf, and retrieving goods can be recognized according to the coordinates of the human skeleton point corresponding to the human body image. During recognition, as shown in Figure 4, the server 1000 first encodes multiple frames of the human skeleton point coordinates in the form of a graph, and then uses it as the input of a convolutional neural network (CNN), such as an action recognition algorithm, to calculate the corresponding recognition As a result, such as reaching out, retracting, touching shelves, etc.

Among them, the encoding of the coordinates of the human skeleton point can specifically be normalization of the coordinates of each human skeleton point, subtracting the coordinates of the neck point from the x and y coordinates of each skeleton point, and dividing by the height of the person, In order to exclude the factors of human height and position, and then sort them according to the coordinate axis: the x-axis is the type of coordinate points, such as hands, feet, shoulders, etc., the y-axis is time, and the z-axis is the abscissa, ordinate, and confidence.

During the action recognition, the server 1000 takes and recognizes the skeleton points of the human body along the time axis at a fixed interval and a fixed length, and then combines the same action recognition results to obtain the start time and end time of each action. Among them, the fixed length refers to the number of frames taken, for example, take the key points of 11 frames, and the fixed interval is set considering that the motion amplitude of two adjacent frames is too small. For example, take a skeleton point every other frame .

When the server 1000 calculates the distance between the target object and the shelf based on the coordinate of the characteristic point in multiple frames, it may be specifically based on the coordinate of the first designated point and the coordinate of the second designated point in the coordinate of the characteristic point, and the predetermined Label the shelf elevation and the floor, and calculate the distance between the target and the shelf.

For example, when calculating the distance between the user's hand and the shelf, the server 1000 may be based on the hand coordinates (that is, the coordinates of the first designated point) and the coordinates of the sole point (that is, the coordinates of the second designated point) in the coordinates of the human skeleton point. , And pre-determined shelf labeling and floor labeling, calculate the distance between the hand and the shelf.

As shown in Figure 5, the rectangle formed by abcd is the ground label, and bcef is the shelf elevation label. The shelf facade needs to be marked according to the four sides of the shelf structure. The two sides ab and cd are on the extension line of the bottom edge of the shelf side. cb is the bottom edge of the shelf. In the actual calculation, it is assumed that the depth difference between hands and feet is negligible, that is, the plane AFH is perpendicular to the shelf plane bcef.

Assuming that cd and ab are approximately parallel, the straight line AF equation is calculated from the sides cd and ab, and the coordinates of point A are calculated. Specifically, given the coordinates of the abcd point, the slope of the straight line cd, ab is k, and the F coordinate is (x _f , y _f ), then the AF straight line equation is y=kx+(y _F -x _F ×k).

Calculate the intersection A of the straight line bc and AF: the straight line bc and AF have been obtained, written as a general formula

a _bc x+b _bc y+c _bc =0

a _AF x+b _AF y+c _AF =0

Then the calculation formula for the intersection A of the straight line bc and AF is:

d(fB)＝d(ef)×r，可得，x _B＝min(x _f，x _e)+|x _e-x _f|×r，y _B＝min(y _f，y _e)+|y _e-y _f|×r。 Then according to the ratio of Ac, Ab, calculate the coordinates of B (x _B , y _B ), d(x) is the distance formula between the two points, and calculate the ratio

d(fB)=d(ef)×r, we can get, x _B =min(x _f ,x _e )+|x _e -x _f |×r,y _B =min(y _f ,y _e )+| y _e -y _f |×r.

Calculate the straight line equation AB based on the coordinates of point A and point B.

Since the direction of the straight line HT and the straight line AF are the same, the straight line equation of HT is calculated from the coordinates of the H point and the direction of the straight line AF. Specifically, the coordinates of point A and point F are known, and the linear equation can be obtained. Suppose the slope of the linear equation of AF is k, the slope of the straight line HT is also k, the coordinates of point H are known (x _H , y _H ), and the linear equation of HT is y=kx+(y _H- x _H × k).

Then calculate the intersection point T of AB and HT. It can be seen from the above that the straight lines AB and HT have been obtained and written as general formulas: a _AB x+b _AB y+c _AB =0, a _HT x+b _HT y+c _HT =0. The calculation formula for the intersection of two straight lines is:

Based on the coordinates of the H point and the T point, the HT length of the line segment can be calculated, that is, the distance from the hand to the shelf. When the distance HT from the hand to the shelf is small enough, it is considered that the hand touches the shelf.

It should be noted that in this step, the distance between the target and the shelf calculated by the server 1000 may be the actual distance in space or the pixel distance on the image. Among them, the actual distance needs to calibrate the corresponding relationship between the pixel and the space to reflect the true distance between the target and the shelf. For example, the approximate ratio between the pixel distance and the space distance at point H is r (the r is based on the actual measurement The approximate value obtained), then the spatial distance (cm/m)=d(HT)*r.

In practical applications, the result obtained by calculating the actual distance in space is more accurate. If the space is not calibrated, only the pixel distance can be used to obtain an approximate value, but since the number of pixels cannot correspond to the centimeter in the space, the result will not be accurate enough.

Step 2400: According to the action category and the distance between the target object and the shelf, a recognition result of the pick and place behavior is obtained.

Specifically, the action category includes extending, retracting, touching the shelf, etc. Between the start time and the end time of each action, the distance between the target and the shelf is gradually reduced, then the server 1000 can recognize This behavior is extending. Between the start time and the end time of each action, the distance between the target and the shelf gradually increases, and the server 1000 can recognize that the behavior is retracting.

In an example, the server 1000 may also obtain a determination result of whether the target object holds a commodity based on the target object image. Specifically, the server 1000 may use the two-classification model to calculate the target image to obtain a determination result of whether the target holds a commodity.

Correspondingly, the server 1000 will obtain the recognition result of the pick-and-place behavior according to the action type, the distance between the target object and the shelf, and the determination result of whether the target object holds a commodity.

In this example, the recognition result includes the action category, and the start time and end time of each category of action. The server 1000 can determine whether the target object is to pick up or release the goods based on the recognition result. Specifically, between the start time and the end time is a process of action, that is, reaching into the shelf or extending the shelf, and combining the determination result of whether the target object holds the commodity in this process, the recognition result of the pick-and-place behavior is obtained. For example, if the target does not hold the commodity during the extension process, and the target holds the commodity during the extension process, this is the process of picking up the goods. The target holds the commodity during the extension process, and the target does not hold the commodity during the extension process, then this is the delivery process.

The method of this embodiment has been described above with reference to the drawings and specific examples. The server obtains the coordinate of the feature point corresponding to the target in each frame of the image to be recognized based on the collected multiple frames of images to be recognized; based on the coordinate of the feature point in the multiple frames, it obtains the recognition result of the action category of the target, and calculates the target The distance between the object and the shelf; according to the action category and the distance between the target object and the shelf, the recognition result of the pick-and-place behavior is obtained. This embodiment uses a monocular camera to measure the distance between the target and the shelf with the help of the labeling of the shelf. The operation is simple, and the recognition of the picking and placing behavior of multiple targets at the same time is realized, and the existing store equipment is modified. The small scale effectively reduces the implementation cost and maintenance cost.

<Device Example>

This embodiment provides a pick-and-place behavior recognition device. The device is, for example, the pick-and-place behavior recognition device 6000 shown in FIG. 6. The pick-and-place behavior recognition device 6000 may include an acquisition module 6100, a calculation module 6200, and an identification module 6300.

Wherein, the acquisition module 6100 is configured to obtain the coordinates of the feature points corresponding to the target in each frame of the image to be identified based on the acquired multiple frames of images to be identified.

The calculation module 6200 is configured to obtain the recognition result of the action category of the target object based on the coordinate of the feature point in multiple frames, and calculate the distance between the target object and the shelf.

The recognition module 6300 is used to obtain the recognition result of the pick-and-place behavior according to the action category and the distance between the target object and the shelf.

Wherein, the multi-frame image to be recognized is acquired by the monocular camera.

Specifically, the acquisition module 6100 is specifically configured to: use each frame of the image to be recognized as the input of the detection algorithm and output the corresponding target image; use the target image as the input of the feature point extraction algorithm to output the coordinate of the feature point .

In an example, the calculation module 6200 can be specifically used to separately encode the feature point coordinates of multiple frames, and use the encoded feature point coordinates as the input of the action recognition algorithm to calculate the corresponding action category and each The start time and end time of the action of the category are used as the recognition result.

In an example, the calculation module 6200 can be specifically configured to calculate the target object and the target object based on the first designated point coordinates and the second designated point coordinates in the characteristic point coordinates, and the predetermined shelf elevation and ground labels. The distance between the shelves.

In an example, the calculation module 6200 can also be used to obtain a determination result of whether the target object holds a commodity based on the target object image; correspondingly, the identification module 6300 is specifically configured to determine whether the target object holds a commodity according to the action category, the target object and the shelf. The distance between the two and the determination result of whether the target object holds the commodity, and the recognition result of the pick-and-place behavior is obtained.

Wherein, when the calculation module 6200 obtains the determination result of whether the target object holds the commodity based on the target object image, it can specifically calculate the target object image by using the two-class model to obtain the determination of whether the target object holds the commodity. result.

The pick-and-place behavior identification device of this embodiment can be used to implement the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and will not be repeated here.

<Equipment Example>

In this embodiment, there is also provided an electronic device, which includes the pick-and-place behavior identification device 6000 described in the device embodiment of this specification; or, the electronic device is the electronic device 7000 shown in FIG. 7 and includes:

The memory 7100 is used to store executable commands.

The processor 7200 is configured to execute the method described in any method embodiment in this specification under the control of the executable command stored in the memory 7100.

The implementation subject of the electronic device according to the executed method embodiment may be a server.

<Computer-readable storage medium embodiment>

This embodiment provides a computer-readable storage medium in which an executable command is stored, and when the executable command is executed by a processor, the method described in any method embodiment in this specification is executed.

The embodiments of this specification may be systems, methods and/or computer program products. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling a processor to implement various aspects of the embodiments of the present specification.

The computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a printer with instructions stored thereon The protruding structure in the hole card or the groove, and any suitable combination of the above. The computer-readable storage medium used here is not interpreted as the instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires Transmission of electrical signals.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device .

The computer program instructions used to perform the operations of the embodiments of this specification may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or one or more programming Source code or object code written in any combination of languages. Programming languages include object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to access the Internet). connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by using the status information of the computer-readable program instructions. The computer-readable program instructions are executed to realize various aspects of the embodiments of this specification.

Here, various aspects of the embodiments of the present specification are described with reference to the flowcharts and/or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present specification. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine that makes these instructions when executed by the processor of the computer or other programmable data processing device , A device that implements the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make computers, programmable data processing apparatuses, and/or other devices work in a specific manner, so that the computer-readable medium storing the instructions includes An article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions onto a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , So that the instructions executed on the computer, other programmable data processing apparatus, or other equipment realize the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions, and operations of the system, method, and computer program product according to multiple embodiments of the embodiments of this specification. In this regard, each block in the flowchart or block diagram can represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction contains one or more executables for realizing the specified logic function. instruction. In some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions Or it can be realized by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that realization by hardware, realization by software, and realization by a combination of software and hardware are all equivalent.

The various embodiments of the embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the described embodiments, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements in the market of each embodiment, or to enable other ordinary technical persons in the technical field to understand the various embodiments disclosed herein. The scope of the embodiments of this specification is defined by the appended claims.

Claims (11)

A method for identifying pick-and-place behavior, including: Based on the collected multiple frames of images to be identified, obtaining the coordinates of the feature points corresponding to the target in each frame of the image to be identified; Obtain the recognition result of the action category of the target based on the coordinates of the feature points in multiple frames, and calculate the distance between the target and the shelf; According to the action category and the distance between the target and the shelf, the recognition result of the pick and place behavior is obtained. The method according to claim 1, wherein the step of obtaining the coordinate of the feature point corresponding to the target object in each frame of the image to be recognized based on the collected multiple frames of the image to be recognized comprises: Each frame of the to-be-recognized image is used as the input of the detection algorithm, and the corresponding target image is output; The target image is used as an input of a feature point extraction algorithm, and the feature point coordinates are output. The method according to claim 1, wherein the step of obtaining the recognition result of the action category of the target object based on the coordinate of the feature points in multiple frames comprises: Encode the feature point coordinates of multiple frames respectively, and use the encoded feature point coordinates as the input of the action recognition algorithm to calculate the corresponding action category, as well as the start time and end time of each category of action, As the recognition result. The method according to claim 1, wherein the step of calculating the distance between the target object and the shelf based on the coordinate of the characteristic points in multiple frames comprises: Based on the coordinates of the first designated point and the coordinate of the second designated point in the coordinate of the characteristic point, and the predetermined shelf elevation label and ground label, the distance between the target and the shelf is calculated. The method according to claim 2, wherein before the step of obtaining the recognition result of the pick and place behavior according to the action category and the distance between the target object and the shelf, the method further comprises: Obtaining a determination result of whether the target object holds a commodity based on the target object image; The step of obtaining the recognition result of the pick-and-place behavior according to the action category and the distance between the target object and the shelf includes: According to the action type, the distance between the target object and the shelf, and the determination result of whether the target object holds a commodity, the recognition result of the pick-and-place behavior is obtained. The method according to claim 5, wherein the step of obtaining a determination result of whether the target object holds a commodity based on the target object image comprises: A two-class model is used to calculate the image of the target, and the result of determining whether the target holds a commodity is obtained. The method according to claim 1, wherein the multiple frames of to-be-recognized images are collected by a monocular camera. The method according to claim 2, wherein the target image includes at least: a human body image, a robot image, or an unmanned aerial vehicle image. A recognition device for pick-and-place behavior, including: An acquisition module, configured to obtain the coordinates of the feature points corresponding to the target in each frame of the image to be identified based on the collected multiple frames of images to be identified; A calculation module, configured to obtain the recognition result of the action category of the target object based on the coordinate of the feature points in multiple frames, and calculate the distance between the target object and the shelf; The recognition module is used to obtain the recognition result of the pick-and-place behavior according to the action category and the distance between the target object and the shelf. An electronic device, comprising the pick-and-place behavior identification device according to claim 9, or the electronic device comprises: Memory, used to store executable commands; The processor is configured to execute the pick-and-place behavior identification method according to any one of claims 1-8 under the control of the executable command. A computer-readable storage medium storing executable instructions, and when the executable instructions are executed by a processor, the method for identifying pick-and-place behaviors according to any one of claims 1-8 is executed.

Images