JP7567996B2 - COUNTING DEVICE, CONTROL METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to counting products.

During inspection work, products to be inspected are managed. For example, Patent Document 1 discloses a technique for analyzing an image of a package to be inspected to identify the name and number of layers of the package, identifying a stacking method corresponding to the name of the package, and identifying the number of packages based on the stacking method and number of layers. The stacking method is determined in advance for each product.

JP 2013-067499 A

In the invention of Patent Document 1, the features of various packages are stored in advance in a database, and the name of the package is identified by comparing the features obtained from an image of the package with the features of each package stored in the database. However, with this method, it takes a long time to identify the name of the package. As a result, it also takes a long time to identify the number of packages.

The present invention was made in consideration of the above-mentioned problems, and one of its objectives is to provide a technology that can identify the number of products in a short period of time.

The counting device of the present invention has: 1) an identification unit that identifies products identified by marks based on marks detected from some of the products included in a captured image; and 2) a counting unit that identifies target products from the captured image that are determined to be the same product as the identified product based on information about the image of the identified product, and counts the number of identified target products.

The control method of the present invention is executed by a computer. The control method includes: 1) a step of identifying products identified by marks based on marks detected from some of the products included in a captured image; and 2) a step of counting to identify target products from the captured image that are determined to be the same product as the identified product based on information about the image of the identified product, and to count the number of the identified target products.

The program of the present invention causes a computer to execute each step of the control method of the present invention.

The present invention provides a technology that can quickly identify the number of products.

4A to 4C are diagrams conceptually illustrating an example of the operation of the counting device according to the first embodiment. FIG. 2 is a block diagram illustrating a functional configuration of a counting device. FIG. 13 is a diagram illustrating a computer for implementing a counting device. 4 is a flowchart illustrating a flow of a process executed by the counting device of the first embodiment. FIG. 13 is a diagram illustrating a case in which an image of a group of products is captured from one direction. A figure illustrating a case where a group of products is imaged from one direction and a second camera generates a second captured image and a depth image. FIG. 11 is a block diagram illustrating a functional configuration of a counting device according to a second embodiment.

The following describes an embodiment of the present invention with reference to the drawings. In all drawings, similar components are given similar reference numerals and descriptions are omitted where appropriate. In addition, unless otherwise specified, in each block diagram, each block represents a functional configuration, not a hardware configuration. In the following description, unless otherwise specified, various predetermined values (such as threshold values) are stored in advance in a storage device accessible from the functional configuration unit that uses the values.

[Embodiment 1]
＜Overview＞
Fig. 1 is a diagram conceptually illustrating the operation of the counting device 2000 according to the first embodiment. Here, the operation of the counting device 2000 described using Fig. 1 is an example for making it easier to understand the counting device 2000, and does not limit the operation of the counting device 2000. Details and variations of the operation of the counting device 2000 will be described later.

The counting device 2000 identifies the number of target products 10 included in the product group 20. The product group 20 is one or more target products 10 gathered in one place. For example, multiple target products 10 stacked on a pallet during inspection work at the time of shipment or arrival of the target products 10 are treated as a product group 20. Below, an overview of the method by which the counting device 2000 identifies the number of target products 10 included in the product group 20 is described.

The counting device 2000 acquires a first captured image 32. The first captured image 32 is generated by a first camera 30. The first camera 30 is positioned so that the mark 12 is within its imaging range. The mark 12 is any mark (e.g., a barcode) that represents the identification information of the target product 10. The counting device 2000 analyzes the first captured image 32 to identify the identification information of the target product 10 from the mark 12 included in the first captured image 32.

The counting device 2000 acquires master information 50 that is associated with the identification information of the target product 10. The master information 50 is used for image analysis to detect the target product 10. The master information 50 is, for example, a template image of the target product 10, image features of the target product 10 (feature quantities on the image), etc.

The counting device 2000 acquires a second captured image 42 that includes the group of products 20. The second captured image 42 is generated by the second camera 40. The second camera 40 is provided so that the group of products 20 is included in its imaging range. The counting device 2000 detects the target products 10 included in the second captured image 42 by analyzing the second captured image 42 using the master information 50. The counting device 2000 determines the number of target products 10 included in the group of products 20 based on the detection result.

Here, one second captured image 42 may not include all of the target products 10 in the product group 20, but may only include some of the target products 10. For example, if the target products 10 are stacked on a pallet, capturing an image of the product group 20 from only one direction may result in some of the target products 10 being hidden by other target products 10.

Therefore, for example, the counting device 2000 determines the number of target products 10 included in the product group 20 based on the number of target products 10 detected from the second captured image 42 and a prior rule regarding the arrangement of the target products 10 in the product group 20. As another example, the counting device 2000 acquires second captured images 42 from each of a plurality of second cameras 40 installed so as to capture images of the product group 20 from different directions, and determines the number of target products 10 included in the product group 20 based on the number of target products 10 detected from each second captured image 42. A more specific method for determining the number of target products 10 will be described later.

<Examples of effects>
The counting device 2000 of this embodiment identifies the identification information of the target products 10 by using the marks 12 included in the first captured image 32. The counting device 2000 also acquires master information 50 corresponding to the identification information of the identified target products 10, and detects the target products 10 from the second captured image 42 by using the master information 50. The counting device 2000 then identifies the number of target products 10 included in the product group 20 based on the detection results of the target products 10.

In this way, in the counting device 2000, when detecting the target products 10 from the second captured image 42 in order to identify the number of the target products 10, the master information 50 corresponding to the identification information of the target products 10 is used. Therefore, when analyzing the second captured image 42, matching using master information of products other than the target products 10 is not performed. Therefore, compared to the invention disclosed in Patent Document 1, the time required for the process of detecting the target products 10 from the image is shorter. Therefore, the target products 10 can be counted in a shorter time. In addition, using the master information 50 of the target products 10 has the effect of reducing errors in product detection (for example, a product included in the second captured image 42 being determined to be a product other than the target product 10).

The counting device 2000 is described in more detail below.

<Example of functional configuration>
2 is a block diagram illustrating a functional configuration of the counting device 2000. The counting device 2000 has an identification unit 2020, a master acquisition unit 2040, and a counting unit 2060. The identification unit 2020 analyzes the first captured image 32 to identify the identification information of the target product 10 represented by the mark 12. The master acquisition unit 2040 acquires master information 50 corresponding to the identification information of the target product 10. The counting unit 2060 detects the target product 10 from the second captured image 42 using the master information 50, and identifies the number of target products 10 included in the product group 20 based on the detection result.

<Example of Hardware Configuration of Counting Device 2000>
Each functional component of the counting device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). A case in which each functional component of the counting device 2000 is realized by a combination of hardware and software will be further described below.

Figure 3 is a diagram illustrating a computer 1000 for realizing the counting device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. In addition, for example, the computer 1000 is a portable computer such as a smartphone or a tablet terminal.

The computer 1000 may be a dedicated computer designed to realize the counting device 2000, or may be a general-purpose computer. In the latter case, for example, by installing a specific application on the computer 1000, the computer 1000 realizes each function of the counting device 2000 (the computer 1000 operates as the counting device 2000). The application is composed of a program for realizing the functional components of the counting device 2000.

The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input/output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path through which the processor 1040, the memory 1060, the storage device 1080, the input/output interface 1100, and the network interface 1120 transmit and receive data to each other. However, the method of connecting the processor 1040 and the like to each other is not limited to a bus connection.

The processor 1040 is a processor of various types, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The memory 1060 is a main storage device realized using a RAM (Random Access Memory) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, an SSD (Solid State Drive), a memory card, or a ROM (Read Only Memory) or the like.

The input/output interface 1100 is an interface for connecting the computer 1000 to an input/output device. For example, an input device such as a keyboard and an output device such as a display device are connected to the input/output interface 1100.

The network interface 1120 is an interface for connecting the computer 1000 to a communication network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). For example, the first camera 30 and the second camera 40 are connected to the network interface 1120 via the communication network. However, the first camera 30 and the second camera 40 may be communicatively connected to the computer 1000 via something other than the network interface 1120 (for example, via the input/output interface 1100), or may not be communicatively connected to the computer 1000. In the latter case, for example, each camera and the computer 1000 are set to be able to access a common storage device. Then, the computer 1000 acquires the first captured image 32 and the second captured image 42 via that storage device.

The storage device 1080 stores a program that realizes each functional component of the counting device 2000 (a program that realizes the application described above). The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the counting device 2000.

<Processing flow>
4 is a flowchart illustrating the flow of a process executed by the counting device 2000 of the first embodiment. The identification unit 2020 acquires the first captured image 32 (S102). The identification unit 2020 analyzes the first captured image 32 to identify the identification information of the target product 10 represented by the mark 12 (S104). The master acquisition unit 2040 acquires master information 50 corresponding to the identification information of the target product 10 (S106). The counting unit 2060 acquires the second captured image 42 (S108). The counting unit 2060 analyzes the second captured image 42 using the master information 50 to detect the target product 10 from the second captured image 42 (S110). The counting unit 2060 identifies the number of target products 10 included in the product group 20 based on the detection result of the target product 10 (S110).

The process flow executed by the counting device 2000 is not limited to that shown in FIG. 4. For example, the timing for acquiring the second captured image 42 can be any timing prior to S110.

<Regarding Target Product 10 and Product Group 20>
The target product 10 is any product that is the subject of counting. Here, the target product 10 may be a single product that is a unit of sale, or a package of multiple products that are units of sale. For example, when multiple identical products are transported in a single cardboard box, the single cardboard box containing the multiple products is treated as one target product 10.

The target products 10 are grouped together and become the subject of some kind of processing. A product group 20 is a collection of multiple target products 10 grouped together in this way as a processing unit. For example, multiple products may be placed on a single pallet, and inspection may be carried out on a pallet basis. Thus, multiple target products 10 placed on a single pallet in this way are treated as a product group 20. Note that when multiple identical products are placed in a single cardboard box, and the cardboard box is collectively used as a processing unit (for example, when multiple cardboard boxes are stacked on a pallet and inspection is carried out on a pallet basis), such a group of multiple cardboard boxes is treated as a product group 20.

For example, the target products 10 are products that are subject to inspection during receiving and shipping operations. Inspection operations during receiving and shipping may be carried out with a group of the same products as one unit (e.g., a pallet). The counting device 2000 is used to determine how many target products 10 are included in a product group 20, which is one unit of target products 10 that are treated as a single unit of multiple products.

When the product group 20 is imaged by the first camera 30 and the second camera 40, the product group 20 may be transported in any manner. For example, the product group 20 may be transported on a conveyer belt. In other examples, the product group 20 may be transported by being placed on a vehicle such as a forklift or on a hand-powered cart.

However, the first camera 30 and the second camera 40 may capture images of the product group 20 in a stationary state. In this case, for example, the product group 20 is transported by any method to the location where the first camera 30 and the second camera 40 are installed, and stopped at that location for a while. Then, after the images are captured by the first camera 30 and the second camera 40, the product group 20 is transported again to another location.

<About Mark 12>
The mark 12 represents the identification information of the target product 10 included in the product group 20. Any mark that can represent the identification information and can be captured by a camera can be used as the mark 12. For example, the mark 12 is a variety of codes (e.g., a barcode or a two-dimensional code) obtained by encoding the identification information of the target product 10. Alternatively, for example, the mark 12 may be a character string (including a numeric string) that represents the identification information of the target product 10.

The mark 12 may be placed in various locations. For example, the mark 12 may be placed at any location on each target product 10. As another example, the mark 12 may be placed at any location on an object, such as a pallet, that is used to group multiple target products 10 into one product group 20.

However, the mark 12 must be placed in a position that can be imaged by the first camera 30. For example, if a mark 12 is placed on each target product 10, the mark 12 placed on at least one target product 10 must be in a state where it can be imaged by the first camera 30. Therefore, for example, it is publicly required to establish operational rules for the arrangement of the target products 10 in the product group 20 and the transportation of the product group 20 so that at least one mark 12 is included in the imaging range of the first camera 30. For example, an operational rule may be established that "when transporting the product group 20, at least one target product 10 must be placed so that the mark 12 is visible (facing outward)." The same applies when the mark 12 is placed on a pallet or the like.

Here, the method of attaching the mark 12 to the target product 10, pallet, etc. is arbitrary. For example, the mark 12 is printed on a sticker that can be attached to the target product 10, pallet, etc. In this case, the mark 12 is attached to the target product 10, pallet, etc. by attaching the sticker to the target product 10, pallet, etc. In other ways, for example, the mark 12 may be attached by directly printing the mark 12 on the outer surface of the target product 10, pallet, etc. Furthermore, the mark 12 may be attached by hand drawing instead of printing.

<Regarding the first camera 30>
The first camera 30 is used to capture the mark 12. In other words, any camera that can satisfy the condition that "the identification information of the target product 10 can be specified from the mark 12 included in the first captured image 32" can be used as the first camera 30. For example, a camera that satisfies such a condition can be a camera used for a barcode reader or a QR code (registered trademark) reader. In addition, for example, a general digital camera (for example, a digital camera provided in a mobile terminal) may be used as the first camera 30. Note that the first camera 30 may be a still camera that captures still images, or a video camera that captures moving images.

The first camera 30 is installed in a position where it can capture an image of the mark 12. For example, when the group of products 20 is being transported, the first camera 30 is installed on the transport path (the path along which the group of products 20 move). More specifically, the first camera 30 is installed near a conveyor belt or near an aisle along which a forklift or the like moves. The first camera 30 may also be installed on a forklift or the like used to transport the group of products 20.

The timing at which the first camera 30 captures images is arbitrary. For example, the first camera 30 may repeatedly capture images at regular intervals regardless of whether the mark 12 is included within its capture range. In this case, as described below, the identification unit 2020 attempts to detect the mark 12 from each of the multiple first captured images 32, and if the mark 12 is detected, identifies the identification information of the target product 10 from the mark 12.

Alternatively, for example, the first camera 30 may be configured to grasp the timing when the mark 12 enters its imaging range and capture an image at that timing. For example, when the product group 20 is transported, a sensor that detects the passage of an object (hereinafter, object detection sensor) is provided on the transport route of the product group 20. The object detection sensor is connected to the first camera 30 so as to transmit a signal to the first camera 30 when it detects the passage of an object. The first camera 30 is also set to capture an image and generate a first captured image 32 when it receives a signal from the object detection sensor. Furthermore, the position of the object detection sensor is previously determined so that the mark 12 is included in the imaging range of the first camera 30 at the timing when the product group 20 is detected by the object detection sensor. In this way, the mark 12 can be included in the first captured image 32 generated by the first camera 30.

<Regarding the second camera 40>
The second camera 40 is used to capture an image of the product group 20 and count the target products 10 contained therein. Therefore, any camera capable of generating a second captured image 42 in which the target products 10 are included in a manner that can be recognized by image analysis can be used as the second camera 40. For example, a general digital camera such as a camera provided in a mobile terminal can be used as the second camera 40.

It is preferable that the second camera 40 can capture a wider range than the first camera 30. For example, the first camera 30 is installed relatively close to the group of products 20 so that the marks 12 occupy a large portion of its angle of view. On the other hand, the second camera 40 is installed relatively far from the group of products 20 so that it can capture an image of the entire group of products 20. However, the first camera 30 does not necessarily have to be installed close to the group of products 20, and may be installed so that it can capture an image of the entire group of products 20, just like the second camera 40.

The second camera 40 is installed in a position where it can capture an image of the group of products 20. For example, when the group of products 20 is being transported, the second camera 40, like the first camera 30, is installed on the transport route, or is provided on a forklift or the like used to transport the group of products 20.

The master information 50 obtained using the first captured image 32 is used to analyze the second captured image 42. Therefore, it is preferable that the first camera 30 is provided in a position where it can capture an image of the product group 20 at an earlier timing than the second camera 40. For example, when both the first camera 30 and the second camera 40 are provided on the movement path of the product group 20, it is preferable that the first camera 30 is provided in front of the second camera 40 in the movement direction of the product group 20.

The second camera 40 may capture images at various times. For example, the second camera 40 may repeatedly capture images at regular intervals regardless of whether the product group 20 is included within its capture range. In this case, as described below, the identification unit 2020 attempts to detect the product group 20 from each of the multiple second captured images 42, and when the product group 20 is detected, counts the target products 10.

In addition, for example, the second camera 40 may be configured to grasp the timing when the product group 20 enters its imaging range and capture an image at that timing. For example, when the product group 20 is transported, the difference between the timing when the mark 12 enters the imaging range of the first camera 30 and the timing when the product group 20 enters the imaging range of the second camera 40 is known (for example, when the product group 20 moves at a constant speed). In addition, by using the object detection sensor described above, the first camera 30 captures an image at the timing when the mark 12 enters the imaging range of the first camera 30. In this case, the second camera 40 is configured to detect that an image has been captured by the first camera 30 and capture an image at a timing when the known difference described above has elapsed from the time when the image was captured by the first camera 30. The fact that an image has been captured by the first camera 30 can be grasped, for example, by transmitting a predetermined signal from the first camera 30 to the second camera 40 in response to the first camera 30 capturing an image, and the second camera 40 receiving the signal. The second camera 40 may also receive a signal from the object detection sensor described above, and capture an image when a predetermined time (such as the known difference described above) has elapsed since the signal was received. In addition, for example, an object detection sensor that determines the timing of image capture by the second camera 40 may be provided in addition to the object detection sensor that determines the timing of image capture by the first camera 30.

The second camera 40 may also capture images at a timing that avoids the influence of the image capture by the first camera 30. Specifically, the first camera 30 is configured to capture images when light is irradiated. The light used to capture images of the marks 12 in this manner may be suitable for capturing images of the marks 12, but may not be suitable for capturing images of the product group 20. For example, the influence of the light irradiated for image capture by the first camera 30 may cause color saturation in the second captured image 42, which may reduce the recognition accuracy of the target products 10 included in the second captured image 42.

Therefore, the second camera 40 is configured to capture an image when the influence of the light irradiated for capturing an image by the first camera 30 has disappeared (when the light irradiation has ended, or after a predetermined time has elapsed since the light irradiation has ended, etc.). This makes it possible to prevent the light used for capturing an image by the first camera 30 from affecting the image captured by the second camera 40, thereby improving the accuracy of the process for recognizing the target product 10 from the second captured image 42, and ultimately the accuracy of counting the target products 10.

For example, the second camera 40 is configured to capture an image a predetermined time after the first camera 30 captures an image. This predetermined time is set to a length that eliminates the influence of light used for capturing images by the first camera 30. In this case, the installation position of the second camera 40 is determined so that the product group 20 is included in the capture range at the timing when the above-mentioned predetermined time has elapsed after the first camera 30 captures an image.

The method of avoiding the influence of the light used for image capture by the first camera 30 is not limited to the above-mentioned method. For example, the second camera 40 may be installed so as to capture an image of a portion of the product group 20 that is not influenced by the light irradiated for image capture by the first camera 30 (i.e., a portion that is not irradiated with light).

<Acquisition of the first captured image 32: S102>
The identification unit 2020 acquires the first captured image 32 (S102). The method by which the identification unit 2020 acquires the first captured image 32 is arbitrary. For example, the identification unit 2020 acquires the first captured image 32 by receiving the first captured image 32 transmitted from the first camera 30. Alternatively, for example, the identification unit 2020 may acquire the first captured image 32 by accessing a storage device in which the first captured image 32 is stored. Note that the storage device in which the first captured image 32 is stored may be provided either inside or outside the first camera 30.

As described above, the first camera 30 may generate the first captured image 32 even when the mark 12 is not included in the image capture range. In this case, for example, the identification unit 2020 attempts to detect the mark 12 from each of the first captured images 32 generated by the first camera 30. Then, when the mark 12 is detected from the first captured image 32, the identification unit 2020 converts the detected mark 12 into identification information and identifies the identification information as the identification information of the target product 10 included in the first captured image 32.

<Identification of the identification information of the target product 10: S104>
The identification unit 2020 identifies the identification information of the target product 10 by using the first captured image 32 (S104). More specifically, the identification unit 2020 identifies the identification information of the target product 10 by performing image analysis on the mark 12 included in the first captured image 32.

For example, the mark 12 is a variety of codes obtained by encoding the identification information of the target product 10. In this case, the identification unit 2020 detects the mark 12 from the first captured image 32 and identifies the identification information of the target product 10 by decoding the detected mark 12. Here, existing technology can be used to analyze and decode the barcode or two-dimensional code contained in the image to identify the identification information represented by them (for example, a JAN (Japan Article Number) code represented by a barcode).

For another example, the mark 12 is a character string representing the identification information of the target product 10. In this case, the identification unit 2020 detects the mark 12 from the first captured image 32, and performs character string recognition processing (OCR: Optical Character Recognition) on the detected mark 12, thereby identifying the identification information of the target product 10 represented by the mark 12. Here, existing technology can be used as the technology for detecting a character string from an image and recognizing the detected character string.

When the mark 12 is a character string, the first captured image 32 may also contain character strings other than the mark 12 (e.g., a product name). In this case, for example, conditions for the character string representing the identification information of the target product 10 (such as a predetermined number of characters, having a predetermined prefix or suffix, or being surrounded by a figure of a predetermined shape) are determined in advance. The identification unit 2020 then identifies, from among the multiple character strings detected from the first captured image 32, a character string that satisfies this condition as one that represents the identification information of the target product 10.

<Acquisition of Master Information 50: S106>
The master acquisition unit 2040 acquires the master information 50 corresponding to the identification information of the specified target product 10 (S106). For this purpose, the identification information of the product and the master information 50 for the product are associated with each other and stored in advance in the master information storage device 60. The master acquisition unit 2040 searches the master information storage device 60 for the identification information of the target product 10 specified by the identification unit 2020, and acquires the master information 50 stored in association with the identification information.

Here, multiple product groups 20 for the same target product 10 may be subject to processing by the counting device 2000. For example, multiple pallets loaded with the same target products 10 may be brought in and inspected one after another. In such a case, the master acquisition unit 2040 may use the same master information multiple times. Therefore, the master information acquired from the master information storage device 60 may be stored in a cache, and when this master information is used again, it may be acquired from the cache, thereby making the acquisition of master information more efficient. Note that the storage device that realizes the cache may be one that requires less time for access from the counting device 2000 than the master information storage device 60, and may be provided either inside or outside the counting device 2000.

<Acquisition of second captured image 42: S108>
The counting unit 2060 acquires the second captured image 42. The method by which the counting unit 2060 acquires the second captured image 42 is similar to the method by which the identifying unit 2020 acquires the first captured image 32.

Here, as described above, the second camera 40 may repeatedly capture images regardless of whether the product group 20 is included in the capture range. In this case, the counting unit 2060 attempts to detect the product group 20 from each of the multiple second captured images 42, identifies the second captured images 42 that include the product group 20, and counts the target products 10 using the second captured images 42. For example, a second captured image 42 that does not include the target product 10 or a second captured image 42 in which a part of the target product 10 is cut off (i.e., a second captured image 42 that includes only a part of the outer surface of the product group 20 that should be included in the second captured image 42) is treated as a second captured image 42 that does not include the product group 20.

<Detection of target product 10: S110>
The counting unit 2060 detects the target product 10 from the second captured image 42 using the master information 50 acquired by the master acquisition unit 2040 (S110). Here, existing technology can be used as the technology for recognizing a specific object from an image using a template image, image features, or the like.

In cases where multiple product groups 20 pass sequentially through the imaging range of the first camera 30 and the second camera 40 (for example, where these product groups 20 are moving sequentially on a conveyer), the identification information of the target products 10 included in each product group 20 is identified. Therefore, the counting unit 2060 needs to identify the identification information of the target products 10 obtained for each of the multiple product groups 20 that has been identified for the product group 20 included in the second captured image 42 to be analyzed. In other words, the counting unit 2060 needs to associate the first captured image 32 and the second captured image 42 in which the same product group 20 has been captured.

For example, the counting unit 2060 associates the first captured image 32 and the second captured image 42 of the same product group 20 based on the generation time of each first captured image 32 and the generation time of each second captured image 42. For example, the counting unit 2060 associates the first captured image 32 and the second captured image 42 whose difference in generation time is small (a predetermined value or less). Then, the counting unit 2060 uses the master information obtained using the identification information of the target product 10 obtained from the first captured image 32 associated with the second captured image 42 for image analysis of the second captured image 42.

As another example, as described above, the first camera 30 and the second camera 40 are configured so that the first camera 30 transmits a signal to the second camera 40 in response to capturing an image by the first camera 30, and the second camera 40 captures an image in response to receiving the signal. In this case, the first captured image 32 generated by the image capturing that triggers the transmission of the signal may be associated with the second captured image 42 generated by the image capturing that is triggered by the signal.

<Identifying the number of target products 10: S112>
The counting unit 2060 identifies the number of target products 10 included in the product group 20 based on the result of recognizing the target products 10 in the second captured image 42 (S112). Here, all of the target products 10 included in the product group 20 are not necessarily included in the second captured image 42. For example, in the case of a product group 20 in which "four target products 10 are arranged vertically and three horizontally, stacked in three layers," the target products 10 hidden inside cannot be imaged by the second camera 40.

Therefore, for example, the counting unit 2060 determines the number of target products 10 included in the product group 20 based on the number and arrangement of the target products 10 recognized from the second captured image 42 and predetermined arrangement rules for the arrangement of the target products 10 in the product group 20. A specific method for determining the number of target products 10 included in the product group 20 will be described below.

Figure 5 is a diagram illustrating a case where the group of products 20 is imaged from one direction. In Figure 5, the plane parallel to the placement surface of the pallet is the xy plane, and the vertical downward direction is the z direction. The imaging direction of the second camera 40 is the y direction. Therefore, the second captured image 42 captures the group of products 20 viewed in the xz plane.

When capturing images only from the y direction in this way, it is difficult to determine from the second captured image 42 how many target products 10 are lined up in the y direction. Therefore, a rule that "N target products 10 are lined up in the y direction" is determined in advance as a placement rule for the target products 10.

The counting unit 2060 analyzes the second captured image 42 to determine how many target products 10 are present in the xz plane view. The counting unit 2060 then multiplies the number of target products 10 in the xz plane view by the number N of target products 10 in the y direction defined by the placement rule to determine the number of target products 10 included in the product group 20.

The group of products 20 may also be imaged from one direction by a second camera 40 that further has a function of generating a depth image in addition to the second captured image 42. In this case, for example, the counting unit 2060 uses the second captured image 42 and the depth image to determine the number of target products 10 as follows.

Figure 6 is a diagram illustrating a case in which the product group 20 is imaged from one direction and the second camera 40 generates a second captured image 42 and a depth image 44. In Figure 6, the imaging direction of the second camera 40 is the z direction. Therefore, from the second camera 40, it is possible to obtain a second captured image 42 showing the arrangement of the target products 10 in an xy planar view and a depth image 44 showing the depth of each position in the z direction. In the depth image 44, the color intensity of each pixel represents the depth (the distance to the object corresponding to that pixel).

The counting unit 2060 detects the target products 10 from the second captured image 42, thereby determining the arrangement of the target products 10 in the xy planar view. Furthermore, the counting unit 2060 uses the depth image 44 to determine how many target products 10 are stacked at the position where each target product 10 is placed. Then, the counting unit 2060 determines the number of target products 10 included in the product group 20 by adding up the number of target products 10 stacked at each position.

For example, the method of using the depth image 44 to identify how many objects are stacked at each position is as follows. First, by using the depth image 44, it is possible to identify the distance D from the second camera 40 to the placement surface of the pallet and the distance a to the topmost target product 10 at a certain position P. In addition, the thickness b of the target products 10 in the z direction is known in advance. By using this information, the number of target products 10 stacked at position P can be calculated as (D-a)/b. Note that the information indicating the thickness of the target products 10 in the z direction is stored in a storage device, for example, together with the master information 50, in association with the identification information of the target products 10.

In addition, instead of using the depth image 44 to grasp the number of target products 10 in the depth direction, multiple second cameras 40 may be provided and multiple second captured images 42 in which the product group 20 is captured from different directions may be used. When two second cameras 40 are provided, for example, two cameras may be provided perpendicularly to the side facing the direction of travel of the product group 20, or two cameras may be provided diagonally forward or backward in the direction of travel of the product group 20. When three second cameras 40 are provided, for example, two second cameras 40 may be provided perpendicularly to the side facing the direction of travel of the product group 20, and one second camera 40 may be provided diagonally upwards forward or backward in the direction of travel of the product group 20.

The method of capturing images of the product group 20 from different directions using multiple second cameras 40 has the advantage that it is easier to detect a situation in which a product other than the target product 10 has been mistakenly mixed in by recognizing the target product 10 from multiple second captured images 42 in which different faces of the product group 20 are captured. When it is detected that a product other than the target product 10 has been mixed in in this way, an error notification is output, for example, as described in embodiment 2 below.

<Method of utilizing the result of identifying the number of target products 10>
There are various methods for utilizing the information on the number of target products 10 included in the product group 20 identified by the counting unit 2060. Specific examples of such utilization methods will be described below.

<<Example of usage 1>>
For example, the number of target products 10 included in a product group 20 is used for managing the product group 20. In this case, the counting unit 2060 stores in a storage device the number of target products 10 included in the product group 20 in association with identification information (e.g., identification information of a pallet) of the product group 20. In this way, it becomes possible to easily manage the target products 10 that are received or shipped in units of product groups 20 (e.g., pallet units).

The method of acquiring the identification information of the product group 20 is arbitrary. For example, a mark representing the identification information of the product group 20 is attached to any position of the product group 20 (for example, any position on the outer surface of the pallet). The counting device 2000 identifies the identification information of the product group 20 from the mark representing the identification information of the product group 20 using a method similar to the method of identifying the identification information of the target product 10 from the mark 12. The identification information of the product group 20 may be captured by the first camera 30, the second camera 40, or another camera prepared other than these.

In addition, various information other than the number of target products 10 may be used as the information managed in association with the identification information of the product group 20. For example, the counting device 2000 further associates the identification information of the product group 20 with the identification information of the target products 10 contained therein. As another example, the counting device 2000 further acquires information on the attributes of the target products 10 included in the product group 20 (hereinafter, attribute information), associates the attribute information with the identification information of the product group 20, and stores the attribute information in the storage device. As the attribute information, any information such as the expiration date of the target products 10 (such as the best-before date or expiration date of food, or the expiration date of medicine) may be used.

For example, a mark (character string or code) representing the attribute information of the target product 10 is displayed on the outer surface of the target product 10. The counting device 2000 identifies the attribute information of the target product 10 by performing image analysis on an image containing this mark. For example, if a character string representing the expiration date of the target product 10 is printed on the outer surface of the target product 10, the counting device 2000 identifies the expiration date of the target product 10 by analyzing an image containing the character string and recognizing the character string. Note that the mark representing the attribute information of the target product 10 may be captured by the first camera 30, the second camera 40, or another image.

<<Example of usage 2>>
In the case where the number of target products 10 included in the product group 20 is known, the counting device 2000 may use the number of target products 10 identified by the counting unit 2060 to check whether the number of target products 10 included in the product group 20 is correct. In this case, the counting device 2000 acquires information indicating the number and range of target products 10 to be included in the product group 20, and compares the number with the number of target products 10 identified by the counting unit 2060 to determine whether the number of target products 10 identified by the counting unit 2060 is incorrect (does not match the correct number or is outside the range of the correct number). If it is determined that the number of target products 10 identified by the counting unit 2060 is incorrect, the counting device 2000 outputs a notification indicating an error. Details of the notification indicating an error will be described in the second embodiment.

<Modification>
In the examples described above, the first camera 30 and the second camera 40 are provided as separate cameras. However, one camera may be used as both the first camera 30 and the second camera 40. For example, the second camera 40 is used as the first camera 30 as well, without providing a separate first camera 30. In this case, the identification unit 2020 identifies the identification information of the target product 10 from the mark 12 included in the second captured image 42. In addition, when a plurality of second cameras 40 are provided, the mark 12 is included in the imaging range of at least one of the second cameras 40.

[Embodiment 2]
7 is a block diagram illustrating a functional configuration of a counting device 2000 according to embodiment 2. Except for the points described below, the counting device 2000 according to embodiment 2 has the same functions as the counting device 2000 according to embodiment 1.

The counting device 2000 of the second embodiment has a notification unit 2080. The notification unit 2080 outputs a notification indicating an error (hereinafter, an error notification). The types of errors output by the notification unit 2080 are described below.

<Error notification example 1>
For example, it is considered that the identification unit 2020 cannot identify the identification information of the target product 10 from the first captured image 32. In this case, the notification unit 2080 outputs an error notification indicating that the identification information of the target product 10 could not be identified. Here, cases in which the identification information of the target product 10 cannot be identified include 1) a case in which the mark 12 is not detected from the first captured image 32, and a case in which the mark 12 detected from the first captured image 32 cannot be converted into the identification information of the target product 10. Therefore, the error notification may further include information indicating whether the case is 1) or 2). Moreover, the error notification may further include the first captured image 32.

Similarly to the case where the identification information of the target product 10 cannot be identified, an error notification may be output when the identification information of the target product 10 is invalid. For example, if it is known in advance that the target product 10 to be inspected is one of three specific types of products, and the identification unit 2020 identifies identification information of a product that does not fall into any of these three types of products, it can be said that the identification information of the identified target product 10 is invalid. Therefore, the notification unit 2080 is configured to output an error notification even when the identification information of the target product 10 is invalid in this way.

<Error notification example 2>
The counting unit 2060 recognizes the target product 10 from the second captured image 42 using the master information. However, it may not be possible to recognize an object included in the product group 20 as the target product 10. This may occur, for example, when the counting unit 2060 uses the master information 50 of the target product 10 to calculate the probability that the product included in the second captured image 42 is the target product 10 (such as the similarity between the image features of the product included in the second captured image 42 and the image features shown in the master information 50 of the target product 10) and determines that the probability is equal to or lower than a predetermined value. This may occur when a part of the second captured image 42 is unclear or when an incorrect product is mixed into the product group 20.

The notification unit 2080 thus outputs an error notification indicating that an object that cannot be recognized as a target product 10 is included in the product group 20. This error notification preferably includes the second captured image 42 that includes the object that could not be recognized as a target product 10. Note that information that highlights the object that could not be recognized as a target product 10 (for example, a frame surrounding the object) may be added to this second captured image 42.

<Error notification example 3>
As described above, the counting device 2000 may acquire attribute information of the target product 10. In this case, if the counting device 2000 is unable to acquire the attribute information of the target product 10 (for example, if an error occurs in the OCR process), the notification unit 2080 may output an error notification indicating that fact.

<Error notification example 4>
As described in "Example 2 of Application Method" of the first embodiment, the counting device 2000 may check whether the number of target products 10 identified by the counting unit 2060 is correct. In this case, if the notification unit 2080 determines that the number of target products 10 is incorrect, it outputs an error notification indicating that fact. Furthermore, this error notification may indicate the number of target products 10 identified by the counting unit 2060 and the correct number or range of target products 10 that is determined in advance.

<Error notification output destination>
The output destination of the error notification is arbitrary. For example, the error notification is sent to the operator's terminal. For example, when the operator receives the error notification, the operator checks the error status by looking at the error notification. The operator may also use an image or the like included in the error notification to provide correct information to the counting device 2000. For example, when the identification information and attribute information of the target product 10 are expressed as a character string, an error notification indicating that they cannot be identified is output. In this case, the operator visually checks the image included in the error notification to identify the identification information and attribute information of the target product 10. Then, the operator inputs the identified correct identification information and attribute information of the target product 10 into the terminal and transmits it to the counting device 2000. When these pieces of information are managed in association with the identification information of the product group 20, the information input by the operator can be used to manage the identification information of the product group 20 in association with the correct information.

The output destination of the error notification is not limited to the operator's terminal, but may be, for example, output to a display device connected to the counting device 2000 or stored as an error log in a storage device.

<Example of hardware configuration>
The hardware configuration of the counting device 2000 of the second embodiment is shown in, for example, Fig. 3, similar to the hardware configuration of the counting device 2000 of the first embodiment. However, the storage device 1080 of the second embodiment stores a program for realizing each function of the counting device 2000 of the second embodiment.

A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.
1. An identification unit that analyzes a first captured image including a mark that represents identification information of a target product, and identifies the identification information represented by the mark;
a master acquisition unit that acquires master information corresponding to the identification information;
A counting device having a counting unit that detects the target products from a second captured image by analyzing the second captured image including a product group consisting of a plurality of the target products using the acquired master information, and identifies the number of the target products included in the product group based on the detection result.
2. The counting device according to 1., wherein the first captured image and the second captured image are the same image.
3. The first captured image is generated by a first camera;
1. The counting device according to claim 1, wherein the second captured image is generated by a second camera different from the first camera.
4. The first camera and the second camera are provided on a moving path of the target product,
3. The counting device according to Item 3, wherein the first camera is provided closer to the user than the second camera in a moving direction of the target products.
5. The first camera is configured to capture an image in a state where light is irradiated to generate the first captured image,
5. The counting device according to 3. or 4., wherein the second camera is controlled to capture an image at a timing when the light is not being irradiated.
6. The counting device according to any one of 1. to 5., further comprising a notification unit that outputs a notification that the second captured image includes a product that has a certainty of being the target product equal to or lower than a threshold value among the products included in the second captured image.
7. The counting device described in any one of 1. to 6., wherein the counting unit specifies the number of the target products included in the product group based on an arrangement of the target products in the second captured image and a rule for arranging products in the product group.
8. A computer-implemented control method comprising:
an identifying step of identifying the identification information represented by the mark by analyzing a first captured image including the mark representing the identification information of the target product;
a master acquisition step of acquiring master information corresponding to the identification information;
a counting step of detecting the target products from a second captured image by analyzing the second captured image, the second captured image including a product group consisting of a plurality of the target products, using the acquired master information, and determining the number of the target products included in the product group based on the detection result.
9. The control method according to 8., wherein the first captured image and the second captured image are the same image.
10. The first captured image is generated by a first camera;
8. The control method according to 8., wherein the second captured image is generated by a second camera different from the first camera.
11. The first camera and the second camera are provided on a moving path of the target product,
10. The control method according to claim 10, wherein the first camera is provided closer to the second camera in terms of a moving direction of the target product.
12. The first camera is configured to capture an image in a state where light is irradiated to generate the first captured image;
12. The control method according to 10. or 11., wherein the second camera is controlled to capture an image at a timing when the light is not being irradiated.
13. The control method according to any one of 8. to 12., further comprising a notification step of outputting a notification that the second captured image is included when the second captured image includes a product whose certainty as the target product is equal to or lower than a threshold value.
14. The control method described in any one of 8. to 13., wherein in the counting step, the number of the target products included in the product group is identified based on an arrangement of the target products in the second captured image and a rule for arranging products in the product group.
15. A program for causing a computer to execute the control method according to any one of 8. to 13.

10 Target product 12 Mark 20 Product group 30 First camera 32 First captured image 40 Second camera 42 Second captured image 44 Depth image 50 Master information 60 Master information storage device 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input/output interface 1120 Network interface 2000 Counting device 2020 Identification unit 2040 Master acquisition unit 2060 Counting unit 2080 Notification unit

Claims (7)

an identification unit that identifies a product identified by a mark based on a mark detected from a part of a plurality of products included in a captured image;
A counting device having a counting unit that identifies target products from the captured image, which are products that are determined to be the same product as the identified product, based on information regarding the image of the identified product, and counts the number of the identified target products. The mark is a bar code or a two-dimensional code.
2. The counting device according to claim 1. The information about the image is a template image of the identified product or a feature amount on the image of the identified product.
3. The counting device according to claim 1 or 2. The counting device according to any one of claims 1 to 3, further comprising a notification unit that outputs a notification that the captured image includes a product whose certainty as the target product is equal to or less than a threshold value among the products included in the captured image. The counting device according to any one of claims 1 to 4, wherein the counting unit determines the number of the target products included in the product group based on the arrangement of the target products in the captured image and a rule for arranging products in the product group consisting of a plurality of the target products. 1. A computer-implemented control method comprising:
a step of identifying a product identified by a mark based on a mark detected from a part of a plurality of products included in a captured image;
A control method comprising: a counting step of identifying target products from the captured image, which are products that are determined to be the same product as the identified product, based on information regarding the image of the identified product, and counting the number of the identified target products. A program for causing a computer to execute the control method described in claim 6.

Images