JP2018139062A - Commodity information acquisition system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a commodity information acquisition system for acquiring information of commodities displayed in a store or the like.SOLUTION: The commodity information acquisition system for acquiring information on displayed commodities, comprises: a display shelf image information analysis processing unit for executing an image analysis process on display shelf image information on which a display state of the commodities at a shop is photographed; and a commodity information storage unit for storing information on the commodities, in which the display shelf image information analysis processing unit reads a commodity code in the display shelf image information, converts the commodity code to a commodity ID corresponding to the commodity code, and specifies the commodity corresponding to the converted commodity ID from the commodity information storage unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a product information acquisition system for acquiring information on products displayed at a store or the like.

  In the retail industry, it is known that the display status of products affects the sales of products. For this reason, it is preferable that a product manufacturer / sales company can grasp and confirm where the product of the company is displayed in the store.

  Therefore, there is a system for managing the display status of merchandise using techniques as shown in Patent Document 1 to Patent Document 3 below.

JP-A-5-342230 JP-A-5-334409 International Publication 2012/029548

  The invention of Patent Document 1 is a system that supports a person who has no knowledge on which display shelves to display merchandise. For this reason, it is impossible to grasp the actual display status of products. Further, Patent Document 2 is a process for cutting out the product image displayed on the display fixture by comparing the image of each product stored in the product master with the image of the display fixture in a state where the product is displayed on the display fixture. It is an invention to execute. However, products are not necessarily displayed facing the front on display fixtures, and light, shadows, reflections, etc. may be reflected. For this reason, even if the display fixture image is compared with the image of each product, the recognition accuracy is low.

  Patent Document 3 is an invention that, when there is a vacant space in a display shelf, specifies a product to be displayed in the vacant space, or notifies the user when a product to be displayed in the display shelf is misplaced. Even in this invention, image matching processing is performed between the product and the product displayed on the display shelf, and the product displayed on the display shelf is specified. The fact is that the accuracy is low.

  In view of the above problems, the present inventors have invented a product information acquisition system that can specify products displayed on display shelves with higher accuracy than before.

  A first invention is a product information acquisition system for acquiring information on displayed products, and the product information acquisition system performs image analysis processing on display shelf image information in which a display state of products in a store is photographed. A display shelf image information analysis processing unit to be executed; and a product information storage unit for storing information related to products. The display shelf image information analysis processing unit reads a product code in the display shelf image information, and The product information acquisition system converts the product ID corresponding to the product code and identifies the product corresponding to the converted product ID from the product information storage unit.

  By comprising like this invention, the goods currently displayed on the display shelf in a store can be specified. In particular, since the displayed product is specified based on the product code, it is possible to specify the product with higher accuracy than in the case of using only conventional image recognition.

  In the above-described invention, the product information acquisition system further includes an output processing unit that outputs display status information of the product specified by the display shelf image information analysis processing unit, and the display shelf image information analysis processing unit Further, it can be configured as a product information acquisition system for specifying display status information indicating the display status of the specified product.

  By outputting the display status information indicating the display status of the specified product, it is possible to grasp the display status of the product.

  In the above-described invention, the product information acquisition system further includes a sample image information storage unit that stores sample image information of a product displayed on a display shelf, and the display shelf image information analysis processing unit includes: The product code in the display shelf image information is read, converted into a product ID corresponding to the product code, the product corresponding to the converted product ID is specified from the product information storage unit, and the display shelf image information or the display shelf By executing an image matching process between the corrected image information obtained by performing the correction process on the image information and the sample image information stored in the sample image information storage unit, a candidate for the product displayed on the display shelf is identified. , It is determined whether the candidate product specified by the image matching process matches the product with the product ID converted from the product code, and the matched product Identifying a merchandise displayed on the shelf can be configured as product information acquisition system.

  In the case of specifying a product by image matching processing, candidates are generally output in descending order of similarity. However, if there is a product with a similar appearance, the similarity may be close. For this reason, it is not possible to specify which product. Therefore, by comparing the product corresponding to the product ID obtained by converting the product code as in the present invention with the product that is a candidate by the image matching process, the accuracy of specifying the displayed product can be improved.

  In the above-described invention, the display shelf image information analysis processing unit matches the candidate product specified by the image matching processing as a result of determining whether the product with the product ID obtained by converting the product code matches. When there is no product, it can be configured as a product information acquisition system that determines that the product with the product ID is a missing product.

  When identifying a product that is displayed only by image matching processing as in the past, if the product is a missing item, the product does not exist in the image information obtained by photographing the display shelf, so which product is displayed. It is not easy to identify what was happening. For example, the product tag information may be specified by OCR recognition. However, in the case of image information in which the display shelf is copied in a wide range, it is extremely difficult to recognize the product tag information. On the other hand, since the displayed product can be identified by using the product code as in the present invention, if the image matching candidate and the product based on the product code do not match, there is a possibility that the product is missing. high. Therefore, by configuring as in the present invention, it is possible to determine that the displayed product is a missing item.

  In the above-described invention, the product information acquisition system further includes a sample image information storage unit that stores sample image information of a product displayed on a display shelf, and the display shelf image information analysis processing unit includes: The product code in the display shelf image information is read, converted into a product ID corresponding to the product code, the product corresponding to the converted product ID is specified from the product information storage unit, and the sample image information of the specified product is obtained. Extracting from the sample image information storage unit, and executing the image matching process between the extracted sample image information and the corrected shelf image information or the corrected image information that has been corrected for the display shelf image information, It can be configured like a product information acquisition system that identifies the products displayed on the display shelf.

  The image matching process using the sample image information requires system load and processing time. Therefore, as in the present invention, by performing image matching processing with sample image information based on a product ID converted from a product code, the load on the system and the processing time can be reduced. In addition, since the image matching process with the sample image information of the product that is highly likely to be displayed is performed, the accuracy of specifying the displayed product can be improved.

  In the above-mentioned invention, when the display shelf image information analysis processing unit executes image matching processing between the extracted sample image information and the display shelf image information or the corrected image information, and does not satisfy a predetermined condition Can be configured as a product information acquisition system that determines that the product with the product ID is a missing product.

  When identifying a product that is displayed only by image matching processing as in the past, if the product is a missing item, the product does not exist in the image information obtained by photographing the display shelf, so which product is displayed. It is not easy to identify what was happening. For example, the product tag information may be specified by OCR recognition. However, in the case of image information in which the display shelf is moved over a wide range, it is extremely difficult to perform OCR recognition on the product tag information. Since the product with the product ID specified from the product code is a product that has a high possibility of being displayed, if the sample image information and the image matching process are performed and the predetermined condition is not satisfied, the product is There is a high possibility of a shortage. Therefore, by configuring as in the present invention, it is possible to determine that the displayed product is a missing item.

  In the above-described invention, a commodity code indicating a missing item is attached to the rear panel of the display shelf, and the commodity attached to the rear panel in the display shelf image information in the display shelf image information analysis processing unit. When the code is read, it is configured as a product information acquisition system that determines that the product with the product ID corresponding to the product code at the position corresponding to the product code attached to the back panel is missing. be able to.

  When the product is missing, the back panel of the display shelf is generally reflected. Therefore, it is possible to determine that the product is a missing item by attaching a product code indicating a missing item to the rear panel of the display shelf and reading it.

  In the above-mentioned invention, the display shelf image information analysis processing unit includes the specified product information, shelf position information, and face number information as display status information indicating the display status of the specified product, and the output The processing unit can be configured as a product information acquisition system that displays the number of product faces as the display status information according to the shelf position.

  With the configuration of the present invention, the number of faces of the specified product can be displayed, which can contribute to the sales strategy of the product.

  In the above-mentioned invention, the display shelf image information analysis processing unit includes the specified product information, shelf position information, and face number information as display status information indicating the display status of the specified product, and the output The processing unit can be configured as a product information acquisition system that outputs the display status information of the product together with information on a shelf of the golden line of a predetermined display shelf.

  By configuring as in the present invention, the number of faces of the specified product can be displayed in relation to information on the shelf of the golden line. Since it is important to be displayed on the Golden Line as a product sales strategy, it can contribute to the product sales strategy.

  The same technical effect can be obtained by configuring the first invention as in the present invention. That is, a product information acquisition system for acquiring information on displayed products, wherein the product information acquisition system performs correction processing on display shelf image information in which the display state of products in a store is photographed. A product that stores a shelf position designation processing unit that accepts a shelf position designation process for the display shelf image information or the corrected image information that has undergone the compensation process, and a product ID and product information An information storage unit, and a product code conversion processing unit that converts a product code in the display shelf image information or the corrected image information into a product ID, and specifies product information corresponding to the product ID based on the product information storage unit And a display status information processing unit for determining the display status information of the specified product by specifying the position of the specified product on the display shelf, and the display status An output processing unit that outputs the broadcast can be configured as commodity information acquisition system comprising a.

  In the above-described invention, the product code can be configured as a product information acquisition system attached to a product tag of a product displayed on a display shelf.

  The product code is preferably attached to the product tag. This is because product tags are frequently attached to the front side of the display shelf.

  In the above-described invention, the product code can be configured as a product information acquisition system which is a color code.

  Various codes can be used as the product code, but the color code can be recognized and converted even if the size of the entire image is about 2%. Therefore, the display shelf can be photographed in a wide range, the number of display shelf image information to be photographed can be reduced, and it is difficult to disturb the store when photographing. Furthermore, if the color code is attached to the product tag, it can be printed at the same time as the product tag is printed, so that the store is not burdened when creating the product tag.

  The first invention can be realized by reading the computer program of the present invention into a computer and executing it. That is, a product information acquisition program that causes a computer to function as a display shelf image information analysis processing unit that executes image analysis processing on display shelf image information in which the display state of products in a store is photographed, wherein the display shelf image information The analysis processing unit reads the product code in the display shelf image information, converts the product code into a product ID corresponding to the product code, and stores the product corresponding to the converted product ID from a storage area storing information about the product. This is a product information acquisition program to be specified.

  By using the product information acquisition system of the present invention, the products displayed on the display shelves can be identified with higher accuracy than before. Therefore, the display status of each product in the store can be acquired with high accuracy.

It is a conceptual diagram which shows typically an example of the whole structure of the merchandise information acquisition system of this invention. It is a conceptual diagram which shows typically an example of a structure of the display fixture image analysis process part in the merchandise information acquisition system of this invention. It is a conceptual diagram which shows typically an example of the hardware constitutions of the computer used with the merchandise information acquisition system of this invention. It is a flowchart which shows typically an example of the process of a goods tag in the goods information acquisition system of this invention. It is a flowchart which shows typically an example of the processing process of registration of the display fixture image in the merchandise information acquisition system of this invention. It is a flowchart which shows typically an example of the processing process of the display fixture image analysis in the goods display image acquisition system of this invention. It is a conceptual diagram which shows an example of a merchandise information storage part typically. It is a conceptual diagram which shows an example of a color bit typically. It is a figure which shows typically an example of a normal product tag and a product tag attached with a product code. It is a figure which shows typically the state in which the product tag was attached to the display shelf. It is a figure which shows typically an example of display shelf image information. It is a figure which shows an example of the screen display of the state which read the extracted display shelf image information. It is a figure which shows an example of the screen display of the state which performs the trapezoid correction process with respect to the extracted display shelf image information. It is a figure which shows an example of the screen at the time of receiving the position specification of a shelf in corrected image information. It is a figure which shows typically an example of a structure of the merchandise information acquisition system in Example 2. FIG. It is a figure which shows typically an example of a structure of the display shelf image information analysis process part of the merchandise information acquisition system in Example 2. FIG. 10 is a flowchart schematically illustrating an example of a processing process in the second embodiment. It is a figure which shows an example of the screen display of the state which specified the face. It is a figure which shows an example of the screen display which displays the recognition result of goods. It is a figure which shows typically an example of the display shelf image information analysis process part in Example 3. FIG. It is a figure which shows typically an example of the display shelf in Example 5. FIG.

  An example of the entire system configuration of the product information acquisition system 1 of the present invention is shown in FIG. Moreover, the conceptual diagram of an example of a structure of the below-mentioned display shelf image information analysis process part 22 in the merchandise information acquisition system 1 is shown in FIG.

  The product information acquisition system 1 is realized in a management server 2 which is a computer used by an organization such as a company that operates the system. FIG. 3 schematically shows an example of the hardware configuration of the computer. The computer is capable of inputting information, such as an arithmetic device 70 such as a CPU for executing arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk for storing information, a display device 72 such as a display for displaying information, and the like. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives processing results of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

  Each means in the present invention is only logically distinguished in function, and may be physically or virtually the same area.

  The management server 2 in the product information acquisition system 1 grasps the display status of the product by the photographer terminal 3 which is a computer used by the photographer who captures the display status of the product in a store such as a retail store and the product information acquisition system 1. Information can be transmitted to and received from the system user terminal 4, which is a computer used by a system user who desires to do so, via a network.

  The management server 2 in the product information acquisition system 1 includes a display shelf image information reception processing unit 20, a display shelf image information storage unit 21, a display shelf image information analysis processing unit 22, a product information storage unit 23, and a display status information storage unit 24. And an output processing unit 25.

  The display shelf image information reception processing unit 20 receives, from the photographer terminal 3, image information (display shelf image information) obtained by photographing a display shelf displaying products at a store. The display shelf image information received here is preferably associated with store identification information (information for identifying a store such as a store name) and shooting date / time information. That is, it is preferable that the photographer terminal 3 accepts the input of the store identification information and the shooting date / time information of the store where the display shelf image information is shot, and the association is performed. Display shelf image information is usually taken for each store, so it is common to accept input of multiple display shelf image information per store, but it accepts input as a single moving image or still image. Also good.

  The display shelf image information storage unit 21 stores the display shelf image information received by the display shelf image information reception processing unit 20. The display shelf image information may be associated with store identification information, shooting date / time information, and the like.

  The display shelf image information analysis processing unit 22 executes an image analysis process described later on the display shelf image information received by the display shelf image information reception processing unit 20 to thereby provide a product code attached to a product tag or a display shelf. And the product ID displayed at that position is associated with the display position of the product on the display shelf. The product ID is information (product identification information) for identifying the product displayed on the shelf of the display shelf, and includes, for example, the JAN code itself or information associated with the JAN code. The product ID may include symbols and the like in addition to numbers and letters. In addition to the information for identifying the product, the product ID may include information indicating the price of the product such as a price at the retail store or a desired retail price. The product code is information in which the product ID is coded by a predetermined method.

  As the product code, a method of displaying information using colors, a method of printing predetermined information with ink that cannot be visually recognized by humans, or the like can be used. For example, as a method of displaying information using colors, there are color codes (color barcodes) such as color bits, chameleon codes, High Capacity Color Barcode, Colorzip, and color dot code (ADEU.NEK). A color code is a code composed of an enumeration of colors used to express information. A color bit, which is an example of a color code, expresses information by changing adjacent colors (color arrangement order). A chameleon code, which is an example of a color code, expresses information by using a combination of cyan, magenta, and yellow colors. Note that the color code is not limited to the above example, as long as it is a code composed of an enumeration of colors used to express information. In the following description, a case where a color bit which is an example of a color code is used as a product code will be described.

  The display shelf image information analysis processing unit 22 converts the product code read from the display shelf image information into a product ID according to a predetermined method. Then, the converted product ID is associated with the position of the display shelf to which the product code is attached, and it is determined that the product corresponding to the converted product ID is placed at the position of the display shelf.

  The display shelf image information analysis processing unit 22 includes a display shelf image information correction processing unit 220, a shelf position designation processing unit 221, a product code conversion processing unit 222, and a display status information processing unit 223.

  The display shelf image information correction processing unit 220 executes image analysis processing on the display shelf image information received by the display shelf image information reception processing unit 20 or the display shelf image information stored in the display shelf image information storage unit 21. The correction process necessary for the process is executed. As the correction process, for example, a trapezoidal correction process or, when one display shelf is divided into a plurality of display shelf image information, a process of combining them into one display shelf image information is executed. An image that has been subjected to the correction process in the display shelf image information correction processing unit 220 will be referred to as corrected image information.

  The product code conversion processing unit 222 recognizes the product code from the display shelf image information and converts the product code into the product ID by a predetermined method when the corrected image information or the correction process is not executed. Further, the product information corresponding to the converted product ID is specified by referring to a product information storage unit 23 described later. The process of recognizing the product code from the image information is a process of identifying the product code area from the image information and converting the product code into a product ID by using a conversion method corresponding to the product code using information displayed in the product code area Should be executed. For example, when a color code is used as a product code, the color code in the image information may be recognized and converted by a known method.

  The display status information processing unit 223 displays the product ID converted by the product code conversion processing unit 222 and the position information on the display shelf on which the product is displayed (the number of the shelf, the right or left of the shelf) For example, coordinate information indicating how far the reference point is). Then, the association information is stored in the display status information storage unit 24 described later.

  As shown in FIG. 7, the product information storage unit 23 stores information (product information) related to products such as a product ID, JAN code, product name, rating, and manufacturer. The product information may include price information such as sales price and suggested retail price.

  The display status information storage unit 24 stores, as display status information, information associated with the product ID determined by the display shelf image information analysis processing unit 22 and the position of the product on the display shelf.

  The output processing unit 25 receives an output request from the system user terminal 4 and outputs display status information.

  Next, an example of the processing process of the product information acquisition system 1 of the present invention will be described with reference to the flowcharts of FIGS.

  First, as a premise for using the product information acquisition system 1, a process for attaching a product code such as a color code to a product tag attached to a display shelf will be described. The person in charge who attaches the product code to the product tag operates a predetermined computer (not shown) and inputs a product ID that can uniquely identify the product. The code is coded by a predetermined code conversion method (S100).

  As shown in FIG. 7, for a product for which the product ID is set to “123456”, the product ID “123456” is converted into a binary value by a predetermined method. In this conversion, it is preferable to perform conversion after adding a check bit (correction code) for improving the accuracy of data. For example, it is assumed that “00000001001011000011111100011” is generated as a binary value when “123456” is converted. The generated binary value is coded by generating color bits according to the correlation defined by the predetermined color bits. An example of the correlation between the numerical value and the change in color is shown in FIG. 8A, and a display example of binary values by color bits is shown in FIG. 8B. FIG. 8C shows an example in which the generated binary value is represented by color bits. In FIG. 8 (c), the squares are not horizontal but are deformed, but the cells are continuous. In the case of a color bit, it is specified that the first cell starts with “red” and the last cell ends with “green” or “blue”. However, other provisions may be provided, and the present invention is not limited to these provisions. Therefore, “0” or “1” is indicated by the change of the adjacent color successively from the “red” square. For the sake of drawing, a pattern is displayed instead of a color, but the corresponding color is actually displayed.

  The product code generated as described above is subjected to a predetermined operation on the computer, so that the printing apparatus prints the product code on the product tag (S110). The product code is preferably printed in a surplus space (blank) in the product tag where no information about the product is displayed. The product tag displays the product name, price, etc. of the product to the consumer, and is attached in the vicinity of the corresponding product on the display shelf, for example, below and above. FIG. 9A shows an example of a normal product tag, and FIG. 9B shows an example of a product tag printed with a product code.

  The product tag on which the product code is printed in S110 is attached in the vicinity of the position where the product is displayed on the display shelf, for example, above or below (S120). FIG. 10 is a diagram schematically illustrating a state in which the product tag is attached to the display shelf.

  As described above, the merchandise tag printed with the merchandise code is attached to the display shelf of the store, and a predetermined photographer can photograph the display shelf.

  Next, a processing process for photographing a store display shelf will be described.

  First, a predetermined photographer goes to a store and photographs display shelves with a photographing device such as a camera, thereby acquiring display shelf image information (S200). An example of display shelf image information is shown in FIG. The acquired display shelf image information is read into the photographer terminal 3 used by the photographer, and the display shelf image information and the store identification information are associated with each other by inputting the identification information of the store (S210). When the photographing device is a portable communication terminal such as a smartphone or a tablet computer, the photographing device and the photographer terminal 3 may be configured integrally. When there are a plurality of display shelves, it is preferable to photograph each display shelf. Further, identification information for identifying the display shelf may be input, and the display shelf image information and the display shelf identification information may be linked. In addition, it may be associated with display shelf image information by receiving input of shooting date and time information.

  Then, when the photographer performs a predetermined operation on the photographer terminal 3, the display shelf image information, the store identification information, the display shelf identification information, and the photographing date / time information are uploaded to the management server 2 (S220). The display shelf image information uploaded from the photographer terminal 3 is received by the display shelf image information reception processing unit 20, and the display shelf image information, store identification information, display shelf identification information, shooting date and time information, etc. are associated with each other. The information is stored in the information storage unit 21. If the upload is successful (S230), the process is terminated. If the upload is not successful (S230), the upload is performed again.

  Next, analysis processing using the uploaded display shelf image information in the management server 2 will be described.

  Display shelf image information sent from the photographer terminal 3 is received by the display shelf image information reception processing unit 20 and stored in the display shelf image information storage unit 21. Then, at a predetermined timing, the system user performs a predetermined operation on the system user terminal 4 to extract the display shelf image information stored in the display shelf image information storage unit 21 to be processed, and displays the display shelf image. Analyzing the display status information based on the information. In this case, the display fixture image analysis processing unit may extract display shelf image information of the corresponding store by receiving store identification information, shooting date / time information, and the like from the system user terminal 4, for example.

  First, the extracted display shelf image information is read and displayed on the screen shown in FIG. 12, and correction processing for the display shelf image information is executed (S300). In other words, the keystone correction process is executed for the display shelf image information so that the shelf level of the display shelf is horizontal and the product is vertical.

  In order to execute the trapezoidal correction process, as shown in FIG. 12, a rectangle (a rectangle indicated by a broken line in FIG. 12) is superimposed on the display shelf image information, and an input that matches the four points of the rectangle to the processing target range. The display shelf image information correction processing unit 220 receives an operation from the system user terminal 4. For example, in FIG. 12, the same display shelf image information extracted is displayed on the left and right, and on the left side is a rectangle for the system user to input keystone correction (a rectangle for the system user to move four points). , A rectangle indicated by a broken line in FIG. The right display shelf image information includes an image after trapezoidal correction processing performed based on the movement of four points of a quadrangle (rectangle indicated by a solid line in FIG. 12) displayed superimposed on the left display shelf image information. Information is displayed. Therefore, each point of a rectangle (rectangle indicated by a broken line) displayed superimposed on the left display shelf image information is a vertical line of a rectangle (rectangle indicated by a solid line) displayed superimposed on the right display shelf image information. , Move to fit the horizon. FIG. 13 shows an example of a screen when four points of a quadrangle (a quadrangle indicated by a broken line) to be displayed superimposed on the left display shelf image information are moved. Each rectangle point displayed on the display shelf image information on the left side (rectangle indicated by a broken line) is displayed on the right side of the display shelf image information on the right side (rectangle indicated by a solid line). After moving to a position that matches the line, the “trapezoid correction confirmation” button on the screen of FIG. 13 is pressed.

  When the display shelf image information correction processing unit 220 receives the “trapezoid correction confirmation” button from the system user terminal 4, the display shelf image information correction processing unit 220 executes trapezoid correction processing on the display shelf image information. , Display shelf image information after correction processing (corrected image information) is displayed on the screen.

  Then, a shelf position designation process is performed for the corrected image information (S310). As the shelf position designation process, the shelf position designation processing unit 221 accepts the designation of the shelf position from the system user terminal 4 in the area where the shelf position designation process is performed in the corrected image information. . That is, designation of an area where a product exists for each shelf is accepted. Then, the identification information of the number of stages is sequentially attached to each shelf area as in the first and second stages. In this case, input of identification information for the number of stages may be accepted, or identification information for the number of stages may be automatically added sequentially in a specified order. FIG. 14 shows an example of a screen when the shelf position designation is received in the corrected image information. Note that FIG. 14 shows a case where an area for which position designation has been received is enlarged and displayed in the corrected image information. Then, when the shelf position designation processing unit 221 accepts the designation of the shelf position, an area that is not designated as a shelf is determined as an area with a product tag.

  As described above, when the shelf position in the corrected image information is designated in the shelf position designation processing unit 221, the product code conversion processing unit 222 applies the product tag to an area not designated as a shelf. Is read (S320), and the product code is converted into the product ID according to the conversion method of the product code and the product ID (S330). In the process of converting the product code into the product ID, image information in the product code area is extracted from the image information to be analyzed, and the information is converted into the product ID according to the product code conversion method. For example, when a color bit is used as a product code, information is converted in accordance with a change in color of adjacent cells in order from a reference cell, for example, a “red” cell. In addition, as described above, when reading the product code, in addition to the region not designated as a shelf, the region designated as the shelf of the corrected image information or the display shelf image information, Processing may be performed over the entire image information.

  When color bits are used as the product code, information can be converted if the size is about 2% of the image. Therefore, for example, even when the corrected image information to be analyzed is a plurality of products displayed as shown in FIG. 14, the product code can be sufficiently converted to the product ID. In the case of color bits, even if a plurality of color bits (product codes) are included in one piece of image information, each can be converted into a product ID, so that a plurality of conversion processes can be performed simultaneously.

  When the product code read from the image information (corrected image information or display shelf image information) to be analyzed as described above is converted into a product ID, the product code conversion processing unit 222 corresponds to the converted product ID. The product information is extracted with reference to the product information storage unit 23.

  Then, the display status information processing unit 223 specifies the position information of the product tag attached with the product code corresponding to the converted product ID. In other words, the position of the shelf and the position from the left and right base points are specified. The number of shelf levels can be specified based on the information on the number of shelf levels received by the shelf position designation processing unit 221. Further, the left and right positions can be specified based on, for example, relative coordinates.

  When the position information is specified as described above, the display status information processing unit 223 associates the product information extracted in S330 with the product ID and the position on the display shelf specified above, and displays the display status information storage unit as display status information. 24 (S340, S350). At this time, the date / time information of the display shelf image information to be processed and the store identification information may be stored in association with each other.

  When the system user terminal 4 obtains display status information at a certain point in time for a certain store by performing a predetermined operation, the display status information on the display shelf of the product is based on the store identification information and the date / time information. The output processing unit 25 can output to the system user terminal 4 (for example, which product is being sold at which position of which display shelf). Also, by specifying the period, the output processing unit 25 extracts the display status information in the display shelf of the product in the specified period, which is stored in the display status information storage unit 24, and sums them up in time series. Changes over time can be output by organizing and outputting. Furthermore, by receiving the designation of a specific product ID, the output processing unit 25 can further extract and output display status information of the designated product ID.

  In this way, the product tag on which the product code is printed is attached to the display shelf, the display shelf image information obtained by capturing the product tag is photographed, and the analysis process is executed to obtain information on the product displayed at a certain point in time. be able to. In addition, changes over time can be grasped.

  Next, as a different embodiment of the present invention, a case will be described in which a product displayed in the display shelf image information is specified, and the number of faces of the product is specified as display status information. The number of faces is information indicating how many of the same product are arranged horizontally on the same shelf (the number of exposures in the horizontal direction). The larger the number of faces, the more products are exposed and the more the sales floor space is secured. Therefore, it is important information for merchandise sales companies.

  The shelf at the position that is the line of sight of the purchaser of the product is called the golden line, and is a particularly important shelf among the display shelves. This is because placing on the Golden Line makes it easier for buyers to see, increasing product recognition and increasing the likelihood of purchase. For this reason, merchandise sales companies encourage stores to place their merchandise on the shelves of the Golden Line, but it has traditionally been necessary to objectively grasp how much they have been placed. I couldn't. Therefore, in this embodiment, as a modification of the first embodiment, the number of product faces can be output as display status information for each position of the shelf, so that the number of product faces can be grasped, and so It becomes possible to grasp whether or not the product was displayed.

  The product information acquisition system 1 of the present embodiment further includes a sample image information storage unit 26, and the display shelf image information analysis processing unit 22 includes a product identification processing unit 224. An example of the configuration of the product information acquisition system 1 in the present embodiment is shown in FIG. 15, and an example of the configuration of the display shelf image information analysis processing unit 22 is shown in FIG.

  After accepting the shelf position designation, the product identification processing unit 224 accepts pressing of the “automatic face extraction” or “face correction” button on the screen of FIG. Is requested by the product identification processing unit 224 and executed.

  The product identification processing unit 224 identifies a face in an area designated as a shelf in the corrected image information or display shelf image information, and further executes a product recognition process from the identified face. And the process which specifies the number of faces about the same goods is performed. The face is an area where products are displayed (exposed).

  There are various face identification processes, which can be switched depending on the type and form of the product. (1) Search for shadows between faces, (2) Is there a repeated image area? Search for patterns, (3) When a certain face is specified, use product features such as the product width is almost constant, and specify the face sequentially for each product width, (4) For each product type Since the product width is determined, the face can be specified by combining one or more of (1) to (4) such as specifying a face for each product width. Also, methods other than (1) to (4) can be used.

  The product identification processing unit 224 displays the identified face area as shown in FIG. 18 for the identified face.

  Next, for each identified face area, image matching processing is performed between the sample image information of each product stored in advance and the image information of the face area, and the product displayed in the face area is recognized. Execute the process. The sample image information of each product is image information taken from a plurality of angles in advance for each product. For example, it is preferable to use the number of pieces of image information that only shows the side surface of the product in any sample image.

  That is, for each identified face unit, sample image information of each product stored in the sample image information storage unit 26 described later and image matching processing are performed, and processing for identifying the product based on the similarity is executed. For example, the degree of similarity may be 75% with the A product, and the degree of similarity with the B product may be 25%, or the product with the highest similarity may be identified as the product on the face. Then, a product ID of the identified product is specified with reference to a sample image information storage unit 26 described later. Note that various processing methods can be used as the image matching processing. For example, processing can be performed by comparing the similarities using feature amounts.

  When a plurality of candidates are listed in descending order of similarity, the product specification processing unit 224 uses the product ID converted from the product code attached to the product tag in the product code conversion processing unit 222 and the product specified by the image matching process. Comparison with the ID is performed, and the matching product ID is specified as the product displayed in the area of the face. In addition, when it does not correspond with any goods, you may handle as "Unknown" thru | or a missing item (The goods should have been displayed if it was originally displayed, but a goods does not exist by selling out etc.). For example, a predetermined notification may be displayed on the screen as “unknown” or “out of stock”, or specified as “out of stock”. Alternatively, an input for designating “out of stock” for the face area may be received from the system user terminal 4.

  Then, for each face, the product identification processing unit 224 calculates the number of faces for each product after executing product recognition processing by image matching processing between the face area and the sample image information of each product. That is, it calculates how many products having the same product ID are arranged in the horizontal direction. Since the products in the face area have already been identified, the number of faces for each product can be calculated by calculating how many products with the same product ID identified for each face are arranged. Note that the position (position information) of the product in each face is also specified. The face of “out of stock” may be counted as the number of faces as it is, or may be processed as “out of stock” (“0” as the number of faces). FIG. 19 shows an example of a screen that displays the product recognition result.

  The display status information processing unit 223 displays the number of faces calculated by the product identification processing unit 224 as described above, the product ID, the number of shelves on which products on the display shelf are displayed, and the position from the right or left on the shelf. The information is associated with each other and stored in the display status information storage unit 24 as display status information.

  The sample image information storage unit 26 stores sample image information of products that may be displayed on the display shelf. As the sample image information, it is preferable to store a plurality of pieces of image information taken from different angles for one product. The sample image information may be associated with information for identifying a product such as a product ID, JAN code, or product name.

  Note that the following processing method can also be used for the second and subsequent processes for the same display shelf in the product identification processing unit 224 in order to further speed up the process. In the second and subsequent processing, there is a precondition that the display shelf image is photographed in the same area at approximately the same angle. If this is not satisfied, the same area is approximately the same. After the correction process is performed so that the angle is obtained, the second and subsequent processes described later are performed.

  As the second and subsequent processing, the product identification processing unit 224 is the second and subsequent processing on the display shelf image information extracted as the correction processing based on the store identification information, display shelf identification information, and shooting date / time information. Is determined, the four points of the quadrangle whose specification was accepted last time are automatically determined, and the keystone correction process is executed. In the shelf position designation process, the area where the designation was accepted last time is automatically determined as the area of each shelf, and the face area is automatically specified.

  Then, the product recognition processing is executed by comparing the image information specifying the face area last time with the image information specifying the face area this time. For example, for each face, the similarity between the previous image information and the current image information is obtained, and when there is sufficient similarity (there is a similarity equal to or greater than a predetermined threshold), the previously determined product ID Is identified as the same product. At this time, since the product position may be shifted, it is preferable to compare with the image information of faces within a predetermined number of ranges as a comparison target.

  By performing as described above, the second and subsequent processes can be executed automatically or with a reduced processing procedure.

  Next, an example of a processing process in the display shelf image information analysis processing unit 22 will be described with reference to a flowchart of FIG. Note that the processing from S400 to S430 is the same as the processing from S300 to S330 in the first embodiment, and a description thereof will be omitted.

  The display shelf image information analysis processing unit 22 converts the product code attached to the product tag into a product ID in S430, extracts the corresponding product information from the product information storage unit 23, and then the product identification processing unit 224 The specific process is executed (S440). That is, in the corrected image information or display shelf image information, (1) search for shadows between faces in an area designated as a shelf, (2) a pattern indicating whether there is a repeated image area Search, (3) When a certain face is specified, use product features such as the product width is almost constant, and specify the face sequentially for each product width. (4) Product width for each product type Therefore, the face area is specified by combining one or more of (1) to (4) such as specifying a face for each product width.

  After identifying the face, the product identification processing unit 224 performs image matching processing with the sample image information of each product stored in the sample image information storage unit 26 for each area of the face, and is displayed in the area of the face. A recognition process for the existing product is executed (S450).

  After recognizing the product in this way, the product identification processing unit 224 calculates the number of faces for each product by calculating how many products having the same product ID are arranged in the horizontal direction (S460), The shelf in the display shelf on which the product is displayed, and the horizontal position (position information) in the shelf are specified.

  The display status information processing unit 223 stores the product ID recognized in S450, the number of faces, and the specified position on the display shelf in the display status information storage unit 24 as display status information (S470, S480). At this time, the date / time information of the display shelf image information to be processed and the store identification information may be stored in association with each other.

  When the system user terminal 4 obtains display status information at a certain point in time for a certain store by performing a predetermined operation, the display status information on the display shelf of the product is based on the store identification information and the date / time information. Can be output to the system user terminal 4 by the output processing unit 25. Also, by specifying the period, the output processing unit 25 extracts the display status information in the display shelf of the product in the specified period, which is stored in the display status information storage unit 24, and sums them up in time series. Changes over time can be output by organizing and outputting. Furthermore, by receiving the designation of a specific product ID, the output processing unit 25 can further extract and output display status information of the designated product ID.

  In addition, by designating which shelf level is the golden line for each display shelf in advance, display status information on the golden line, or display status information on each display shelf, and which display shelf is the golden line It is possible to output information together.

  In this way, the product tag on which the product code is printed is attached to the display shelf, the display shelf image information obtained by capturing the product tag is photographed, and the analysis process is executed to obtain information on the product displayed at a certain point in time. be able to. In addition, changes over time can be grasped. Further, since the displayed product is identified based on the display shelf image information of the store, even if the product is missing, it can be specified. That is, if the product ID can be recognized from the product code of the product tag but the product ID cannot be specified in the face area, the product can be processed as a missing item.

  In Example 1 and Example 2, when the price of a product is not specified based on the product code printed on the product tag, the product name, price, and rating are calculated by performing image analysis processing on the product tag. May be recognized.

  In this case, the display shelf image information analysis processing unit 22 further includes a product tag recognition processing unit 225. An example of the display shelf image information analysis processing unit 22 in this embodiment is shown in FIG.

  The product tag recognition processing unit 225 identifies a product tag region by recognizing a predetermined closed region, preferably a rectangular region, for an area that is not the shelf position that has been designated by the shelf position designation processing unit 221. Then, product information such as price, product name, rating, etc. is extracted from the identified product tag area by pattern matching with OCR or numeric font.

  The product code conversion processing unit 222 stores the product information such as price, product name, and rating recognized by the product tag recognition processing unit 225 and the product ID converted by the product code conversion processing unit 222 in the product information storage unit 23. Compared with the stored product information, if the product information such as the product name and rating match and the price is different, the price recognized by the product tag recognition processing unit 225 is preferentially handled as product information. Also good. In this case, the price information of the product stored in the product information storage unit 23 may be updated. By such processing, even when the price information that is easily changed is updated, the price information stored in the product information storage unit 23 can also be updated. Note that the product information subjected to the above comparison in the product code conversion processing unit 222 is stored in the display status information storage unit 24 as display status information by the display status information processing unit 223 as in the first and second embodiments. Let

  Next, in the second embodiment and the third embodiment, processing when the product code attached to the product tag is used in the image information matching processing in the display shelf image information analysis processing unit 22 will be described.

  For example, in the product recognition processing in S450, the product identification processing unit 224 performs product code conversion processing when matching image information between the face area image information and the sample image information stored in the sample image information storage unit 26. The sample image information of the product corresponding to the product ID converted by the unit 222 is preferentially used as a candidate for image information matching processing. And if it is a similarity of a certain threshold or more as a result of this matching process, it will identify with the goods of the said goods ID.

  In addition, since there may be a slight gap between the face area and the product tag area, the product code attached to the product tag located near the face is converted into the product code conversion process. It is good to go in order from the product ID converted by the unit 222. That is, based on the proximity of the distance between the face and the product tag, the priorities of the candidate product IDs are ranked, and the sample image information that performs image information matching processing may be used as a candidate.

  Generally, processing time and system load are generated in image information matching processing, and the processing time and system load can be greatly reduced by the configuration of the present invention. As a result of matching the sample image information of the product ID based on the product code and the image information of the face area, if the threshold does not satisfy the predetermined condition, the product of the product ID (the product corresponding to the product tag) ) Is not displayed, that is, it can be determined as a missing item.

  Further, the product tag recognition processing unit 225 may perform the product tag recognition processing when executing the processing of the product code conversion processing unit 222. Further, when the product code conversion processing unit 222 recognizes the product code, a closed region including the product code, preferably a rectangular region may be recognized as the product tag region. When the product ID is associated with the price in the product information storage unit 23, the processing may not be performed, or the price information stored in the product information storage unit 23 is preferentially processed by the OCR process. , May be used for image matching processing.

  In each of the above-described embodiments, the case where the product code is attached to the product tag has been described. However, a predetermined product code may be attached to an area other than the product code, for example, the rear panel 100 of the display shelf. An example of the display shelf in this case is shown in FIG. A product code in which predetermined information is encoded is affixed to the rear panel 100 of the display shelf. This product code may be a product code indicating “out of stock”, or a product code in which information indicating a stockout of a product to be displayed at that position is encoded (displayed as information on a missing product). Product ID of the product).

  When the display shelf is photographed in S200, if there is a shortage, the product code on the back panel 100 is also reflected. At that time, the product code conversion processing unit 222 reads and recognizes the “shortage” or the product ID of the product to be displayed at the position. In this case, the product code conversion processing unit 222 reads the product code not only in the area other than the shelf designated by the shelf position designation processing unit 221 but also in the entire display shelf image information and corrected image information. Process.

  By performing such processing, it is possible to recognize the shortage of the displayed product without performing image matching processing using sample image information on the image information as described in the second to fourth embodiments. Can be done.

  In each of the first to fifth embodiments, the processing order can be changed as necessary.

  By using the product information acquisition system 1 of the present invention, the products displayed on the display shelf can be identified with higher accuracy than before. Therefore, the display status of each product in the store can be acquired with high accuracy.

1: Product information acquisition system 2: Management server 3: Photographer terminal 4: System user terminal 20: Display shelf image information reception processing unit 21: Display shelf image information storage unit 22: Display shelf image information analysis processing unit 23: Product Information storage unit 24: Display status information storage unit 25: Output processing unit 26: Specimen image information storage unit 70: Computing device 71: Storage device 72: Display device 73: Input device 74: Communication device 100: Rear panel 220: Display shelf Image information correction processing unit 221: shelf position designation processing unit 222: product code conversion processing unit 223: display status information processing unit 224: product specification processing unit 225: product tag recognition processing unit

Claims (13)

A product information acquisition system for acquiring information on displayed products,
The product information acquisition system includes:
A display shelf image information analysis processing unit that executes image analysis processing on display shelf image information in which the display state of the product in the store is photographed;
A product information storage unit for storing information about the product;
With
The display shelf image information analysis processing unit
The product code in the display shelf image information is read, converted into a product ID corresponding to the product code,
A product corresponding to the converted product ID is identified from the product information storage unit,
A product information acquisition system characterized by this. The product information acquisition system further includes:
An output processing unit for outputting the display status information of the product specified by the display shelf image information analysis processing unit,
The display shelf image information analysis processing unit further includes:
Specifying display status information indicating the display status of the specified product,
The product information acquisition system according to claim 1. The product information acquisition system further includes:
A sample image information storage unit for storing sample image information of products displayed on the display shelf,
The display shelf image information analysis processing unit
The product code in the display shelf image information is read, converted into a product ID corresponding to the product code,
A product corresponding to the converted product ID is identified from the product information storage unit,
The display shelf image information is displayed on the display shelf by executing an image matching process between the display shelf image information or the corrected image information obtained by correcting the display shelf image information and the sample image information stored in the sample image information storage unit. Identify potential product candidates,
Determining whether the candidate product identified by the image matching process matches the product with the product ID converted from the product code, and identifying the matched product as a product displayed on the display shelf;
The product information acquisition system according to claim 1 or claim 2, wherein In the display shelf image information analysis processing unit,
If there is no matching product as a result of determining whether the candidate product specified by the image matching process matches the product with the product ID converted from the product code, the product with the product ID is missing. It is determined that
The product information acquisition system according to claim 3. The product information acquisition system further includes:
A sample image information storage unit for storing sample image information of products displayed on the display shelf,
The display shelf image information analysis processing unit
The product code in the display shelf image information is read, converted into a product ID corresponding to the product code,
A product corresponding to the converted product ID is identified from the product information storage unit,
Extracting the sample image information of the identified product from the sample image information storage unit,
The product displayed on the display shelf is identified by executing an image matching process between the extracted sample image information and the display shelf image information or the corrected image information obtained by correcting the display shelf image information. Do,
The product information acquisition system according to claim 1 or claim 2, wherein In the display shelf image information analysis processing unit,
When image matching processing between the extracted sample image information and the display shelf image information or the corrected image information is executed and a predetermined condition is not satisfied, it is determined that the product with the product ID is missing Do,
The product information acquisition system according to claim 5. A product code indicating a missing item is attached to the rear panel of the display shelf,
In the display shelf image information analysis processing unit,
When the product code attached to the back panel in the display shelf image information is read, the product with the product ID corresponding to the product code at the position corresponding to the product code attached to the back panel is missing. Judge that there is,
The product information acquisition system according to claim 1, wherein the product information acquisition system is a product information acquisition system. The display shelf image information analysis processing unit
The display status information indicating the display status of the specified product includes information on the specified product, information on the shelf position and the number of faces,
The output processing unit
Displaying the number of product faces according to the shelf position as the display status information;
The product information acquisition system according to any one of claims 2 to 7, wherein The display shelf image information analysis processing unit
The display status information indicating the display status of the specified product includes information on the specified product, information on the shelf position and the number of faces,
The output processing unit
As the display status information of the product, it is output together with information on the shelf of the golden line of a predetermined display shelf.
The product information acquisition system according to any one of claims 2 to 7, wherein A product information acquisition system for acquiring information on displayed products,
The product information acquisition system includes:
A display shelf image information correction processing unit for performing correction processing on display shelf image information in which the display state of the product in the store is photographed;
A shelf position designation processing unit that accepts a shelf position designation process for the display shelf image information or the corrected image information after the correction process;
A product information storage unit for storing a product ID and product information;
A product code conversion processing unit for converting a product code in the display shelf image information or the corrected image information into a product ID, and specifying product information corresponding to the product ID based on the product information storage unit;
A display status information processing unit configured to specify display status information of the specified product by specifying a position of the specified product on a display shelf;
An output processing unit for outputting the display status information;
A product information acquisition system comprising: The product code is attached to a product tag of a product displayed on a display shelf.
The product information acquisition system according to any one of claims 1 to 10, wherein: The product code is a color code,
12. The product information acquisition system according to claim 1, wherein the product information acquisition system is a product information acquisition system. Computer
A display shelf image information analysis processing unit for performing image analysis processing on display shelf image information in which the display state of products in the store is photographed;
A product information acquisition program that functions as
The display shelf image information analysis processing unit
The product code in the display shelf image information is read, converted into a product ID corresponding to the product code,
A product corresponding to the converted product ID is identified from a storage area storing information about the product;
A product information acquisition program characterized by that.

Images