EP4202765B1

EP4202765B1 - Method and system for counting bakeable food products

Info

Publication number: EP4202765B1
Application number: EP22215325.6A
Authority: EP
Inventors: Rafael JUAN FERNÁNDEZ
Original assignee: Dulcesa SL
Current assignee: Dulcesa SL
Priority date: 2021-12-23
Filing date: 2022-12-21
Publication date: 2025-02-12
Anticipated expiration: 2042-12-21
Also published as: EP4202765A1; ES3013589T3; ES2944786A1; EP4202765C0

Description

OBJECT OF THE INVENTION

The present invention relates to the technical field of quality control on food product production lines and, more specifically, to computer vision systems for food products to be baked for the automatic counting and production control thereof.

BACKGROUND OF THE INVENTION

Currently, the productive resources in the food industry seek to achieve the greatest possible efficiency, but without having to renounce high product customisation, flexibility and quick adaptation to different scenarios in a large production plant, which are provided by the infrastructure.
Mass production poses a challenge in terms of process supervision, control and management, among other problems. Although many stages of the production chain are automated, other tasks that are among the simplest are usually still carried out manually, which, in addition to the natural errors introduced by human intervention, entails an interruption in the information chain that feeds the control systems that are increasingly widespread in the industry, based on Big Data technologies and artificial intelligence.
Specifically, mass-produced bakeable food products, under very specific manufacturing parameters such as baking time, are continuously introduced by means of conveyor belts into a baking module. This process usually requires control personnel to confirm correct operation, which is done visually (with the accuracy that the operator can guarantee) or, sometimes, it is directly omitted and quality control of the already baked product is postponed. In both cases, the efficiency and ability to respond to errors is low.
Faced with this problem, it would be desirable to carry out an immediate and automatic count of the products in order to supervise the process, detect their presence and observe, for example, that the products move at the expected speed, that they are properly distributed along the conveyor belt, that they receive the expected cooking and, in general, that errors in the cooking process avoided by detecting and counting all the products.
It is known from prior art the document US 6233966 B1 (DELPUECH BERNARD ), disclosing a plant for the treatment of food products which includes a freezing tunnel having a detector for determining the quality and the quantity of products treated by the system. It is also known the document "Design of fruit Segregation and Packaging Machine" (KHEKARE GANESH ET AL) related to the sorting of goods through a process of training and testing images which leads the machine to identify the goods based on its color, texture, shape, and defects.
Therefore, a solution is needed in the state of the art to give continuity to smart production systems and offer a computer vision system that allows bakeable food products to be controlled and supervised in a simple, efficient and integrated manner on a production line.

DESCRIPTION OF THE INVENTION

To achieve the objectives of implementing smart production systems and avoiding the aforementioned drawbacks, the present invention describes, in a first aspect, a method for counting bakeable food products that comprises the following steps:

arranging an optical sensor in an inspection area through which food products conveyed by a conveyor belt pass;
capturing, by the optical sensor, an image of the bakeable food products as they pass through the inspection area;
identifying, by a processor module, a number of pixels in the captured image that correspond to a food product, based on a colour contrast threshold;
determining, by the processor module, the presence of a product in the inspection area if the number of identified pixels is greater than a pre-established minimum threshold; and
updating a product counter, adding one unit each time the presence of a product is determined;

In one of the embodiments of the invention, applying, by the processor module, an HSV conversion to the captured image is envisaged. More specifically, one embodiment envisages applying, by the processor module, a conversion to black and white of the captured image. More specifically, one embodiment comprises applying, by the processor module, a colour mask on the captured image which eliminates any intermediate value between black and white. Thus, advantageously, the colour transformations allow the background to be removed, which results in a simpler and better-quality image processing.
According to a specific embodiment of the present invention, the processing of the images further comprises applying, by the processor module, a blur filter on the captured image. More specifically, one of the embodiments envisages applying, by the processor module, an erosion filter on the captured image. More specifically, one of the embodiments envisages applying, by the processor module, a dilation filter on the captured image. Thus, advantageously, the filters applied to the captured images make it possible to eliminate small defects in the conversion or enhance the shape of the product for better definition.
According to one of the particular embodiments of the invention, the pre-established minimum threshold of identified pixels to determine the presence of the product is set to at least one third of the width of the captured image. Thus, advantageously, it is guaranteed that the product sufficiently occupies the inspection area so as to avoid failures due to deformation or poor positioning of the product.
Optionally, in one of the embodiments of the invention, it is envisaged that in order to determine the presence of a product in the inspection area if the number of identified pixels is greater than a pre-established minimum threshold, it must also be true that the identified pixels are adjacent pixels. Thus, advantageously, erroneous readings caused by two products positioned too close together are avoided.
In a second aspect, the present invention relates to a system for counting bakeable food products, which comprises the following elements:

an optical sensor for capturing images in an inspection area through which food products conveyed by a conveyor belt pass;
a power module to supply electrical power; and
a processor module configured to: receive the images captured by the sensor; determine the presence, in the inspection area, of a product conveyed by the belt, based on identifying a number of contrast pixels greater than a predetermined minimum threshold and further verifying that the number of identified pixels is greater than said threshold in a certain number of successive image capturing; and update a counter when the presence of a product is determined.

Additionally, in one of the embodiments of the invention, an LED lighting module arranged to illuminate the inspection area is envisaged. Thus, advantageously, the system is independent of external lighting conditions and can work under more homogeneous conditions.
One specific embodiment of the invention further comprises a conveyor belt for conveying bakeable food products to an oven, wherein the optical sensor is arranged on the conveyor belt along an axis perpendicular to the conveying direction. More specifically, it is envisaged that the optical sensor comprises hinged fastening means that can be coupled to a support structure, to adjust the inspection area along a line perpendicular to the conveying direction.
In a specific embodiment of the invention, the processor module is a Raspberry Pi computer board and the optical sensor is a Raspberry Pi camera module.

BRIEF DESCRIPTION OF THE FIGURES

To complete the description of the invention, and for the purpose of helping to make the features thereof more readily understandable, according to a preferred exemplary embodiment thereof, a set of drawings is included where, by way of illustration and not limitation, the following figures have been represented:

Figure 1 represents a perspective view of one of the preferred embodiments of the system of the present invention for recognising and counting bakeable food products arranged on a conveyor belt.
Figure 2 represents a perspective view of one of the preferred embodiments of the system of the present invention, showing hinged fastening means in detail.
Figure 3 represents a perspective view of one of the preferred embodiments of the system of the present invention, showing an LED lighting device in detail.

Figure 4 represents a flow chart of the general process of the invention.
Figure 5 represents a detailed flow chart of the steps of the processing stage, according to one of the embodiments of the invention.
Figure 6 represents a detailed flow chart of the steps of the search stage, according to one of the embodiments of the invention.
Figure 7 represents, by way of example, one of the images processed in one of the embodiments of the present invention to detect the presence and subsequently count bakeable food products.
Figure 8 schematically represents the hardware and software structure of a smart factory in which the present invention is integrated.

DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

Preferred embodiments of the present invention are described in detail below with the help of the accompanying figures.
Figure 1 shows a perspective view of one of the preferred embodiments of the system of the invention for recognising and counting bakeable food products (1) arranged on a conveyor belt (2). The outer casing (3) that houses the power module, the processor module and the optical sensor (4), the latter two from the Raspberry Pi series, is arranged on the food products in a plane parallel to the conveyor belt, so that the optical sensor has a clear view of the products.
Figure 2 shows a perspective view of one of the preferred embodiments of the system of the invention for recognising and counting bakeable food products (5) arranged on a conveyor belt (6). In this embodiment, one of the envisaged fastening means is shown in detail, in which a hinged part (7) fastens the casing (3) to a support structure (8), so that the inclination of the casing can be oriented and thereby adjust the image captured by the optical sensor and, therefore, the inspection area.
Figure 3 shows a perspective view of one of the preferred embodiments of the system of the invention for recognising and counting bakeable food products arranged on a conveyor belt. In this embodiment, the casing also has an LED lighting device (9).
The support structures for fastening the computer vision and counting device of the present invention can vary greatly. As in the embodiment of Figure 3, a guide system (10) can be used to allow longitudinal displacements of the casing along the guide (10) to vary the inspection area on different points of the conveyor belt. Alternatively, the casing can be fastened by means of screws to a fixed element of the structure. Alternatively, the casing can be coupled to a rod or bar, with a square or round section, as long as they allow the casing to be displaced transversely to the conveying direction of the conveyor belt.
Once the camera is connected to the Raspberry computer board via the CSI bus, the LED lighting module is powered by the GPIO bus at 3.3V or 5V, depending on the needs, or even by an external power supply. With the connections made, the casing is placed in the inspection area for its operation.
The operation of the present invention is generally based on the fact that, after performing an HSV conversion of the image by applying colour filters, blurring, eroding, dilation of the pixels and their conversion to black and white, the pixels in a certain area are counted and, after a minimum margin of pixels has been exceeded, the determination is made that there is a product in said area and a product counter is updated.
Figure 4 schematically shows the main functional blocks of the invention, according to the main process. Following the flow chart represented, this process begins with the capture stage (11), in which the optical sensor acquires the images of the products; this is followed by a processing stage (12) for processing the captured images; after processing, a search stage (13) for searching for the product in the images occurs; and finally, a last stage of information transmission (14) is carried out.
Figure 5 outlines the steps of the processing stage (12) according to one of the embodiments wherein, once there is an image capture (15), an inspection area (16) is defined in which the product is to be searched. In general, the inspection area should be wide enough to accommodate a product, or a large part of it, preferably defined perpendicular to the product conveying line. Next, a first transformation of the image from RGB to HSV is performed (17), a colour mask is applied (18) so that any colour that does not fall within the given spectrum tends to 0 (black) and the rest tends to 1 (white), which will give a better contrast between product and background, and a second transformation is applied to the resulting greyscale image (19). These colour space transformations will allow the background of the image to be removed.
Afterwards, various filters are applied to eliminate small defects or enhance the shape of the product. In this case, the following filters are specifically described: a first blur or distortion filter (20), which blurs the image and eliminates small gaps in the product, allowing the product's transformation to black and white (21) to be more homogeneous; a second erosion filter (22), which eliminates loose pixels that do not correspond to the product image, making it possible to eliminate colour impurities in the background; and a third dilation filter (23), which increases the size of the white areas, contributing to a better marking of the product and a greater number of white pixels. After this process, only white pixels remain in the processed image, which correspond to the food product. An inspection area that is more adjusted than the initial area is defined (24) and finally, the white pixels in the image in this adjusted inspection area are counted (25) to determine whether or not they are greater than a pre-established minimum threshold, which will define in the next stage whether or not the presence of the product in the image is confirmed.
For cases in which the food products are uniform, or due to their dimensions it is guaranteed that they easily pass through the inspection area, a minimum threshold of white pixels can be established to determine the presence of the product close to the width of the image. In practice, a value of at least a third or half of the width of the image is typically used to ensure that there will always be a number of pixels that ensures the presence of the product, avoiding failures due to deformation or poor positioning of the product prior to the image capture.
Figure 6 outlines the search stage (13) according to one of the embodiments. Once the total number of white pixels (25) has been obtained from the processing stage (12), the search for the product basically consists of evaluating whether certain requirements are met to determine that there is a product (29), which in this case are: exceeding a pre-established minimum number of pixels (26), exceeding a pre-established minimum number of readings (27) and exceeding a minimum time between product rows (28). The minimum number of readings is the number of successive times that the white pixel count must be greater than the threshold. It is established to avoid erroneous product detections caused by impurities or loose paste stains on the conveyor belt. Furthermore, the requirement of establishing a minimum time between rows seeks to avoid gaps within one same product and to avoid counting the same product twice. Additionally, one more criterion can be added to verify that, prior to detecting the presence of the current product, a gap has been detected (30) and, if so, the product counter is updated by one unit (31) and the search is concluded (32). To detect gaps, the flow chart takes a path that diverges from product detection. In this case, if the result of comparing the total white pixels is not greater than the pre-established minimum threshold, the presence of a product is directly ruled out; however, to determine that there is a gap (35), an evaluation is still performed to determine that the number of readings is greater than a pre-established minimum (33) and that a minimum time between product rows is exceeded (34). If these requirements are not met, the search is terminated (32) without determining the presence of a gap.
Other possible requirements that may be imposed in different embodiments of the invention include requiring that the minimum pixel threshold for determining the presence of a product relates to pixels that are together, or also providing a tolerance for small gaps.
Finally, continuing the flow chart of Figure 4, the last stage of information transmission relates to configuring the entire loop of the diagram to determine whether data is to be sent, when it is going to be sent and how it is going to be sent. Among the options thereof, the area where the camera is located can be defined and it can be coded according to factory parameters. For example, selecting when to send the information can be configured, as long as there is a minimum time gap between products, designed, for example, for discontinuous product cans or if the number of counted units is greater than the minimum entered. It can also be configured to send the histogram (total number of white pixels during the entire product selection process in all readings) for an analysis of results and subsequent adjustment of values and the sending of results by UDP protocol or for execution by a web page.
Figure 7 represents, by way of example, one of the images processed in one of the embodiments of the present invention to detect the presence and subsequently count bakeable food products. The bottom (36) is represented completely in black (although in the figure it has not been coloured to optimise ink in a possible reproduction on paper), offering the maximum possible contrast with the white pixels, which represent the areas of the image occupied by the products (37).
The computer vision system for detecting the presence and subsequently counting bakeable food products on a production line of the present invention is included more generally in a smart factory ecosystem, wherein one of the main objectives is the connectivity of all control devices in the factory with other systems and people having different profiles, to achieve faster, more efficient and more flexible processes. In that sense, Figure 8 schematically represents the hardware and software structure of said installation, wherein all the control devices (80) distributed throughout the factory are located at the base and they communicate through their corresponding communication controllers (81) with a central server (82). The central server, in communication with the cloud (83), allows access from different platforms (84, 85, 86) to use the information collected at the source by the control devices in tasks of supervising, display, data logging, etc.
The data collected by the control devices are connected to PLC controllers to store the information they provide and to be able to subsequently process it from any of the platforms (84, 85, 86) to which access to said information has been provided. A PLC (programmable logic controller), also known as programmable automaton, is an industrial computer that processes all the data of a machine, such as sensors, buttons, timers and any input signal, to subsequently control the actuators (pistons, motors, valves, etc.) and thus be able to control any industrial process automatically. These devices have great advantages, such as for example offering a simple way to automate processes, which allows a company to improve its competitiveness by increasing productivity at reasonable costs and without sacrificing quality, reducing the cost of labour or, thanks to process supervising, saving on operations, maintenance and even energy. Some of the devices that can be used in a smart factory, such as the one in which the present invention is encompassed, are the Omron company PLC, the Siemens PLC or the Allen-Bradley PLC.
In more detail, the control devices (80) deployed by a smart factory with a computer vision system such as the one described by the present invention can integrate scales for different types of weighing during the process; metal detectors to alert the user of any metal objects intercepted by the detector's electromagnetic field, which automatically ejects the product from the production line; X-ray systems for detecting dense contaminants such as glass, metal, mineral stone, calcified bone and high-density plastic, regardless of their size, shape or location inside the product, which is automatically ejected in the case of detection following parallel operation to the metal detector; high-precision dynamic weighers to verify the weight of the product on the conveyor belt, unaffected by the vibrations of said belt and reliable so as to properly discard incomplete products at high speed; expiry date printers for continuously printing an expiry date on products; or Raspberry Pi boards with various functions adapted to each situation, such as temperature probes, computer vision or integration of new devices.
The communication controllers (81) are necessary to establish communication between the different control devices and the central server (83), for example of the OPC Unified Architecture type. Specific communication drivers are used for each piece of equipment to be able to recognise which devices are connected and thus make optimal use of them. To use the data collected by the control devices in the form of variables or registers on different data logging or display and supervising platforms, all the devices send the data via these drivers to the OPC UA server, wherein each driver is given a different configuration to achieve successful communication on one same platform with all the connected devices.
The platforms from which a user can access and process the information initially collected by the control devices can comprise, for example, a SCADA system (84), which is an industrial automation and control tool used in production processes that can control, supervise, collect data, analyse data and generate reports remotely through a computer application. Its main function is to evaluate the data in order to correct possible errors. Nowadays, it is a fundamental element in modern industrial plants, since it helps to maintain efficiency, processes data to make smarter decisions, communicates system problems to help reduce downtime or inactivity, and allows for a historical study to be performed to anticipate future failures.
Moreover, among the platforms with access to system information, it is useful to have an alarm system (85) in which certain variables to be controlled are received, whether they come from a PLC, databases or other integrated systems, where scripts can be created with conditions that, when met, activate an alarm. Selected users can be notified of the alarms via e-mail or telephone, although a sound or light system that can more immediately capture the attention of operators can also be implemented.
A smart factory like the one described herein also needs a database (86) between the platforms. These factories often have an enterprise resource planning (ERP) system to address the business needs from a process standpoint and to integrate company-wide information systems. Thus, the control devices (80) directly communicate with the ERP, through TCP, sending data that will be stored in a table of said database (86). For example, a scale would send the line on which it is working, the ordinal of said scale, the moment of sending, the value of the weight, the established maximum and minimum, and all the information that the user needs to save. Communication with this database (86) is two-way, such that variables can be created from a query to the database and these variables can be processed, for example, in a SCADA system as described above. Therefore, a log of all the necessary information is obtained to subsequently process it according to the needs.
The present invention should not be limited by the embodiment herein described. Other arrangements may be carried out by those skilled in the art based on the present description. Accordingly, the scope of the invention is defined by the following claims.

Claims

A method for counting bakeable food products, 2. the method comprises the following steps:
- arranging an optical sensor (4) in an inspection area through which bakeable food products (1, 5) conveyed by a conveyor belt (2, 6) pass;

- capturing, by the optical sensor, an image of the bakeable food products as they pass through the inspection area (15);

- identifying, by a processor module, a number of pixels (25) in the captured image that correspond to one of the food products, based on a colour contrast threshold;

- determining, by the processor module, the presence of a product (31) in the inspection area if the number of identified pixels is greater than a pre-established minimum threshold (26); and

- updating a product counter (31), adding one unit each time the presence of a product (29) is determined;
wherein determining the presence of a product in the inspection area if the number of identified pixels is greater than a pre-established minimum threshold further comprises verifying that the number of identified pixels is greater than said threshold in a certain number of successive image capturing (27).
The method according to claim 1, which further comprises applying, by the processor module, an HSV conversion (17) to the captured image.
The method according to any of the preceding claims, which further comprises applying, by the processor module, a conversion to black and white of the captured image (21).
The method according to any of the preceding claims, which further comprises applying, by the processor module, a colour mask (18) on the captured image which eliminates any intermediate value between black and white.
The method according to any of the preceding claims, which further comprises applying, by the processor module, a blur filter (20) on the captured image.
The method according to any of the preceding claims, which further comprises applying, by the processor module, an erosion filter on the captured image.
The method according to any of the preceding claims, which further comprises applying, by the processor module, a dilation filter (23) on the captured image.
The method according to any of the preceding claims, wherein the pre-established minimum threshold of identified pixels to determine the presence of the product is set to at least one third of the width of the captured image.
The method according to any of the preceding claims, wherein determining the presence of a product in the inspection area if the number of identified pixels is greater than a pre-established minimum threshold further comprises verifying that the identified pixels are adjacent pixels.
A system for counting bakeable food products comprising the following elements:
- an optical sensor (4) for capturing images in an inspection area (15) through which food products (1, 5) conveyed by a conveyor belt (2, 6) pass;

- a power module to supply electrical power; and

- a processor module,
the system is characterised in that the processor module is configured to: receive the images captured by the sensor; determine the presence (29), in the inspection area, of a product conveyed by the belt, based on identifying a number of contrast pixels (25) greater than a predetermined minimum threshold and further verifying that the number of identified pixels is greater than said threshold in a certain number of successive image capturing (27); and update a counter (31) when the presence of a product is determined.
The system according to claim 10, which further comprises an LED lighting module (9) arranged to illuminate the inspection area.
The system according to any of the claims 10-11, which further comprises a conveyor belt (2, 6) for conveying bakeable food products (1, 5) to an oven, wherein the optical sensor is arranged on the conveyor belt along an axis perpendicular to the conveying direction.
The system according to claim 12, wherein the optical sensor further comprises hinged fastening means (7) that can be coupled to a support structure (8), to adjust the inspection area along a line perpendicular to the conveying direction.
The system according to any of claims 10-13, wherein the processor module is a Raspberry Pi computer board and the optical sensor is a Raspberry Pi camera module.