WO2022059117A1 - Video processing device, image capturing device, and video processing method - Google Patents

Abstract

The present invention provides a video processing device capable of efficiently carrying out an operation of applying labels to captured images.　Thus, a video processing device 100 comprises: a moving image processing unit 19 which processes a video signal output by an image capturing element 2 into moving image files of prescribed periods and outputs said moving image files; a storage unit 17 which stores operator action data 170 that includes action content of the operator and processing content corresponding thereto; and a recognition processing unit 18 which determines the action content of the operator on the basis of sensor signals output by sensor elements 3, generates action recognition results information on the basis of the operator action data 170 in the storage unit 17, and outputs the result to the moving image processing unit 19. When the recognition processing unit 18 has output information relating to label application as the action recognition results information, the moving image processing unit 19 applies a label relating to said label application to the moving image file of the prescribed period which includes the point in time when the action recognition results information was output.

Description

Video processing device, image pickup device, and video processing method

The present invention relates to a video processing device, an imaging device, and a video processing method.

In recent years, in the field of image recognition, machine learning using Deep Neural Network (DNN) called deep learning has been actively studied, and a highly accurate machine learning model is generated using a huge amount of computational resources and learning data. It is possible.

Among machine learning, in supervised machine learning, when generating a machine learning model, a large amount of learning data that combines image data and annotations (correct answer labels) for the image data is required. However, it takes a lot of time and effort to collect a large amount of image data and add annotations (correct labels) to each of these image data. Therefore, it is a big obstacle in applying the image recognition technology using machine learning to the actual system.

As a method of annotating image data, Patent Document 1 describes a method of performing character recognition on a captured image captured by an imaging unit and assigning a character string acquired by the character recognition to the captured image as an annotation. Is disclosed.

Japanese Unexamined Patent Publication No. 2010-28486

As an example of an actual system that applies image recognition technology using a machine learning model, in a factory production line, an object passing over the production line is photographed with a camera, image recognition is performed on the captured image, and the object is subjected to image recognition. An inspection system that detects abnormalities and defects in objects can be mentioned.

When collecting the training data required for the machine learning model used in the inspection system as described above, it is conceivable to use the method disclosed in Patent Document 1, but even if an abnormality occurs in the object, it may occur. The character string "abnormal" does not appear in the captured image of the object. Therefore, even if the method disclosed in Patent Document 1 is used, the character string included in the photographed image is only added to the photographed image as an annotation, and the photographed image in which the object in which the abnormality has occurred is, for example, , "Abnormal" cannot be labeled.

Also, in the production line, the rate of abnormal objects being generated is generally extremely low compared to normal objects. Therefore, it is not efficient to randomly photograph all objects and later label them as "abnormal".

Therefore, an object of the present invention is to provide a video processing device, an imaging device, and a video processing method capable of efficiently carrying out the work of assigning a label to a photographed image.

In order to solve the above problems, one of the representative video processing devices of the present invention is
A video processing unit that processes the video signal output from the image sensor into a video file for a predetermined period and outputs it.
A storage unit that stores worker behavior data including worker behavior content and corresponding processing content,
A recognition processing unit that determines the behavior content of a worker based on a sensor signal output from a sensor element, generates behavior recognition result information based on the worker behavior data in the storage unit, and outputs it to the moving image processing unit. And with
The moving image processing unit
When the recognition processing unit outputs information related to label assignment as the action recognition result information, the label related to the label assignment is added to the moving image file for the predetermined period including the output time of the action recognition result information. ..

According to the present invention, since the moving image processing unit assigns label information to the moving image file according to the labeling action of the worker, the work of assigning the label to the moving image file can be efficiently performed.

Issues, configurations and effects other than the above will be clarified by the following description in the embodiment for carrying out the invention.

It is an overall block diagram which shows the image processing system 1 which concerns on Example 1. FIG. It is a schematic block diagram which shows the image processing apparatus 100 which concerns on Example 1. FIG. It is a data composition diagram which shows the worker behavior data 170 stored in the storage part 17 of the image processing apparatus 100 which concerns on Example 1. FIG. It is a flowchart which shows the operation of the image processing apparatus 100 which concerns on Example 1. FIG. It is a figure explaining the operation when the action content of the worker U is determined to be the start point setting action 172 in the image processing apparatus 100 which concerns on Example 1. FIG. In the video processing apparatus 100 according to the first embodiment, it is a figure explaining the operation when the action content of a worker U is determined to be a label-imparting action 171. It is a figure explaining the operation when the action content of the worker U is determined to be the end point setting action 173 in the image processing apparatus 100 which concerns on Example 1. FIG. It is a schematic block diagram which shows the image processing apparatus 100 which concerns on Example 2. FIG. It is a schematic block diagram which shows the image pickup apparatus 200 which concerns on Example 3. FIG. It is a schematic block diagram which shows the image pickup apparatus 200 which concerns on Example 4. FIG. It is a block diagram which shows the configuration example of the computer system 300.

Hereinafter, the image processing device, the image pickup device, and the image processing method according to the embodiment of the present invention will be described with reference to the drawings.

(Example 1)
FIG. 1 is an overall configuration diagram showing a video processing system 1 according to the first embodiment.

The video processing system 1 is a system used in the production line 102 of the factory that manufactures the object 101. The production line 102 is composed of, for example, a belt conveyor that conveys the object 101 in the conveying direction indicated by the arrow A in FIG. The worker U is an inspector for visually inspecting whether each object 101 flowing on the production line 102 one after another is normal (good product) or abnormal (defective product).

The video processing system 1 includes a video processing device 100, an image pickup element 2 for an object, a sensor element 3 for a worker, a terminal device 4 for a worker, and a moving image file storage device 5 as its configuration.

The video processing device 100 is connected to an image pickup element 2, a sensor element 3, a worker terminal device 4, and a moving image file storage device 5, respectively, and transmits and receives various signals and data by wire or wirelessly. The video processing device 100 processes the video signal output from the image pickup device 2 into a video file for a predetermined period and outputs the video signal to the video file storage device 5. Details of the video processing apparatus 100 will be described later.

The image sensor 2 for an object is composed of, for example, a CMOS sensor, a CCD sensor, or the like. The image sensor 2 is installed above the production line 102, for example, images the object 101 flowing on the production line 102 and the production line 102, and outputs the imaged image signal to the image processing apparatus 100.

The working sensor element 3 includes at least one of an image pickup element 30, a microphone element 31, and a contact element 32. The sensor element 3 is installed around the image pickup element 2 in order to monitor the behavior of the worker U working near the installation location of the image pickup element 2. In this embodiment, the sensor element 3 includes all of the image pickup element 30, the microphone element 31, and the contact element 32.
The image sensor 30 for an operator is composed of, for example, a CMOS sensor, a CCD sensor, or the like, like the image sensor 2 for an object. The image pickup device 30 takes an image of the whole or a part of the body of the worker U, and outputs the imaged image signal to the image processing device 100.

The microphone element 31 converts the voice uttered by the worker U into an audio signal as an electric signal, and outputs the converted audio signal to the image processing device 100.

The contact element 32 is composed of, for example, a contact type or non-contact type switch, button, lever or the like. The contact element 32 converts the operation performed by the worker U by hand or foot into a contact signal as an electric signal, and outputs the converted contact signal to the image processing device 100.

The worker terminal device 4 is composed of, for example, a desktop computer, a notebook computer, a tablet computer, a smartphone, or the like. The worker terminal device 4 is installed at a position where the worker U can see, and displays a display screen based on the display data output from the video processing device 100. The worker terminal device 4 may be configured as a display device such as a liquid crystal display or an organic EL display, for example.

The moving image file storage device 5 is composed of, for example, a server device such as a file server or a storage device. The moving image file storage device 5 stores the moving image file output from the video processing device 100 and operates as a database of the moving image file. The moving image file storage device 5 may be composed of, for example, a desktop computer, a notebook computer, a tablet computer, a smartphone, or the like, similarly to the worker terminal device 4.

In FIG. 1, the number of the image pickup element 2, the image pickup element 30, the microphone element 31, and the contact element 32 is one each, but the number is not limited to this, and may be a plurality. Further, the installation positions of the image pickup element 2, the image pickup element 30, the microphone element 31, and the contact element 32 are not limited to the example of FIG. 1, and may be appropriately changed.

(Configuration of video processing device 100)
FIG. 2 is a schematic configuration diagram showing the video processing apparatus 100 according to the first embodiment. The video processing device 100 includes a video signal input unit 11, a video signal processing unit 12, an audio signal input unit 13, an audio signal processing unit 14, a contact signal input unit 15, a contact signal processing unit 16, a storage unit 17, and a recognition processing unit 18. And a moving image processing unit 19.

The video signal input unit 11 is composed of, for example, a connector such as a BNC connector and a receiver IC. An image pickup element 2 for an object and an image pickup element 30 for an operator are connected to the video signal input unit 11, and a video signal is input to each. The video signal processing unit 12 performs processing such as color correction and gamma correction on the video signal input via the video signal input unit 11.

The audio signal input unit 13 is composed of, for example, an XLR connector and a receiver IC. A microphone element 31 is connected to the audio signal input unit 13, and an audio signal is input. The audio signal processing unit 14 performs processing such as spectrum analysis and noise removal on the audio signal input via the audio signal input unit 13.

The contact signal input unit 15 is composed of, for example, a connector of any type and a receiver IC. A contact element 32 is connected to the contact signal input unit 15, and a contact signal is input. The contact signal processing unit 16 performs processing such as on / off state detection for the contact signal input via the contact signal input unit 15.

The storage unit 17 stores the worker action data 170 in which the processing content executed by the video processing device 100 is associated with the action content of the worker U when the worker U takes a predetermined action. ..

FIG. 3 is a data configuration diagram showing worker behavior data 170 stored in the storage unit 17 of the video processing device 100 according to the first embodiment.

In the storage unit 17, the "behavior content" of the worker U obtained by analyzing the sensor signal as the worker behavior data 170, and the "sensor type", "processing type", and "processing" corresponding to each behavior content are stored. "Content" is remembered. In the worker behavior data 170, the sensor type and the processing type are additional information, and one or both of them may be omitted.

The processing contents include "labeling process" for assigning an electronic label to a video file, "start point setting process" for setting a start point of a video file, and "end point setting process" for setting an end point of a video file. It is remembered. In the label assignment process, label information 171a (“abnormal” in the example of FIG. 3) indicating a label to be assigned to the moving image file is also stored.

In the processing type, for example, "labeling" for assigning a label to a video file and "period setting" for the start point and end point of the video file are stored as information for classifying the processing contents executed by the video processing apparatus 100. ing.

In the action content, the specific action of the worker U is stored in correspondence with each processing content. In the example of FIG. 3, two action contents of "raising both hands" and "operating the foot switch" are stored as action contents (labeling action 171) corresponding to the label giving process. As the action contents corresponding to the start point setting process (start point setting action 172), two action contents of "raising the left hand" and "starting and speaking" are stored. As the action content corresponding to the end point setting process (end point setting action 173), two action contents of "raising the right hand" and "speaking with the end" are stored.

The sensor type stores information that identifies the sensor element 3 to be analyzed when the sensor signal output from the sensor element 3 is analyzed to determine the action content of the worker U. In the example of FIG. 3, an "image sensor" is stored as a sensor type corresponding to the action contents of "raising both hands", "raising the left hand", and "raising the right hand". A "microphone element" is stored as a sensor type corresponding to the action contents of "start and speak" and "end and speak". A "contact element" is stored as a sensor type corresponding to the action content of "operating the foot switch".

The action content, sensor type, processing type, and processing content type stored in the worker behavior data 170 are not limited to the above, and the sensor type, processing type, and processing content are set according to various behavior content. can do. For example, it is possible to classify two types of continuous actions as one action content and set the processing content accordingly. Further, it is possible to set a plurality of sensor types for one action content.

Further, in the example of FIG. 3, the label related to label assignment is one type of "abnormal", but it is also possible to correspond to a plurality of types of labels. For example, when the content of the abnormality is subdivided into three, the worker behavior data 170 is given the first label "there is a first abnormality". "Raise the left hand with one finger of the left hand extended" and the second action content "Finger of the left hand" corresponding to the second label assignment process for assigning the second label "there is a second abnormality" "Raise your left hand with two extended hands" and the third action content corresponding to the third label assignment process that gives the third label "Third abnormality" "The state where three fingers of the left hand are extended" It suffices to remember "raise your left hand with."

The recognition processing unit 18 analyzes the sensor signals (video signal, audio signal, and contact signal) output from the sensor element 3 and determines the action content of the worker U working in the vicinity of the installation location of the image pickup element 2. At that time, the recognition processing unit 18 takes any of the action contents stored in the storage unit 17 as the worker action data 170 by the worker U, that is, the labeling action 171 and the start point setting action 172 and the end point setting action 173. If it is determined that the behavior has been determined, the behavior recognition result information for specifying the determined behavior content is output to the moving image processing unit 19. Further, the recognition processing unit 18 analyzes the video signal output from the image sensor 2 and acquires the characters included in the video.

The recognition processing unit 18 includes a video recognition unit 180 and a character recognition unit 183 connected to the video signal processing unit 12, a voice recognition unit 181 connected to the voice signal processing unit 14, and a contact signal processing unit 16. It is provided with a contact recognition unit 182 connected to the above.

The video recognition unit 180 applies a known video analysis method to the video signal of the image pickup device 30 for the worker processed by the video signal processing unit 12, and determines the action content of the worker U. The voice recognition unit 181 applies a known voice analysis method to the voice signal of the microphone element 31 processed by the voice signal processing unit 14, and determines the action content of the worker U. The contact recognition unit 182 applies a known contact analysis method to the contact signal of the contact element 32 processed by the contact signal processing unit 16, and determines the action content of the worker U. Then, the video recognition unit 180, the voice recognition unit 181 and the contact recognition unit 182 determine the action content of the worker U, generate the action recognition result information based on the worker action data 170, and output it to the moving image processing unit 19. do.

It should be noted that, on the premise that the worker U is wearing, for example, an emblem or the like as a mark on his left hand, the image recognition unit 180 recognizes the worker U in response to the detection of the mark, and further, the said. By tracking the position of the mark, for example, the position or movement of the left hand wearing the mark may be recognized.

The character recognition unit 183 applies a known character recognition method to the video signal of the image sensor 2 for the object processed by the video signal processing unit 12, and describes the object 101 itself or the package of the object 101. Get the character that was done. Then, the character recognition unit 183 generates character recognition result information based on the result of acquiring the character, and outputs the character recognition result information to the moving image processing unit 19.

Based on the action recognition result information and the character recognition result information output by the recognition processing unit 18, the moving image processing unit 19 displays the video signal of the image pickup element 2 for the object processed by the video signal processing unit 12 as a moving image for a predetermined period. It is processed into a file and output to the moving image file storage device 5.

Specifically, when the recognition processing unit 18 outputs information related to a start point instruction as action recognition result information, the moving image processing unit 19 sets the output time point of the action recognition result information as the starting point of the moving image file. Further, when the recognition processing unit 18 outputs the information related to the end point instruction as the action recognition result information, the moving image processing unit 19 sets the output time point of the action recognition result information to the end point of the moving image file. Then, the moving image processing unit 19 outputs a moving image file for a predetermined period separated by the set start point and ending point.

It should be noted that any method may be adopted as the method in which the moving image processing unit 19 processes the video signal into a moving image file for a predetermined period and outputs it. For example, the moving image processing unit 19 starts recording the moving image file based on the video signal according to the setting of the start point, and ends the recording of the moving image file based on the video signal according to the setting of the ending point, thereby moving the moving image for a predetermined period. You may output the file. Further, the moving image processing unit 19 temporarily stores the video signals for a certain time in the past in the storage unit 17, sets the start point and the end point, and then stores the set start point and end point in the storage unit 17. By cutting out the stored video signal, a moving image file for a predetermined period may be output. At that time, any file format and resolution of the video file may be adopted.

Further, when the recognition processing unit 18 outputs information related to label assignment as action recognition result information, the moving image processing unit 19 displays the label related to the label assignment (“abnormal” in this embodiment) as the action recognition result information. It is given to the video file for a predetermined period including the output time of. At that time, the moving image processing unit 19 also adds the character recognition result information to the moving image file.

When the moving image processing unit 19 imparts label information and character recognition result information to the moving image file, the label information and character recognition result information may be recorded as attribute information of the moving image file, or may be recorded separately from the moving image file. It may be recorded in a file and the file may be associated with the moving image file.

The moving image processing unit 19 outputs display data for displaying various information to the worker terminal device 4. The display data may be, for example, behavior recognition result information, character recognition result information, label information given to a moving image file, real-time video based on the video signal of the image pickup element 2 for an object, or a video signal of the image pickup element 30 for a worker. Includes real-time video based on the video, and time information indicating the start and end points of the video file.

(Operation of video processing device 100)
FIG. 4 is a flowchart showing the operation of the video processing apparatus 100 according to the first embodiment. The video processing apparatus 100 repeatedly executes a series of processes (video processing method) shown in FIG. 4, for example, at predetermined control cycles.

First, in step S10, the recognition processing unit 18 monitors the action content of the worker U by analyzing the sensor signal output from the sensor element 3. Specifically, the video recognition unit 180, the voice recognition unit 181 and the contact recognition unit 182 analyze the video signal, the voice signal and the contact signal, respectively, so that the worker U can store the worker action data 170 as the storage unit 17. It is monitored whether or not any of the labeling action 171, the start point setting action 172, and the end point setting action 173 stored in the above is taken.

In step S20, as a result of the recognition processing unit 18 analyzing the sensor signal in step S10, whether the action content of the worker U corresponds to any of the label assigning action 171 and the start point setting action 172 and the end point setting action 173. Judge whether or not. If "Yes" in step S20, the process proceeds to step S21. On the other hand, if "No" in step S20, the process returns to step S10 and the process of monitoring the behavior of the worker U is continued.

In step S21, the recognition processing unit 18 generates behavior recognition result information that specifies the behavior content of the worker U based on the worker behavior data 170, outputs the information to the video processing unit 19, and proceeds to step S30.

In step S30, the moving image processing unit 19 corresponds to the action content by referring to the worker action data 170 of the storage unit 17 based on the action content of the worker U output by the recognition processing unit 18 as the action recognition result information. Determine the processing content. That is, when the action recognition result information is the start point setting action 172 (information regarding the start point setting), the process proceeds to the start point setting process (step S40) corresponding to the start point setting action 172, and the label assignment action 171 (information regarding the label assignment) is performed. In the case of, the process proceeds to the start point setting process (step S51) corresponding to the label assigning action 171 through the start point setting confirmation process (step S50), and in the case of the end point setting action 173 (information about the end point setting), the start point setting is performed. After the confirmation process (step S60), the process proceeds to the end point setting process (step S61) corresponding to the end point setting action 173.

In this embodiment, the behavior recognition result information has been described as information for specifying the behavior content of the worker behavior data 170, but the behavior recognition result information may be information for specifying the processing content of the labeling action 171. In that case, the recognition processing unit 18 refers to the worker behavior data 170 of the storage unit 17 based on the behavior content of the worker U obtained by analyzing the sensor signal, and the processing content corresponding to the behavior content. By specifying the above, the action recognition result information may be generated and output to the moving image processing unit 19. Then, when the recognition processing unit 18 outputs the start point setting process (information regarding the start point setting) as the action recognition result information, the moving image processing unit 19 proceeds to step S40 and performs the label assignment process (information regarding the label assignment). If it is output, the process proceeds to step S50, and if the end point setting process (information about the end point setting) is output, the process may proceed to step S60.

In step S40, the moving image processing unit 19 sets the output time of the action recognition result information by the recognition processing unit 18, that is, the time when the action content of the worker U is determined to be the start point setting action 172 as the start point of the moving image file. Start recording video files. In step S41, the moving image processing unit 19 outputs display data indicating that the recording of the moving image file has started to the worker terminal device 4.

FIG. 5 is a diagram illustrating an operation when the action content of the worker U is determined to be a start point setting action 172 in the video processing device 100 according to the first embodiment. In addition, in FIG. 5 and FIGS. 6 and 7 described later, the description of the microphone element 31 and the contact element 32 is omitted.

When the worker U standing by near the installation location of the image sensor 2 visually recognizes that the object 101 flowing on the production line 102 has reached in front of him, he raises his left hand. This action of "raising the left hand" is determined by the recognition processing unit 18 as the start point setting action 172, so that the moving image processing unit 19 starts recording the moving image file. Further, on the display screen 40 of the worker terminal device 4, it is displayed that the start point setting action 172 is recognized and the recording of the moving image file is started based on the display data output from the moving image processing unit 19. Further, on the display screen 40, a real-time image 41 based on the image signal of the image pickup element 2 and a real-time image 42 based on the image signal of the image pickup element 30 for the operator are displayed.

In step S50, it is confirmed whether or not the start point has been set before the label assigning action 171. If the start point has been set, the process proceeds to step S51. If the start point has not been set, step S51 has not been set. , The process is terminated without proceeding to S52. As a result, if the recording of the moving image file has not started, the labeling process is not performed, so that malfunction can be prevented.

In step S51, the moving image processing unit 19 labels the moving image file being recorded at the time when the action recognition result information by the recognition processing unit 18 is output, that is, when the action content of the worker U is determined to be the labeling action 171. Give. In step S52, the moving image processing unit 19 outputs display data indicating that the moving image file is labeled to the worker terminal device 4.

FIG. 6 is a diagram illustrating an operation when the action content of the worker U is determined to be the labeling action 171 in the video processing device 100 according to the first embodiment.

Worker U visually confirms the object 101 that has reached in front of him, and when he finds an abnormality, raises both hands. This action of "raising both hands" is determined by the recognition processing unit 18 as the labeling action 171 so that the moving image processing unit 19 assigns the "abnormal" label to the moving image file currently being recorded. Further, on the display screen 40 of the terminal device 4 for workers, the label giving action 171 is recognized based on the display data output from the moving image processing unit 19, and the moving image file is given the label "abnormal". Is displayed.

In addition, when a plurality of label giving actions 171 are stored in the storage unit 17, a plurality of labels are given during recording of the moving image file, for example, the first label giving action and the second labeling action. When the action 171 is recognized, both the first label information and the second label information may be given to the moving image file, for example, a predetermined priority (action taken later by the worker U). Any label information may be given according to (priority is given to).

In step S60, it is confirmed whether or not the start point has been set before the end point setting action 173, and if the start point has been set, the process proceeds to step S61, and if the start point has not been set, step S61 has not been set. The process ends without proceeding to S63. As a result, if the recording of the moving image file has not started, the end point setting process is not performed, so that a malfunction can be prevented.

In step S61, the moving image processing unit 19 sets the output time point of the action recognition result information by the recognition processing unit 18, that is, the time point when the action content of the worker U is determined to be the end point setting action 173, as the end point of the moving image file. End recording of the video file. Then, in step S62, the moving image processing unit 19 outputs display data indicating that the recording of the moving image file is completed to the worker terminal device 4.

In step S63, the moving image processing unit 19 outputs the moving image file to the moving image file storage device 5. At that time, if the label assignment action 171 of the worker U is recognized during the recording of the moving image file, the moving image file is output in a state where the label information of "abnormality" is given in step S51. The label. On the other hand, when the label assignment action 171 of the worker U is not recognized, the moving image file may be output with the label information of "normal" attached as the default label, or the label information. May be output in a state where is not given.

FIG. 7 is a diagram illustrating an operation when the action content of the worker U is determined to be the end point setting action 173 in the video processing apparatus 100 according to the first embodiment.

When the worker U visually recognizes that the object 101 has passed in front of him, he raises his right hand. The action of "raising the right hand" is determined by the recognition processing unit 18 as the end point setting action 173, so that the moving image processing unit 19 ends the recording of the moving image file. Further, on the display screen 40 of the worker terminal device 4, it is displayed that the end point setting action 173 is recognized based on the display data output from the moving image processing unit 19, and the recording of the moving image file is completed.

In each step shown in FIG. 4, steps S10 to S21 correspond to recognition processing steps, and steps S30 to S62 correspond to moving image processing steps. As a modification of each step shown in FIG. 4, the timing of adding the label information to the moving image file may be the timing of outputting the moving image file of step S63 instead of the timing of step S51. Further, instead of outputting the moving image file to the moving image file storage device 5 in step S63, for example, the moving image file may be stored in the storage unit 17. Further, the timing of outputting the moving image file in step S63 is not limited to immediately after step S62, and may be executed at any timing. Further, the execution order of step S62 and step S63 may be interchanged.

As described above, the series of processing (video processing method) shown in FIG. 4 is completed. By repeatedly executing a series of processes shown in FIG. 4, the image processing apparatus 100 outputs a plurality of moving image files in which each object 101 flowing one after another on the production line 102 is imaged by the image sensor 2. At that time, the label information of "abnormality" was given to the moving image file in which the object 101 in which the abnormality was found by the visual inspection of the worker U was imaged according to the labeling action 171 of the worker U. It is output in the state.

Therefore, according to the present embodiment, the moving image processing unit 19 assigns the label information to the moving image file according to the label giving action 171 of the worker U, so that the work of assigning the label to the moving image file is efficient. Can be carried out.

Further, since the moving image processing unit 19 sets the starting point and the ending point of the moving image file according to the start point setting action 172 and the end point setting action 173 of the worker U, the period for recording as a moving image file can be easily instructed. Further, the capacity of the moving image file is reduced as compared with the case where the moving image file is constantly recorded, and the moving image file is limited to the period in which the object 101 is imaged, so that the usability as learning data is improved. Can be done.

(Example 2)
FIG. 8 is a schematic configuration diagram showing the video processing apparatus 100 according to the second embodiment.

The video processing system 1 according to the first embodiment includes an image pickup element 2 for imaging an object 101 and an image pickup element 30 (an example of a sensor element 3) for imaging a worker U as separate devices. On the other hand, the image processing system 1 according to the second embodiment includes an image pickup element 2 that images an object 101 and an operator U within the same angle of view, so that the image pickup element 2 takes an image as a sensor element 3. It also serves as an element 30. The image processing system 1 shown in FIG. 8 includes a microphone element 31 and a contact element 32 as the sensor element 3, but the microphone element 31 and the contact element 32 may be omitted.

In the video processing device 100 according to the present embodiment, the video signal of the image pickup element 2, the audio signal of the microphone element 31, and the contact signal of the contact element 32 are input. Since the basic configuration and operation of the video processing apparatus 100 according to the present embodiment are the same as those of the video processing apparatus 100 according to the first embodiment, the differences between the two will be mainly described here.

An image pickup element 2 is connected to the video signal input unit 11, and the video signal output from the image pickup element 2 is a recognition processing unit 18 and a moving image processing unit via the video signal input unit 11 and the video signal processing unit 12. It is output to 19.

The video recognition unit 180, the voice recognition unit 181 and the contact signal processing unit 16 of the recognition processing unit 18 include a video signal output from the image pickup element 2 and a sensor signal (voice signal and contact signal) output from the sensor element 3. Is analyzed, and the action content of the worker U working in the vicinity of the installation location of the image pickup element 2 is determined. Further, the character recognition unit 183 of the recognition processing unit 18 analyzes the video signal output from the image sensor 2 and acquires the characters included in the video.

The moving image processing unit 19 processes the video signal of the image pickup element 2 processed by the video signal processing unit 12 into a moving image file for a predetermined period and outputs it based on the action recognition result information and the character recognition result information by the recognition processing unit 18. do.

According to this embodiment, even when the image sensor 2 is installed so as to image the object 101 and the worker U, the moving image processing unit 19 labels the moving image file according to the labeling action 171 of the worker U. Give information. Therefore, it is possible to efficiently carry out the work of assigning a label to a moving image file while simplifying the system configuration.

(Example 3)
FIG. 9 is a schematic configuration diagram showing the image pickup apparatus 200 according to the third embodiment.

The video processing system 1 according to the first embodiment includes a video processing device 100 and an image pickup device 2 as separate devices. On the other hand, the image processing system 1 according to the third embodiment includes an image pickup device 200 in which the image processing device 100 and the image pickup element 2 are configured as one device.

According to this embodiment, even when the image pickup device 200 is configured as a device in which the image processing device 100 and the image pickup element 2 are integrated, the moving image processing unit 19 responds to the labeling action 171 of the worker U. Add label information to the video file. Therefore, it is possible to efficiently carry out the work of assigning a label to a moving image file while simplifying the system configuration.

(Example 4)
FIG. 10 is a schematic configuration diagram showing the image pickup apparatus 200 according to the fourth embodiment.

As described above, the image processing system 1 according to the third embodiment includes an image pickup device 200 (imaging element 2) that images an object 101 and an image pickup element 30 (an example of a sensor element 3) that images an operator U. Provided as a separate device. On the other hand, in the image processing system 1 according to the fourth embodiment, the image pickup device 200 includes the image pickup element 2 that images the object 101 and the worker U within the same angle of view, so that the image pickup element 2 is a sensor. It also serves as an image pickup device 30 as the element 3. The video processing system 1 shown in FIG. 10 includes a microphone element 31 and a contact element 32 as the sensor element 3, but the microphone element 31 and the contact element 32 may be omitted.

According to this embodiment, even when the image sensor 2 is installed so as to image the object 101 and the worker U, the moving image processing unit 19 labels the moving image file according to the labeling action 171 of the worker U. Give information. Therefore, it is possible to efficiently carry out the work of assigning a label to a moving image file while simplifying the system configuration.

(Configuration example of computer system 300)
FIG. 11 is a block diagram showing a configuration example of the computer system 300. The computer system 300 described below may be applied to each device of each of the above embodiments. At that time, the configuration of the computer system 300 may be appropriately modified and applied according to the function of each device.

The computer system 300 includes a processor 302, a memory 304, a terminal interface 312, a storage interface 314, an I / O (input / output) device interface 316, and a network interface 318 as its main components. These main components are connected to each other via a bus structure including a memory bus 306, an I / O bus 308, a bus interface unit 309, and an I / O bus interface unit 310.

The processor 302 is composed of one or a plurality of arithmetic processing units (CPU, MPU, GPU, DSP, etc.). In the example of FIG. 11, the processor 302 is composed of two CPUs 302A and 302B. Processor 302 may execute instructions stored in memory 304 and include an onboard cache.

The memory 304 is composed of a volatile memory (DRAM, SRAM, etc.) or a non-volatile memory (ROM, flash memory, etc.) for storing various data and the program 350. The memory 304 represents the virtual memory of the computer system 300 and may include the virtual memory of another computer system connected to the computer system 300 via the network 330.

The program 350 may include an instruction or description executed on the processor 302, or may include an instruction or description interpreted by another instruction or description. Program 350 also hardware in place of or in addition to processor-based systems via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices. It may be carried out by hardware.

The program 350 may be stored in the storage device 322 instead of the memory 304, or may be stored in a computer-readable recording medium such as a CD, DVD, or USB memory in an installable file format or an executable file format. It may be stored and provided. The program 350 may also be stored in an external computer system and provided by downloading via network 330.

The display system 324 is composed of a display controller, a display memory, or both. The display controller outputs video, audio, or both data to the display device 326. The display memory is a dedicated memory for buffering video data. The function provided by the display system 324 may be realized by an integrated circuit including a processor 302.

The display device 326 is composed of, for example, a liquid crystal display, an organic EL display, electronic paper, a projector, or the like. The display device 326 is connected to another device having a display screen (for example, a portable device such as a notebook computer, a tablet computer, a smartphone, a television, etc.) to cause the other device to function as the display device 326. You may.

The terminal interface 312 is configured to be connectable to a user I / O device 320 such as a user input device or a user output device, and is a communication for the processor 302 and the user I / O device 320 to communicate with each other by wire or wirelessly. Provide a route. The user I / O device 320 is, for example, a keyboard, mouse, keypad, touchpad, trackball, button, light pen, or other pointing device.

The storage interface 314 is configured to be connectable to the storage device 322, and provides a communication path for the processor 302 and the storage device 322 to communicate with each other. The storage device 322 has one or more disk drives, and is composed of, for example, a magnetic disk drive, a semiconductor disk drive, or the like.

The I / O device interface 316 is configured to be connectable to an external I / O device 340 and provides a communication path for the processor 302 and the external I / O device 340 to communicate with each other by wire or wirelessly. The external I / O device 340 is, for example, any device such as a camera, scanner, microphone, speaker, printer, fax machine, or any sensor such as a biometric sensor, environment sensor, motion sensor or the like. The computer system 300 may include a part or all of the above-mentioned external I / O device 340.

The network interface 318 is configured to be connectable to a network 330 such as the Internet or an intranet, and provides a communication path for the processor 302 and the network 330 to communicate with each other by wire or wirelessly. The external I / O device 340 may be connected to the network interface 318 via the network 330.

The bus structure in the computer system 300 provides a communication path that directly connects between the processor 302, the memory 304, the bus interface unit 309, the display system 324, and the I / O bus interface unit 310. The bus structure may include, for example, a hierarchical structure, a star structure or a web structure point-to-point link, a plurality of hierarchical buses, and parallel or redundant communication paths. Further, the function provided by the bus interface unit 309 may be realized by an integrated circuit including a processor 302.

The computer system 300 is, for example, a desktop computer, a laptop computer, a tablet computer, a smartphone, a mobile phone or any other suitable electronic device. The computer system 300 may be a client-type computer, a server-type computer, or a cloud-type computer.

When the video processing device 100 and the image pickup device 200 are realized in the computer system 300 having the above configuration, the video signal input unit 11, the audio signal input unit 13, and the contact signal input unit 15 are one or a plurality of I / O device interfaces. It is composed of 316. The video signal processing unit 12, the audio signal processing unit 14, the contact signal processing unit 16, the recognition processing unit 18, and the moving image processing unit 19 are composed of one or a plurality of processors 302. The storage unit 17 is composed of one or a plurality of memories 304 and at least one of the one or a plurality of storage devices 322.

When the video signal processing unit 12, the audio signal processing unit 14, the contact signal processing unit 16, the recognition processing unit 18, and the moving image processing unit 19 are composed of a plurality of processors 302, each unit is composed of a separate processor 302. Alternatively, an arbitrary combination of parts may be configured by separate processors 302. For example, the video signal processing unit 12, the audio signal processing unit 14, and the contact signal processing unit 16 may be configured by the first processor, and the recognition processing unit 18 and the moving image processing unit 19 may be configured by the second processor. Further, when the processor 302 executes the program 350, the processor has at least a part of the functions of the video signal processing unit 12, the audio signal processing unit 14, the contact signal processing unit 16, the recognition processing unit 18, and the moving image processing unit 19. For example, the functions of the recognition processing unit 18 and the moving image processing unit 19 may be realized.

The image pickup element 2 and the sensor element 3 in the image processing system 1 correspond to the external I / O device 340, and the worker terminal device 4 and the moving image file storage device 5 correspond to other computer systems connected to the network 330. do. The worker terminal device 4 may correspond to the display device 326.

(Supplementary information)
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention. For example, the present invention is not limited to the one including all the configurations described in the above-described embodiment, and includes the one in which a part of the configurations is deleted. Further, it is possible to add or replace a part of the configuration according to a specific embodiment with the configuration according to another embodiment.

The worker behavior data 170 may be appropriately changed by the worker U operating, for example, the worker terminal device 4. Further, when the worker U operates, for example, the worker terminal device 4, the recognition processing unit 18 analyzes the sensor signal, and the analysis parameters when determining the action content of the worker U are appropriately adjusted. May be good.

Further, in each of the above embodiments, the recognition processing unit 18 has been described as recognizing both the start point setting action 172 and the end point setting action 173, but the recognition processing unit 18 has described the start point setting action 172 and the end point setting action 173. The process of recognizing at least one of them may be omitted. In that case, the start point and the end point of the moving image file are set by other means. For example, the recognition processing unit 18 may analyze the video signal output from the image sensor 2 and set the frame-in and frame-out time points of the object 101 at the start point and the end point of the moving image file, respectively. Further, for example, a trigger generator such as a transmission sensor or a reflection sensor is installed on the upstream side and the downstream side of the installation location of the image pickup element 2, and the start point and the end point of the moving image file are set based on the trigger signal of the trigger generator. Each may be set.

Further, in each of the above embodiments, the video processing device 100 and the image pickup device 200 have been described as including the voice signal input unit 13, the voice signal processing unit 14, the contact signal input unit 15, and the contact signal processing unit 16. May be omitted depending on the presence or absence of the sensor element 3 connected to the image processing device 100 and the image pickup device 200. For example, if the microphone element 31 is not connected, the audio signal input unit 13 and the audio signal processing unit 14 may be omitted, and if the contact element 32 is not connected, the contact signal input unit 15 and the contact signal processing unit 15 may be omitted. Part 16 may be omitted.

Further, in each of the above embodiments, the recognition processing unit 18 has been described as including the character recognition unit 183, but the character recognition unit 183 may be omitted. In that case, the moving image processing unit 19 may output the moving image file assuming that the character recognition result by the character recognition unit 183 has not been acquired.

1 ... video processing system, 2 ... image pickup element, 3 ... sensor element, 4 ... worker terminal device, 5 ... video file storage device, 11 ... video signal input unit, 12 ... video signal processing unit, 13 ... audio signal input Unit, 14 ... Audio signal processing unit, 15 ... Contact signal input unit, 16 ... Contact signal processing unit, 17 ... Storage unit, 18 ... Recognition processing unit, 19 ... Movie processing unit, 30 ... Imaging element, 31 ... Microphone element, 32 ... Contact element, 100 ... Video processing device, 170 ... Worker action data, 171 ... Labeling action, 172 ... Start point setting action, 173 ... End point setting action, 180 ... Video recognition unit, 181 ... Voice recognition unit, 182 ... Contact recognition unit, 183 ... Character recognition unit, 200 ... Imaging device, 300 ... Computer system, 302 ... Processor, 302A, 302B ... CPU, 304 ... Memory, 306 ... Memory bus, 308 ... I / O bus, 309 ... Bus interface Unit, 310 ... I / O bus interface unit, 312 ... Terminal interface, 314 ... Storage interface, 316 ... I / O device interface, 318 ... Network interface, 320 ... User I / O device, 322 ... Storage device, 324 ... Display System, 326 ... Display device, 330 ... Network, 340 ... External I / O device, 350 ... Program

Claims (5)

A video processing unit that processes the video signal output from the image sensor into a video file for a predetermined period and outputs it.
A storage unit that stores worker behavior data including worker behavior content and corresponding processing content,
A recognition processing unit that determines the behavior content of a worker based on a sensor signal output from a sensor element, generates behavior recognition result information based on the worker behavior data in the storage unit, and outputs it to the moving image processing unit. And with
The moving image processing unit
When the recognition processing unit outputs information related to label assignment as the action recognition result information, the label related to the label assignment is added to the moving image file for the predetermined period including the output time of the action recognition result information.
Video processing equipment. The moving image processing unit
When the recognition processing unit outputs information related to a start point instruction as the action recognition result information, the output time point of the action recognition result information is set as the start point of the predetermined period.
When the recognition processing unit outputs information related to the end point instruction as the action recognition result information, the output time point of the action recognition result information is set to the end point of the predetermined period.
The video processing apparatus according to claim 1. Image sensor and
A video processing unit that processes the video signal output from the image sensor into a video file for a predetermined period and outputs it.
A storage unit that stores worker behavior data including worker behavior content and corresponding processing content,
A recognition processing unit that determines the behavior content of a worker based on a sensor signal output from a sensor element, generates behavior recognition result information based on the worker behavior data in the storage unit, and outputs it to the moving image processing unit. And with
The moving image processing unit
When the recognition processing unit outputs information related to label assignment as the action recognition result information, the label related to the label assignment is added to the moving image file for the predetermined period including the output time of the action recognition result information.
Imaging device. The moving image processing unit
When the recognition processing unit outputs information related to a start point instruction as the action recognition result information, the output time point of the action recognition result information is set as the start point of the predetermined period.
When the recognition processing unit outputs information related to the end point instruction as the action recognition result information, the output time point of the action recognition result information is set to the end point of the predetermined period.
The imaging device according to claim 3. It is a video processing method that processes the video signal output from the image sensor into a video file for a predetermined period and outputs it.
The action content of the worker is determined based on the sensor signal output from the sensor element, and the data described in the storage unit is based on the action content of the worker and the worker action data including the corresponding processing content. A recognition processing step that generates and outputs action recognition result information,
When the recognition processing step outputs information related to label assignment as the action recognition result information, the label related to the label assignment is added to the moving image file for the predetermined period including the output time point of the action recognition result information.
When the recognition processing step outputs information related to a start point instruction as the action recognition result information, the output time point of the action recognition result information is set as the start point of the predetermined period.
When the recognition processing step outputs information related to an end point instruction as the action recognition result information, the image processing step includes a moving image processing step in which the output time point of the action recognition result information is set to the end point of the predetermined period.
Video processing method.

Images