WO2024140973A1 - Action counting method and related device - Google Patents

Abstract

The present application discloses an action counting method and a related device, capable of combining a plurality of types of information to complete a target action technique, and beneficial to improving the final obtained accuracy of a target action counting result. The method of the present application comprises: obtaining a target video, then first performing first processing on the target video, to obtain position information of a target object in the target video. Next, adding to the target video the position information of the target object in the target video, to obtain a processed target video. Then, performing second processing on the processed target video, to obtain an action sequence of the target object, the action sequence being able to comprise a plurality of actions, the plurality of actions generally comprising a target action and remaining actions, and the plurality of actions being sorted according to the time of appearance thereof in the processed target video. Finally, processing the plurality of actions, to obtain the number of target actions of the target object.

Description

A motion counting method and related device

This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 29, 2022, with application number 202211716276.0 and invention name “A motion counting method and related equipment”, the entire contents of which are incorporated by reference in this application.

Technical Field

The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to an action counting method and related equipment.

Background technique

In mining, cargo handling, mechanical construction and other operational scenarios, it is often necessary to count the number of certain actions of people and/or machines in the operational process to determine whether people and/or machines have successfully completed the operation. Based on this, a solution that uses AI technology to automatically complete action counting has emerged.

For ease of explanation, the following is a schematic introduction to the mining drilling scenario. In a tunnel in a mine, an operator is required to operate a drilling machine to drill the side wall of the tunnel, and the drilling depth is pre-set. In order to count the number of drillings by the operators (that is, the number of drill rods driven into the side wall), the equipment can collect real-time video of the scene and use a neural network model to perform target detection on the collected video to determine the position information of the drill rod in the video. Then, based on the position information of the drill rod in the video, the number of drillings by the operators can be counted.

However, in the above-mentioned action counting process, only the influence of the position information of the object is considered, and the factors considered are relatively single, resulting in low accuracy of the final action counting result.

Summary of the invention

The embodiments of the present application provide an action counting method and related equipment, which can integrate multiple information to complete the target action technology, which is beneficial to improving the accuracy of the final counting result of the target action.

A first aspect of an embodiment of the present application provides an action counting method, the method comprising:

After acquiring the target video of the operation area, the target video may be input into the first model, so as to perform a first processing on the target video through the first model, thereby obtaining the position information of the target object in the target video.

After obtaining the position information of the target object in the target video, the position information of the target object in the target video may be added to the target video, thereby obtaining a processed target video.

After obtaining the processed target video, the processed target video can be input into the second model to perform a second processing on the processed target video through the second model, so as to obtain an action sequence of the target object. The action sequence may include multiple actions, and these multiple actions usually include several target actions and several other actions. These multiple actions can be sorted according to the appearance time of these multiple actions in the processed target video, for example, target action, other actions, target action, other actions, target action, other actions, and so on.

After obtaining the action sequence of the target object, the action sequence can be processed to determine the number of target actions of the target object.

It can be seen from the above method that after obtaining the target video, the target video can be first processed to obtain the position information of the target object in the target video. Then, the position information of the target object in the target video can be added to the target video to obtain the processed target video. Then, the processed target video can be processed for the second time to obtain the action sequence of the target object. The action sequence can include multiple actions, which usually include the target action and the remaining actions, and the multiple actions are sorted according to the appearance time in the processed target video. Finally, the action sequence can be processed to obtain the number of target actions of the target object. In the above process, not only the influence of the position information of the target object is considered, but also the mutual influence between the multiple actions of the target object, that is, the influence of the time sorting of the target action and the remaining actions. The factors considered are relatively comprehensive, which is conducive to improving the accuracy of the final counting result for the target action.

In a possible implementation, the target video includes N video frames, N ≥ 2, and the position information is added to the target video to obtain the processed target video, including: extracting features from the i-th video frame to obtain a first feature of the i-th video frame; fusing the position information of the target object in the i-th video frame and the first feature to obtain a second feature of the i-th video frame; extracting features from the second feature to obtain a processed i-th video frame, i = 1, ..., N. In the above implementation, for the N video frames of the target video, For the ith video frame, the feature extraction can be performed on the ith video frame first, so as to obtain the first feature of the ith video frame. After obtaining the first feature of the ith video frame, the position information of the target object in the ith video frame and the first feature of the ith video frame can be fused to obtain the second feature of the ith video frame. After obtaining the second feature of the ith video frame, the feature extraction can be performed on the second feature to obtain the processed ith video frame. For the remaining video frames except the ith video frame among the N video frames, the same operation as that performed on the ith video frame can be performed on the remaining video frames, so that the processed N video frames, that is, the processed target video, can be finally obtained.

In a possible implementation, the method further includes: obtaining confidences of multiple actions, the confidences being obtained by performing a second processing on the processed target video; determining the number of target actions of the target object based on the action sequence includes: determining an action whose confidence is greater than or equal to a confidence threshold in the action sequence; determining the number of target actions of the target object based on an action whose confidence is greater than or equal to the confidence threshold. In the aforementioned implementation, after performing a second processing on the processed target video using the second model, not only the action sequence of the target object can be obtained, but also the confidences of multiple actions in the action sequence can be obtained. After obtaining the action sequence of the target object and the confidences of multiple actions, an action whose confidence is greater than or equal to the confidence threshold can be selected in the action sequence. Then, after obtaining an action whose confidence is greater than or equal to the confidence threshold, it is possible to determine how many target actions of the target object there are in total among the actions whose confidence is greater than or equal to the confidence threshold, which is equivalent to determining the number of target actions of the target object.

In one possible implementation, the confidence threshold is determined based on the position information of the target object in the target video, the statistical information of the confidence of multiple actions, and the pixel information of the target video, or the confidence threshold is a preset value. In the aforementioned implementation, the confidence threshold can be obtained in a variety of ways: (1) After obtaining the confidence of the multiple actions, the confidence of the multiple actions can be calculated first to obtain the statistical information of the confidence of the multiple actions. Then, the position information of the target object in the target video, the statistical information of the confidence of the multiple actions, and the pixel information of the target video can be calculated to obtain the confidence threshold. (2) The confidence threshold can be a preset value.

In one possible implementation, determining the number of target actions of the target object based on the action sequence also includes: determining the actions whose order conforms to a preset order among the actions whose confidence is greater than or equal to the confidence threshold; and determining the number of target actions of the target object based on the actions whose order conforms to the preset order. In the aforementioned implementation, after obtaining the actions whose confidence is greater than or equal to the confidence threshold, the actions whose order conforms to the preset order may be selected among the actions whose confidence is greater than or equal to the confidence threshold. Then, after obtaining the actions whose order conforms to the preset order, it may be determined how many target actions of the target object there are in total among the actions whose order conforms to the preset order, which is equivalent to determining the number of target actions of the target object.

In one possible implementation, the method further includes: segmenting the target video to obtain sub-videos presenting the work content; in the action sequence, determining the actions that appear in the sub-video; based on the action sequence, determining the number of target actions of the target object includes: determining the number of target actions of the target object based on the actions that appear in the sub-video. In the aforementioned implementation, after obtaining the action sequence, the target video can be segmented into two sub-videos, the content presented by the first sub-video is the work content, and the content presented by the second sub-video is the non-work content. Then, in this action sequence, the action that appears in the first sub-video can be selected. In this way, after obtaining the actions that appear in the sub-video, it is possible to determine the total number of target actions of the target object in the actions that appear in the sub-video, which is equivalent to determining the number of target actions of the target object.

In one possible implementation, the first processing is target detection, and the second processing is temporal action positioning.

A second aspect of an embodiment of the present application provides an action counting device, which includes: a first processing module, used to perform a first processing on a target video to obtain position information of a target object in the target video; an adding module, used to add the position information of the target object in the target video to the target video to obtain a processed target video; a second processing module, used to perform a second processing on the processed target video to obtain an action sequence of the target object, the action sequence including multiple actions sorted according to the appearance time in the processed target video, and the multiple actions include a target action; a first determination module, used to determine the number of target actions of the target object based on the action sequence.

It can be seen from the above device that after obtaining the target video, the target video can be first processed to obtain the position information of the target object in the target video. Then, the position information of the target object in the target video can be added to the target video to obtain the processed target video. Then, the processed target video can be processed for the second time to obtain the action sequence of the target object. The action sequence can include multiple actions, which usually include the target action and the remaining actions, and the multiple actions are sorted according to the appearance time in the processed target video. Finally, the action sequence can be processed to obtain the number of target actions of the target object. In the above process, not only the influence of the position information of the target object is considered, but also the mutual influence between the multiple actions of the target object, that is, the influence of the time sorting of the target action and the remaining actions. The factors considered are relatively comprehensive, which is conducive to improving the accuracy of the final counting result for the target action.

In one possible implementation, the target video includes N video frames, N≥2, and a module is added to: perform feature extraction on the i-th video frame to obtain a first feature of the i-th video frame; fuse the position information of the target object in the i-th video frame and the first feature to obtain a second feature of the i-th video frame; perform feature extraction on the second feature to obtain a processed i-th video frame, i=1,...,N.

In one possible implementation, the device also includes: an acquisition module, used to obtain confidences of multiple actions, where the confidences are obtained by performing a second processing on the processed target video; a first determination module, used to: determine actions whose confidences are greater than or equal to a confidence threshold in an action sequence; and determine the number of target actions of the target object based on actions whose confidences are greater than or equal to the confidence threshold.

In one possible implementation, the confidence threshold is determined based on the position information of the target object in the target video, statistical information of the confidences of multiple actions, and pixel information of the target video, or the confidence threshold is a preset value.

In one possible implementation, the first determination module is further used to: determine, among actions whose confidence is greater than or equal to a confidence threshold, actions whose order conforms to a preset order; and determine the number of target actions of the target object based on the actions whose order conforms to the preset order.

In one possible implementation, the device also includes: a segmentation module, used to segment the target video to obtain a sub-video presenting the work content; a second determination module, used to determine the action appearing in the sub-video in the action sequence; and a first determination module, used to determine the number of target actions of the target object based on the action appearing in the sub-video.

In one possible implementation, the first processing is target detection, and the second processing is temporal action positioning.

A third aspect of an embodiment of the present application provides an action counting device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the action counting device performs the method described in the first aspect or any possible implementation method of the first aspect.

A fourth aspect of an embodiment of the present application provides a model training device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device is used to train a neural network model, and the trained model can be applied to the method described in the first aspect or any possible implementation method of the first aspect.

A fifth aspect of an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect or any possible implementation method of the first aspect.

A sixth aspect of an embodiment of the present application provides a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes the method described in the first aspect or any possible implementation method of the first aspect.

In a possible implementation manner, the processor is coupled to the memory through an interface.

In a possible implementation, the chip system also includes a memory, in which a computer program or computer instructions are stored.

A seventh aspect of an embodiment of the present application provides a computer storage medium, which stores a computer program. When the program is executed by a computer, the computer implements the method described in the first aspect or any possible implementation method of the first aspect.

An eighth aspect of the embodiments of the present application provides a computer program product, which stores instructions, and when the instructions are executed by a computer, enables the computer to implement the method described in the first aspect or any possible implementation method of the first aspect.

In an embodiment of the present application, after acquiring the target video, the target video may be first processed for the first time to obtain the position information of the target object in the target video. Then, the position information of the target object in the target video may be added to the target video to obtain the processed target video. Then, the processed target video may be processed for the second time to obtain the action sequence of the target object, and the action sequence may include multiple actions, which usually include the target action and the remaining actions, and the multiple actions are sorted according to the appearance time in the processed target video. Finally, the action sequence may be processed to obtain the number of target actions of the target object. In the aforementioned process, not only the influence of the position information of the target object is considered, but also the mutual influence between the multiple actions of the target object, that is, the influence of the time sorting of the target action and the remaining actions, and the factors considered are relatively comprehensive, which is conducive to improving the accuracy of the final counting result for the target action.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a structure of an artificial intelligence main framework;

FIG2a is a schematic diagram of a structure of an action counting system provided in an embodiment of the present application;

FIG2b is another schematic diagram of the structure of the action counting system provided in an embodiment of the present application;

FIG2c is a schematic diagram of a device related to action counting provided in an embodiment of the present application;

FIG3 is a schematic diagram of the architecture of the system 100 provided in an embodiment of the present application;

FIG4 is a flow chart of an action counting method provided in an embodiment of the present application;

FIG5 is a schematic diagram of the fusion of video frames and position information provided by an embodiment of the present application;

FIG6 is a schematic diagram of video segmentation provided in an embodiment of the present application;

FIG7 is a schematic diagram of a comparison result provided in an embodiment of the present application;

FIG8 is a schematic diagram of a structure of an action counting device provided in an embodiment of the present application;

FIG9 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application;

FIG10 is a schematic diagram of a structure of a training device provided in an embodiment of the present application;

FIG. 11 is a schematic diagram of the structure of a chip provided in an embodiment of the present application.

Detailed ways

The embodiments of the present application provide an action counting method and related equipment, which can integrate multiple information to complete the target action technology, which is beneficial to improving the accuracy of the final counting result of the target action.

The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

In mining, cargo handling, mechanical construction and other operational scenarios, it is often necessary to count the number of certain actions of people and/or machines in the operational process to determine whether people and/or machines have successfully completed the operation. Based on this, a solution that uses AI technology to automatically complete action counting has emerged.

For ease of explanation, the following is a schematic introduction to the mining drilling scenario. In a tunnel in a mine, an operator is required to operate a drilling machine to drill the side wall of the tunnel, and the drilling depth is pre-set as the operating target. The drilling depth is usually proportional to the number of drillings. In order to count the number of drillings by the operators (that is, the number of drill rods driven into the side wall), the equipment can collect on-site videos in real time, and use a neural network model to perform target detection on the collected videos to determine the position information of the drill rod in the video. For example, the position information of the drill rod in the video can be the position of the center of the drill rod-video time change curve. Then, based on the position information of the drill rod in the video, the number of times the operator completes the action of taking the drill rod can be determined, which is equivalent to obtaining the number of times the operator completes the drilling action.

However, in the above-mentioned counting process of the target action (e.g., drilling), only the influence of the position information of the object (e.g., drill rod) is considered, and the factors considered are relatively single, resulting in the final counting result for the target action (i.e., the number of target actions) having low accuracy.

In order to solve the above problems, the embodiment of the present application provides an action counting method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by sensing the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for data processing is a common application of artificial intelligence.

First, the overall workflow of the artificial intelligence system is described. Please refer to Figure 1. Figure 1 is a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

(1) Infrastructure

The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnection networks For example, sensors communicate with the outside world to acquire data, which is then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

(2) Data

The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

(4) General capabilities

After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

(5) Smart products and industry applications

Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

Next, several application scenarios of this application are introduced.

FIG2a is a schematic diagram of a structure of an action counting system provided in an embodiment of the present application, wherein the action counting system includes a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, a personal computer or an information processing center. The user device is the initiator of the action counting, and as the initiator of the action counting request, the user usually initiates the request through the user device.

The above-mentioned data processing device can be a device or server with data processing function such as a cloud server, a network server, an application server and a management server. The data processing device receives the image processing request from the intelligent terminal through the interactive interface, and then performs image processing in the form of machine learning, deep learning, search, reasoning, decision-making, etc. through the memory for storing data and the processor link for data processing. The memory in the data processing device can be a general term, including local storage and databases for storing historical data. The database can be on the data processing device or on other network servers.

In the action counting system shown in FIG2a, the user device can receive the user's instruction, the user device can shoot the target video, and then initiate a request to the data processing device, so that the data processing device executes the video processing application for the target video obtained by the user device, thereby obtaining the processing result for the target video. Exemplarily, after receiving the user's instruction, the user device can shoot the operation area based on the instruction to obtain the target video. Then, the user device can initiate a processing request for the target video to the data processing device, so that the data processing device can perform a series of processing on the target video based on the request using a neural network model, thereby obtaining the processing result of the target video, that is, the number of target actions of the target object in the operation area, so as to determine whether the operation is completed based on the processing result of the target video.

In FIG. 2 a , the data processing device may execute the action counting method according to the embodiment of the present application.

Figure 2b is another structural diagram of the action counting system provided in an embodiment of the present application. In Figure 2b, the user device directly serves as a data processing device. The user device can directly obtain instructions from the user and directly process them by the hardware of the user device itself. The specific process is similar to that of Figure 2a. Please refer to the above description and will not be repeated here.

In the action counting system shown in FIG2b, the user device can receive the user's instruction, and the user device can shoot the operation area based on the instruction to obtain the target video. Then, the user device can use the neural network model to perform a series of processing on the target video to obtain the processing result of the target video, that is, the number of target actions of the target object in the operation area, so as to determine whether the operation is completed based on the processing result of the target video.

In FIG. 2b , the user equipment itself can execute the action counting method of the embodiment of the present application.

FIG. 2c is a schematic diagram of a device related to action counting provided in an embodiment of the present application.

The user device in the above Figures 2a and 2b can specifically be the local device 301 or the local device 302 in Figure 2c, and the data processing device in Figure 2a can specifically be the execution device 210 in Figure 2c, wherein the data storage system 250 can store the data to be processed of the execution device 210, and the data storage system 250 can be integrated on the execution device 210, and can also be set on the cloud or other network servers.

The processors in Figures 2a and 2b can perform data training/machine learning/deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to execute video processing applications on the video, thereby obtaining corresponding processing results.

Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of the present application. In Figure 3, the execution device 110 is configured with an input/output (I/O) interface 112 for data interaction with an external device. The user can input data to the I/O interface 112 through the client device 140. The input data may include: various tasks to be scheduled, callable resources and other parameters in the embodiment of the present application.

When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

Finally, the I/O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

It is worth noting that the training device 120 can generate corresponding target models/rules based on different training data for different goals or tasks, and the corresponding target models/rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from the training samples collected by the data collection device 160.

In the case shown in FIG. 3 , the user can manually give input data, and the manual giving can be operated through the interface provided by the I/O interface 112. In another case, the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I/O interface 112 and the output results of the output I/O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I/O interface 112 directly stores the input data of the input I/O interface 112 and the output results of the output I/O interface 112 as new sample data in the database 130.

It is worth noting that FIG3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG3, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110. As shown in FIG3, a neural network can be obtained by training according to the training device 120.

The embodiment of the present application also provides a chip, which includes a neural network processor NPU. The chip can be set in the execution device 110 as shown in Figure 3 to complete the calculation work of the calculation module 111. The chip can also be set in the training device 120 as shown in Figure 3 to complete the training work of the training device 120 and output the target model/rule.

Neural network processor NPU, NPU is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of NPU is the operation circuit, and the controller controls the operation circuit to extract data from the memory (weight memory or input memory) and perform operations.

In some implementations, the arithmetic circuit includes multiple processing units (process engines, PEs) internally. In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory and performs matrix operations with matrix B. The partial results or final results of the matrix are stored in the accumulator.

The vector calculation unit can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit can be used for network calculations of non-convolutional/non-FC layers in neural networks, such as pooling, batch normalization, local response normalization, etc.

In some implementations, the vector computation unit can store the processed output vector to a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate an activation value. The arithmetic unit generates normalized values, merged values, or both. In some implementations, the vector of processed outputs can be used as activation input to the arithmetic circuit, for example for use in subsequent layers in a neural network.

The unified memory is used to store input data and output data.

The weight data is directly transferred from the external memory to the input memory and/or the unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.

The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC and instruction fetch memory through the bus.

An instruction fetch buffer connected to the controller, used to store instructions used by the controller;

The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.

Generally, the unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memories.

Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

(1) Neural Network

A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

The work of each layer in the neural network can be described by the mathematical expression y=a(Wx+b): From a physical level, the work of each layer in the neural network can be understood as completing the transformation from input space to output space (i.e., the row space to column space of the matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimension increase/reduction; 2. Zoom in/out; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are completed by Wx, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. The vector W determines the spatial transformation from input space to output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network is to finally obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, or more specifically, learning the weight matrix.

Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the predicted value of the current network with The target value that is really desired is then updated according to the difference between the two layers of the neural network weight vector (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict lower, and it is adjusted continuously until the neural network can predict the target value that is really desired. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

(2) Back propagation algorithm

Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

The method provided in the present application is described below from the training side of the neural network and the application side of the neural network.

The model training method provided in the embodiment of the present application involves the processing of data sequences, and can be specifically applied to methods such as data training, machine learning, and deep learning, to perform symbolic and formalized intelligent information modeling, extraction, preprocessing, and training on the training data, and finally obtain a trained neural network (the first model, the second model, etc. in the embodiment of the present application); and the action counting method provided in the embodiment of the present application can use the above-mentioned trained neural network to input the input data (the target video in the embodiment of the present application, the processed target video, etc.) into the trained neural network to obtain output data (such as the position information of the target object in the target video in the present application, multiple actions of the target object, etc.). It should be noted that the model training method and summary generation method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

FIG4 is a flow chart of an action counting method provided in an embodiment of the present application. As shown in FIG4 , the method includes:

401. Perform a first process on a target video to obtain position information of a target object in the target video.

In this embodiment, after acquiring a target video of an operating area (for example, a tunnel in a mine, a warehouse in logistics, etc.), the target video can be input into a first model (a trained neural network model) to perform a first processing (for example, object detection) on the target video through the first model, thereby obtaining the position information of the target object in the target video.

It should be noted that the target video usually includes N consecutive video frames (N is a positive integer greater than or equal to 2), so the position information of the target object in the target video also includes the position information of the target object in the N video frames. For the i-th video frame among the N video frames (i=1, ..., N), the position information of the target object in the i-th video frame may include: the coordinates and size of the center point of the detection frame in the i-th video frame, wherein the detection frame is a polygonal frame surrounding the target object.

402. Add the position information of the target object in the target video to the target video to obtain a processed target video.

After obtaining the position information of the target object in the target video, the position information of the target object in the target video may be added to the target video, thereby obtaining a processed target video.

Specifically, the processed target video can be obtained by:

(1) For the i-th video frame among the N video frames of the target video, feature extraction (e.g., convolution, etc.) may be performed on the i-th video frame first, thereby obtaining a first feature of the i-th video frame. It should be noted that the first feature of the i-th video frame is usually a multi-dimensional feature, and the i-th video frame itself is usually a three-dimensional (i.e., three-channel) image. The dimension of the first feature of the i-th video frame may be the same as or different from the dimension of the i-th video frame, and there is no limitation here.

(2) After obtaining the first feature of the i-th video frame, the position information of the target object in the i-th video frame and the first feature of the i-th video frame can be fused (superimposed) to obtain the second feature of the i-th video frame. It should be noted that the first feature of the i-th video frame is usually a multi-dimensional feature, the second feature of the i-th video frame is also a multi-dimensional feature, and the dimension of the first feature of the i-th video frame and the second feature of the i-th video frame are usually the same.

(3) After obtaining the second feature of the i-th video frame, feature extraction (e.g., aggregation and convolution, etc.) can be performed on the second feature to obtain the processed i-th video frame. It should be noted that the second feature of the i-th video frame is usually a multi-dimensional feature, and the processed i-th video frame itself is usually a three-dimensional (i.e., three-channel) image. The dimension of the second feature of the i-th video frame can be The dimension of the i-th video frame after processing remains the same or may be different, and there is no restriction here.

(4) For the remaining video frames except the i-th video frame among the N video frames, the same operation as that performed on the i-th video frame can be performed on the remaining video frames, so that N processed video frames can be finally obtained, that is, the processed target video.

For example, as shown in FIG5 (FIG5 is a schematic diagram of the fusion of video frames and position information provided by an embodiment of the present application), in a mining drilling scene, a person and a machine are drilling in a tunnel in the mining field. After the drilling is completed, a video of the tunnel can be collected in real time. The content of the video is the process of the person cooperating with the machine to take out the drill rod that has been driven into the side wall of the tunnel. In this process, the number of drill rods taken from the machine by the person is the number of drill rods driven into the side wall by the person cooperating with the machine, that is, the number of times the person cooperates with the machine to drill. After obtaining the video, the target detection model can be used to perform target detection on the video, so as to obtain the position information of the machine in the video, the position information of the person in the video, the position information of the drill rod in the video, and the position information of the person's hand in the video.

Since the video contains multiple video frames, for any video frame, feature extraction can be performed on the video frame first, so as to obtain the initial features of the 4 channels of the video frame. Then, the position information of the machine in the video frame can be fused with the initial features of the 1st channel of the video frame, so as to obtain the intermediate features of the 1st channel of the video frame, and the position information of the person in the video frame can be fused with the initial features of the 2nd channel of the video frame, so as to obtain the intermediate features of the 2nd channel of the video frame, and the position information of the drill rod in the video frame can be fused with the initial features of the 3rd channel of the video frame, so as to obtain the intermediate features of the 3rd channel of the video frame, and the position information of the hand in the video frame can be fused with the initial features of the 4th channel of the video frame, so as to obtain the intermediate features of the 4th channel of the video frame. Next, feature extraction can be performed on the intermediate features of the 4 channels of the video frame, so as to obtain the processed video frame.

For the remaining video frames except the video frame in the multiple video frames, the same operation as that performed on the video frame can be performed on the remaining video frames, so that multiple processed video frames, that is, processed videos, can be obtained in the end.

403. Perform a second process on the processed target video to obtain an action sequence of the target object, where the action sequence includes multiple actions sorted according to the appearance time in the processed target video, and the multiple actions include the target action.

After obtaining the processed target video, the processed target video can be input into the second model (trained neural network model) to perform a second processing on the processed target video through the second model (for example, temporal action positioning), so as to obtain an action sequence of the target object, where the action sequence includes multiple actions, which usually include several target actions and several remaining actions, and the multiple actions can be sorted according to the appearance time of the multiple actions in the processed target video, for example, target action, remaining actions, target action, remaining actions, target action, remaining actions, and so on.

Still like the above example, after obtaining the processed video, the temporal action positioning model can be used to perform temporal action positioning on the processed video, thereby obtaining 275 actions, including the machine pulling out the drill rod, the person taking off the drill rod, the person putting the drill rod on the ground, the machine pulling out the drill rod, the person taking off the drill rod, the person putting the drill rod on the ground, the machine pulling out the drill rod, the person taking off the drill rod, and the person putting the drill rod on the ground. These 275 actions are sorted according to the time when they appear in the processed video, including 100 actions of the machine pulling out the drill rod, 100 actions of the person taking off the drill rod, and 75 actions of the person putting the drill rod on the ground.

404. Determine the number of target actions of the target object based on the action sequence.

After obtaining the multiple actions of the target object, the multiple actions may be processed to determine the number of target actions of the target object.

Specifically, the number of target actions of the target object can be obtained in the following ways:

(1) After the processed target video is subjected to the second processing by using the second model, not only the action sequence of the target object can be obtained, but also the confidences of multiple actions in the action sequence can be obtained.

(2) After obtaining the action sequence of the target object and the confidence of multiple actions, the action with a confidence greater than or equal to the confidence threshold can be selected in the action sequence. The confidence threshold can be obtained in a variety of ways: (2.1) After obtaining the confidence of these multiple actions, the confidence of these multiple actions can be calculated first to obtain the statistical information of the confidence of these multiple actions (for example, the variance and average of the confidence of these multiple actions, etc.). Then, the position information of the target object in the target video, the statistical information of the confidence of these multiple actions, and the pixel information of the target video (for example, the inter-frame pixel value difference and the average pixel value of N video frames of the target video, etc.) can be calculated to obtain the confidence threshold. (2.2) The confidence threshold can be a preset value, and the size of the preset value can be set according to actual needs, and there is no restriction here.

Still like the above example, after getting 275 actions and the confidence of 275 actions, it is found that the confidence of 30 actions out of 100 "machine pulls out the drill rod" is lower than the confidence threshold, so these 30 actions can be eliminated. Similarly, it is found that the confidence of 30 actions out of 100 "people take down the drill rod" is lower than the confidence threshold, so these 30 actions can be eliminated, and it is found that the confidence of 25 actions out of 75 "people put the drill rod on the ground" is lower than the confidence threshold, so these 25 actions can be eliminated. In this way, only the actions with confidence higher than or equal to the confidence threshold are retained. 190 actions worth.

(3) After obtaining actions whose confidence is greater than or equal to the confidence threshold, actions whose order is consistent with the preset order can be selected from the actions whose confidence is greater than or equal to the confidence threshold.

Still like the above example, after getting 190 actions, since the preset order is machine pulls out the drill rod → person takes down the drill rod → person puts the drill rod on the ground, then, among the 190 actions, it is found that the order of 20 actions in the 70 "machine pulls out the drill rod" does not conform to the preset order, and these 20 actions can be eliminated. Similarly, it is found that the order of 20 actions in the 70 "person takes down the drill rod" does not conform to the preset order, and these 20 actions can be eliminated. It is found that the order of 10 actions in the 50 "person puts the drill rod on the ground" does not conform to the preset order, and these 10 actions can be eliminated. In this way, only 140 actions whose order conforms to the preset order are retained.

(4) After obtaining the actions whose order conforms to the preset order, the target video can be divided into two sub-videos, the content presented by the first sub-video is the work content, and the content presented by the second sub-video is the non-work content. Then, among the actions whose order conforms to the preset order, the action appearing in the first sub-video can be selected.

Still as in the above example, as shown in Figure 6 (Figure 6 is a schematic diagram of video segmentation provided by an embodiment of the present application), the inter-frame pixel value difference of multiple video frames of the video can be calculated, and the video can be divided into operation segments (segments in the box in Figure 6) and non-operation segments based on the information of the inter-frame pixel value difference. Then, among these 140 actions, it is found that 15 of the 50 "machine pulls out the drill rod" actions do not appear in the operation segment, and these 15 actions can be eliminated. Similarly, it is found that 15 of the 50 "people take off the drill rod" actions do not appear in the operation segment, and these 15 actions can be eliminated. It is found that 20 of the 40 "people put the drill rod on the ground" actions do not appear in the operation segment, and these 20 actions can be eliminated. In this way, only 90 actions that appear in the operation segment are retained.

(5) After obtaining the actions appearing in the sub-video, the total number of target actions of the target object can be determined in the actions appearing in the sub-video, which is equivalent to determining the number of target actions of the target object.

Still taking the above example, among the 90 actions, there are 35 "people taking down drill rods". Therefore, it can be determined that the number of drilling times in which people cooperated with the machine was 35, that is, 35 drill rods were driven into the side wall of the tunnel.

Furthermore, after all target actions are determined, if it is determined that the interval between the occurrence times of any two adjacent target actions is outside the preset range, a prompt may be given to the management personnel. The staff may determine whether to insert an additional target action between the two target actions and issue an instruction. If the instruction indicates that an additional target action needs to be inserted, the additional target action is inserted between the two target actions and the number of target actions is updated. If the instruction indicates that an additional target action does not need to be inserted, the number of the original target action is retained.

Still taking the above example, among the 35 "people taking down the drill rod", if the interval between the occurrence time of two certain actions is 2 minutes, and the preset range is 1 minute, the manager can be prompted whether to insert 1 "person taking down the drill rod" between the two actions. If the instruction issued by the manager is necessary, the number of "person taking down the drill rod" can be updated to 36, that is, the number of drilling times in which people cooperate with the machine is updated to 36.

It should be understood that this embodiment is merely a schematic introduction to executing all of (2) to (4). In actual applications, any one or part of (2) to (4) may also be executed, and this is not limited here.

In addition, the method provided in the embodiment of the present application can be compared with the method provided in the related art. The comparison results are shown in Figure 7 (Figure 7 is a schematic diagram of the comparison results provided in the embodiment of the present application). The accuracy and recall rate of the method provided in the embodiment of the present application are both over 95%, due to the method provided in the related art.

In an embodiment of the present application, after acquiring the target video, the target video may be first processed for the first time to obtain the position information of the target object in the target video. Then, the position information of the target object in the target video may be added to the target video to obtain the processed target video. Then, the processed target video may be processed for the second time to obtain multiple actions of the target object, and these multiple actions are sorted according to the time of appearance in the processed target video. Finally, the multiple actions may be processed to obtain the number of target actions of the target object. In the aforementioned process, not only the influence of the position information of the target object is considered, but also the mutual influence between the multiple actions of the target object, that is, the influence of the time sorting of the target action and the remaining actions, and the factors considered are relatively comprehensive, which is conducive to improving the accuracy of the final counting result for the target action.

The above is a detailed description of the action counting method provided in the embodiment of the present application. The following is an introduction to the action counting device provided in the embodiment of the present application. FIG8 is a structural schematic diagram of the action counting device provided in the embodiment of the present application. As shown in FIG8 , the device includes:

A first processing module 801 is used to perform a first processing on the target video to obtain position information of the target object in the target video;

An adding module 802 is used to add the position information of the target object in the target video to the target video to obtain a processed target video;

The second processing module 803 is used to perform a second processing on the processed target video to obtain an action sequence of the target object. comprising a plurality of actions sorted according to the time of appearance in the processed target video, the plurality of actions including the target action;

The first determination module 804 is configured to determine the number of target actions of the target object based on the multiple actions.

In an embodiment of the present application, after acquiring the target video, the target video may be first processed for the first time to obtain the position information of the target object in the target video. Then, the position information of the target object in the target video may be added to the target video to obtain the processed target video. Then, the processed target video may be processed for the second time to obtain the action sequence of the target object, and the action sequence may include multiple actions, which usually include the target action and the remaining actions, and the multiple actions are sorted according to the appearance time in the processed target video. Finally, the action sequence may be processed to obtain the number of target actions of the target object. In the aforementioned process, not only the influence of the position information of the target object is considered, but also the mutual influence between the multiple actions of the target object, that is, the influence of the time sorting of the target action and the remaining actions, and the factors considered are relatively comprehensive, which is conducive to improving the accuracy of the final counting result for the target action.

In one possible implementation, the target video includes N video frames, N≥2, and a module 802 is added to: perform feature extraction on the i-th video frame to obtain a first feature of the i-th video frame; fuse the position information of the target object in the i-th video frame and the first feature to obtain a second feature of the i-th video frame; perform feature extraction on the second feature to obtain a processed i-th video frame, i=1,...,N.

In one possible implementation, the device also includes: an acquisition module, used to obtain confidences of multiple actions, where the confidences are obtained by performing a second processing on the processed target video; a first determination module 804, used to: determine actions whose confidences are greater than or equal to a confidence threshold in the action sequence; and determine the number of target actions of the target object based on actions whose confidences are greater than or equal to the confidence threshold.

In one possible implementation, the confidence threshold is determined based on the position information of the target object in the target video, statistical information of the confidences of multiple actions, and pixel information of the target video, or the confidence threshold is a preset value.

In one possible implementation, the first determination module 804 is further used to: determine, among actions whose confidence is greater than or equal to a confidence threshold, actions whose order conforms to a preset order; and determine the number of target actions of the target object based on the actions whose order conforms to the preset order.

In one possible implementation, the device also includes: a segmentation module, which is used to segment the target video to obtain a sub-video presenting the work content; a second determination module, which is used to determine the action appearing in the sub-video in the action sequence; and a first determination module 804, which is used to determine the number of target actions of the target object based on the action appearing in the sub-video.

In one possible implementation, the first processing is target detection, and the second processing is temporal action positioning.

It should be noted that the information interaction, execution process, etc. between the modules/units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. The specific contents can be referred to the description in the method embodiment shown above in the embodiment of the present application, and will not be repeated here.

The embodiment of the present application also relates to an execution device, and FIG. 9 is a structural schematic diagram of the execution device provided by the embodiment of the present application. As shown in FIG. 9, the execution device 900 can be specifically manifested as a mobile phone, a tablet, a laptop computer, an intelligent wearable device, a server, etc., which is not limited here. Among them, the first model and the second model described in the corresponding embodiment of FIG. 4 can be deployed on the execution device 900, which is used to implement the function of action counting in the corresponding embodiment of FIG. 4. Specifically, the execution device 900 includes: a receiver 901, a transmitter 902, a processor 903 and a memory 904 (wherein the number of processors 903 in the execution device 900 can be one or more, and FIG. 9 takes one processor as an example), wherein the processor 903 may include an application processor 9031 and a communication processor 9032. In some embodiments of the present application, the receiver 901, the transmitter 902, the processor 903 and the memory 904 may be connected via a bus or other means.

The memory 904 may include a read-only memory and a random access memory, and provides instructions and data to the processor 903. A portion of the memory 904 may also include a non-volatile random access memory (NVRAM). The memory 904 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

The processor 903 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

The method disclosed in the above embodiment of the present application can be applied to the processor 903, or implemented by the processor 903. The processor 903 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 903 or an instruction in the form of software. The above processor 903 can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include a dedicated integrated circuit. (application specific integrated circuit, ASIC), field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 903 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 904, and the processor 903 reads the information in the memory 904 and completes the steps of the above method in combination with its hardware.

The receiver 901 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 902 can be used to output digital or character information through the first interface; the transmitter 902 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 902 can also include a display device such as a display screen.

In an embodiment of the present application, in one case, the processor 903 is used to process the target video through the first model and the second model in the embodiment corresponding to Figure 4, etc., so as to obtain a processing result of the target video.

The embodiment of the present application also relates to a training device, and FIG. 10 is a structural schematic diagram of the training device provided by the embodiment of the present application. As shown in FIG. 10, the training device 1000 is implemented by one or more servers. The training device 1000 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1014 (for example, one or more processors) and a memory 1032, and one or more storage media 1030 (for example, one or more mass storage devices) storing application programs 1042 or data 1044. Among them, the memory 1032 and the storage medium 1030 can be short-term storage or permanent storage. The program stored in the storage medium 1030 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processor 1014 can be configured to communicate with the storage medium 1030 to execute a series of instruction operations in the storage medium 1030 on the training device 1000.

The training device 1000 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1058; or, one or more operating systems 1041, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

Specifically, the training device can complete model training to obtain a first model and a second model, etc., which can be applied to the aforementioned action counting method.

An embodiment of the present application also relates to a computer storage medium, in which a program for signal processing is stored. When the program is run on a computer, the computer executes the steps executed by the aforementioned execution device, or the computer executes the steps executed by the aforementioned training device.

An embodiment of the present application also relates to a computer program product, which stores instructions, which, when executed by a computer, enable the computer to execute the steps executed by the aforementioned execution device, or enable the computer to execute the steps executed by the aforementioned training device.

The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device end, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

Specifically, please refer to FIG. 11 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be a neural network processor NPU 1100. NPU 1100 is mounted on the host CPU (Host CPU) as a coprocessor, and tasks are assigned by the Host CPU. The core part of the NPU is the operation circuit 1103, which is controlled by the controller 1104 to extract matrix data from the memory and perform multiplication operations.

In some implementations, the operation circuit 1103 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1103 is a two-dimensional systolic array. The operation circuit 1103 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1103 is a general-purpose matrix processor.

For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes matrix B from the weight memory 1102 The corresponding data is cached on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1101 and performs matrix operation with the matrix B. The partial result or the final result of the matrix is stored in the accumulator 1108.

The unified memory 1106 is used to store input data and output data. The weight data is directly transferred to the weight memory 1102 through the direct memory access controller (DMAC) 1105. The input data is also transferred to the unified memory 1106 through the DMAC.

BIU stands for Bus Interface Unit, that is, bus interface unit 1113, which is used for the interaction between AXI bus and DMAC and instruction fetch buffer (IFB) 1109.

The bus interface unit 1113 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1109 to obtain instructions from the external memory, and is also used for the storage unit access controller 1105 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1106 or to transfer weight data to the weight memory 1102 or to transfer input data to the input memory 1101.

The vector calculation unit 1107 includes multiple operation processing units, and when necessary, further processes the output of the operation circuit 1103, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional/fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of the predicted label plane, etc.

In some implementations, the vector calculation unit 1107 can store the processed output vector to the unified memory 1106. For example, the vector calculation unit 1107 can apply a linear function; or, a nonlinear function to the output of the operation circuit 1103, such as linear interpolation of the predicted label plane extracted by the convolution layer, and then, for example, a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 1107 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1103, for example, for use in a subsequent layer in a neural network.

An instruction fetch buffer 1109 connected to the controller 1104 is used to store instructions used by the controller 1104;

Unified memory 1106, input memory 1101, weight memory 1102 and instruction fetch memory 1109 are all on-chip memories. External memories are private to the NPU hardware architecture.

The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, training equipment, or data center to another website, computer, training equipment, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage The medium can be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, etc. that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)).

Claims (17)

An action counting method, characterized in that the method comprises: Performing a first process on the target video to obtain position information of the target object in the target video; Adding the position information of the target object in the target video to the target video to obtain a processed target video; Performing a second process on the processed target video to obtain an action sequence of the target object, wherein the action sequence includes a plurality of actions sorted according to the appearance time in the processed target video, and the plurality of actions include a target action; Based on the action sequence, the number of the target action is determined. The method according to claim 1, characterized in that the target video includes N video frames, N ≥ 2, and the adding the position information to the target video to obtain the processed target video comprises: Performing feature extraction on the i-th video frame to obtain a first feature of the i-th video frame; Fusing the position information of the target object in the i-th video frame and the first feature to obtain a second feature of the i-th video frame; Feature extraction is performed on the second feature to obtain the i-th video frame after processing, where i=1, ..., N. The method according to claim 1 or 2, characterized in that the method further comprises: Obtaining confidences of the multiple actions, where the confidences are obtained by performing a second process on the processed target video; The determining the number of the target action based on the action sequence includes: In the action sequence, determining an action whose confidence is greater than or equal to a confidence threshold; Based on the actions whose confidence is greater than or equal to a confidence threshold, the number of the target actions is determined. The method according to claim 3 is characterized in that the confidence threshold is determined based on the position information of the target object in the target video, the statistical information of the confidence of the multiple actions and the pixel information of the target video, or the confidence threshold is a preset value. The method according to claim 3 or 4, characterized in that the determining the number of the target action based on the action sequence further comprises: Among the actions whose confidence is greater than or equal to the confidence threshold, determine the actions whose order conforms to the preset order; Based on the actions whose order matches the preset order, the number of times of the target action is determined. The method according to any one of claims 1 to 5, characterized in that the method further comprises: Segmenting the target video to obtain sub-videos presenting the work content; In the action sequence, determining an action that appears in the sub-video; The determining the number of the target action based on the action sequence includes: Based on the action appearing in the sub-video, the number of times the target action occurs is determined. The method according to any one of claims 1 to 6 is characterized in that the first processing is target detection and the second processing is temporal action positioning. An action counting device, characterized in that the device comprises: A first processing module, used to perform a first processing on the target video to obtain position information of the target object in the target video; An adding module, used for adding the position information of the target object in the target video to the target video to obtain a processed target video; A second processing module is used to perform a second processing on the processed target video to obtain an action sequence of the target object, wherein the action sequence is a plurality of actions sorted according to the appearance time in the processed target video, and the plurality of actions include a target action; The first determination module is used to determine the number of target actions of the target object based on the action sequence. The device according to claim 8, wherein the target video comprises N video frames, N ≥ 2, and the adding module is used to: Performing feature extraction on the i-th video frame to obtain a first feature of the i-th video frame; Fusing the position information of the target object in the i-th video frame and the first feature to obtain a second feature of the i-th video frame; Feature extraction is performed on the second feature to obtain the i-th video frame after processing, where i=1, ..., N. The device according to claim 8 or 9, characterized in that the device further comprises: An acquisition module, used for acquiring confidences of the multiple actions, wherein the confidences are obtained by performing a second process on the processed target video; The first determining module is used to: In the action sequence, determining an action whose confidence is greater than or equal to a confidence threshold; Based on the actions whose confidence is greater than or equal to a confidence threshold, the number of the target actions is determined. The device according to claim 10 is characterized in that the confidence threshold is determined based on the position information of the target object in the target video, the statistical information of the confidence of the multiple actions and the pixel information of the target video, or the confidence threshold is a preset value. The device according to claim 10 or 11, characterized in that the first determining module is further used to: Among the actions whose confidence is greater than or equal to the confidence threshold, determine the actions whose order conforms to the preset order; Based on the actions whose order matches the preset order, the number of times of the target action is determined. The device according to any one of claims 8 to 12, characterized in that the device further comprises: A segmentation module, used to segment the target video to obtain sub-videos presenting the operation content; A second determination module, configured to determine, in the action sequence, an action that appears in the sub-video; The first determination module is used to determine the number of times the target action occurs based on the action that appears in the sub-video. The device according to any one of claims 8 to 13 is characterized in that the first processing is target detection and the second processing is temporal action positioning. An action counting device, characterized in that the device comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the device executes the method according to any one of claims 1 to 7. A computer storage medium, characterized in that the computer storage medium stores one or more instructions, and when the instructions are executed by one or more computers, the one or more computers implement the method described in any one of claims 1 to 7. A computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the method according to any one of claims 1 to 7.