WO2018149376A1 - Video abstract generation method and device - Google Patents

Abstract

A video abstract generation method and device. The video abstract generation method comprises: obtaining a target search condition, and searching an established database to obtain a first track set containing tracks satisfying the target search condition (S101); classifying at least two tracks, which overlap with each other, in the first track set into a group according to overlapping state information, each group being determined as a combined track (S102); performing translation on tracks, which satisfies a preset translation condition, in tracks to be translated to the same target time period along the timeline (S103); obtaining a first target original image for each track in the target time period from the database (S104); obtaining a first abstract background image for generating a video abstract (S105); and splicing the first target original images and the first abstract background image to generate the video abstract (S106). The method can improve the visual effect of a video abstract generated in the case that a target track is complex.

Description

Method and device for generating video summary

The present application claims the priority of the Chinese Patent Application entitled "A Method and Apparatus for Generating a Video Abstract" by the Chinese Patent Office, filed on Feb. 17, 2017, the entire disclosure of which is hereby incorporated by reference. in.

Technical field

The present application relates to the field of video processing technologies, and in particular, to a method and an apparatus for generating a video digest.

Background technique

With the ever-increasing demands for video data processing and the increasing volume of video data, users hope to be able to create a summary for a long video, and use a quick view to make better use of the video, for example, to maintain social security, combat Video surveillance of crimes. Therefore, video summary technology came into being. The video summary technology analyzes the structure and content of the video, extracts the meaningful part from the original video, that is, the moving target, and combines the moving target with the background scene in a specific way to form a simple and fully capable video content. A summary of a video is a simple summary of long video content, usually represented by a sequence of static or dynamic images, and the original information is preserved.

The corresponding technique for generating a video summary is a technique for generating a video summary based on a target object, and the method includes the following steps: First, by analyzing an input video, generating a video structured description file, and establishing a correlation according to the video structured description file; a database, wherein the video structured description file includes attribute information of the target object in the video and trajectory information of the target object; secondly, the established database is retrieved to extract the trajectory information of the moving target; and finally, each target object is further The trajectory is analyzed, the target trajectory is translated on the time axis, and the trajectory of the target object at different times is arranged in the same picture to generate a summary video. The technology can meet the user's need to generate a video summary for a specific target object, and generate a video summary with a short duration, a compact motion, and a high concentration ratio.

However, when the trajectory of the target object is arranged, the actual situation is complicated, and there are cases where the trajectories of the plurality of target objects overlap. Since the above technique repeats the trajectory of the target object, the trajectory of the overlapping target object is performed. Excluded, when the trajectories of multiple target objects overlap, the summary video loses the information of the associated target object; and due to the loss of the associated target object information, a target object is frequently generated in the generated video summary. And the phenomenon of disappearing, resulting in poor visual effects.

Summary of the invention

The purpose of the embodiment of the present application is to provide a method and a device for generating a video summary, so as to improve the visual effect of the video summary generated when the target trajectory is complicated. The specific technical solutions are as follows:

In a first aspect, the embodiment of the present application provides a method for generating a video summary, where the method includes:

Obtaining a target retrieval condition, and retrieving the established database to obtain a first trajectory set including a trajectory that meets the target retrieval condition, wherein the database stores each trajectory extracted from a video frame containing the target object Trajectory information and target original image, wherein the trajectory information of each trajectory includes: overlapping state information with other trajectories;

And dividing, according to the overlapping state information, at least two tracks that overlap in the first set of tracks into a group, each group being determined as a combined track;

Translating a trajectory of the to-be-translated trajectory that meets the preset translation condition to the same target time period along the time axis, wherein the trajectory to be translated includes: the combined trajectory and/or the overlapping of the first trajectory set Trajectory

Acquiring, from the database, a first target original image corresponding to each track in the target time period;

Obtaining a first summary background image for generating a video summary;

The first target original image is spliced with the first abstract background image to generate a video summary.

Optionally, the target retrieval condition includes: retrieving a time period and/or retrieval attribute information of the target object;

When the retrieval condition includes only the retrieval time period, the searching for the established database includes:

Searching the database according to the retrieval time period to obtain a trajectory of each target object in the retrieval time period;

When the retrieval condition includes only the retrieval attribute information of the target object, the searching for the established database includes:

Searching the database according to the retrieval attribute information of the target object, and obtaining a trajectory of each target object that matches the retrieval attribute information;

When the retrieval condition includes the retrieval time period and the retrieval attribute information of the target object, the searching for the established database includes:

And searching the database according to the retrieval time period and the retrieval attribute information of the retrieval target object, and obtaining a trajectory of each target object that matches the retrieval attribute in the retrieval time period.

Optionally, before the acquiring the target search condition, the method further includes:

Extract each target object from the input video;

Extracting trajectory information and attribute information of each target object, where the trajectory information includes: movement information of the trajectory and overlapping state information with other trajectories;

Storing the track information and attribute information of each target object into a video structured target description file;

Generating the database according to the video structured target description file;

Extracting, from a video frame that includes the target object, an original image and a mask map of each frame corresponding to the trajectory information, and determining, according to the original image and the mask map, a target original of each frame corresponding to the trajectory information. Figure

The original image, the mask map, and the target original image are stored in the database.

Optionally, the mask map of each frame corresponding to the track information is extracted from the video frame that includes the target object, including:

Extracting a motion mask of the target object from a video frame containing the target object;

Determining an initial mask map according to the motion mask;

Determining an edge point set of the initial mask map;

Extracting a convex set in the set of edge points to form a convex set of points of the mask map;

Filling the convex hull corresponding to the convex hull point set to obtain a final mask map.

Optionally, the trajectory of the to-be-translated trajectory that meets the preset translation condition is translated to the same target time period along the time axis, including:

Establish a queue to be translated and a summary queue;

And a track that is not in the target time segment in the first track set is used as a to-be-translated track, and is stored in a to-be-translated queue, where the to-be-translated track is not in the first track set. a combined trajectory in the target time period and a trajectory in which no overlap occurs;

Storing the trajectory in the target time period in the first trajectory set into the summary queue;

Extracting the current to-be-translated trajectory from the to-be-translated queue, and obtaining a rectangular frame of each target object in the video frame corresponding to the current to-be-translated trajectory according to the original image in the database;

Calculating an overlapping area between a rectangular frame of each target object and a rectangular frame of the target object in the video frame corresponding to each track that has been stored in the summary queue;

When the overlapping area is less than or equal to a preset overlapping parameter threshold, the current to-be-translated trajectory is translated to the target time period, and stored in the summary queue;

And the step of acquiring the first target original image corresponding to each track of the target time period from the database, including:

Obtaining a first target original image corresponding to each track in the summary queue.

Optionally, the track information of each track further includes: a target box information set of the target object;

The splicing the first target original image and the first abstract background image to generate a video summary, including:

Determining, according to the target frame information set of the target object in the trajectory information, a first position of the first target original image in the first abstract background image;

Copying each first target original image to a corresponding first position in the first abstract background image to generate a video summary.

Optionally, the copying the first target original image to the corresponding first location in the first summary background image includes:

If the target objects in the first target original image overlap, the pixel value of the overlapping portion of the corresponding trajectory is set to be the mean value of the target original image pixel value of each target object, and the pixel value of the non-overlapping portion is each target. The pixel value of the target original image of the object, and the image to be copied is obtained;

Copying each first target original image to a corresponding first location in the first summary background image to generate a video summary, including:

Copying each of the first target original images that do not overlap each of the to-be-copied image and the target object to a corresponding first position in the first summary background image to generate a video summary.

Optionally, before storing the original image, the mask map, and the target original image in the database, the method further includes:

Obtain a summary background image according to a preset period;

Storing a summary background image of each cycle acquired into the database;

The obtaining a first abstract background image for generating a video summary includes:

Dividing the target time period into time sub-segments corresponding to preset periods according to time corresponding to each preset period;

Determining, in the target time period, a first preset period corresponding to a time sub-segment containing the most track;

Obtaining, from the database, a first summary background image corresponding to the first preset period.

Optionally, before the acquiring the target search condition, the method further includes:

Display the user interaction interface according to user instructions;

Receiving and saving a target retrieval condition input by the user through the user interaction interface, a preset translation condition, and a preset period for generating a summary background image;

The method further includes:

Performing the step of retrieving the established database when receiving a startup request input by the user through the user interaction interface;

When the interrupt request input by the user through the user interaction interface is received, the process of generating the video summary is ended.

In a second aspect, the embodiment of the present application provides a device for generating a video summary, where the device includes:

a retrieval module, configured to acquire a target retrieval condition, and retrieve the established database to obtain a first trajectory set including a trajectory that meets the target retrieval condition, wherein the database stores the video frame from the target object Extracting the trajectory information of each trajectory and the target original image, wherein the trajectory information of each trajectory includes: overlapping state information with other trajectories;

a combination module, configured to divide at least two tracks that overlap in the first set of tracks into a group according to the overlapping state information, and each group is determined as a combined track;

a panning module for translating a trajectory of the to-be-translated trajectory that satisfies the preset panning condition to the same target time segment along the time axis, wherein the to-be-translated trajectory comprises: the combined trajectory and/or the first trajectory set No overlapping trajectories occur;

a first acquiring module, configured to acquire, from the database, a first target original image corresponding to each track in the target time period;

a second obtaining module, configured to obtain a first abstract background image for generating a video summary;

a splicing module, configured to splicing each of the first target original images and the first abstract background image to generate a video summary.

Optionally, the target retrieval condition includes: retrieving a time period and/or retrieval attribute information of the target object;

When the retrieval condition includes only the retrieval time period, the retrieval module is specifically configured to:

Searching the database according to the retrieval time period to obtain a trajectory of each target object in the retrieval time period;

When the search condition includes only the search attribute information of the target object, the search module is specifically configured to:

And searching the database according to the retrieval attribute information of the target object, and obtaining a trajectory of each target object that matches the retrieval attribute information;

When the retrieval condition includes the retrieval time period and the retrieval attribute information of the target object, the retrieval module is specifically configured to:

And searching the database according to the retrieval time period and the retrieval attribute information of the retrieval target object, and obtaining a trajectory of each target object that matches the retrieval attribute in the retrieval time period.

Optionally, the device further includes:

a first extraction module, configured to extract each target object from the input video;

a second extraction module, configured to extract trajectory information and attribute information of each target object, where the trajectory information includes: movement information of the trajectory, and overlapping state information with other trajectories;

a first storage module, configured to store track information and attribute information of each target object into a video structured target description file;

a database generating module, configured to generate the database according to the video structured target description file;

a third extraction module, configured to extract an original image and a mask map of each frame corresponding to the trajectory information from the video frame that includes the target object, and determine, according to the original image and the mask map, the trajectory information corresponding to the trajectory information The original image of each frame of the frame;

And a second storage module, configured to store the original image, the mask map, and the target original image into the database.

Optionally, the third extraction module includes:

a first extraction submodule, configured to extract a motion mask of the target object from a video frame that includes the target object;

a first determining submodule, configured to determine an initial mask according to the motion mask;

a second determining submodule, configured to determine an edge point set of the initial mask map;

a second extraction submodule, configured to extract a convex set in the set of edge points, and form a convex punctual point set of the mask map;

The filler submodule is configured to fill the convex hull corresponding to the convex hull point set to obtain a final mask map.

Optionally, the translation module includes:

a queue creation sub-module for establishing a queue to be translated and a summary queue;

a first storage sub-module, configured to use a trajectory in the target trajectory that is not in the target time segment as a trajectory to be translated, and store the trajectory in the to-be-translated trajectory, where the trajectory to be translated is the a combined trajectory in a set of trajectories that is not in the target time period and a trajectory that does not overlap;

a second storage submodule, configured to store, in the first time set, a track in the target time period into the summary queue;

a third extraction sub-module, configured to sequentially extract a current to-be-translated trajectory from the to-be-translated queue, and obtain a rectangular frame of each target object in the video frame corresponding to the current to-be-translated trajectory according to the original image in the database ;

An operation submodule, configured to calculate an overlapping area between a rectangular frame of each target object and a rectangular frame of the target object in a video frame corresponding to each track that has been stored in the summary queue;

a third storage submodule, configured to: when the overlapping area is less than or equal to a preset overlapping parameter threshold, translate the current to-be-translated trajectory to the target time segment, and store the trajectory to the summary queue;

The first acquiring module is specifically configured to:

Obtaining a first target original image corresponding to each track in the summary queue.

Optionally, the track information of each track further includes: a target box information set of the target object;

The splicing module includes:

a third determining submodule, configured to determine, according to the target box information set of the target object in the trajectory information, a first position of the first target original image in the first abstract background image;

The video summary generation sub-module is configured to copy each first target original image to a corresponding first position in the first summary background image to generate a video summary.

Optionally, the third determining submodule is specifically configured to:

If the target objects in the first target original image overlap, the pixel value of the overlapping portion of the corresponding trajectory is set to be the mean value of the target original image pixel value of each target object, and the pixel value of the non-overlapping portion is each target. The pixel value of the target original image of the object, and the image to be copied is obtained;

The splicing module is specifically configured to:

Copying each of the first target original images that do not overlap each of the to-be-copied image and the target object to a corresponding first position in the first summary background image to generate a video summary.

Optionally, the device further includes:

An operation module, configured to obtain a summary background image according to a preset period;

a third storage module, configured to store the acquired summary background image of each period into the database;

The second acquiring module includes:

a sub-module, configured to divide the target time segment into a time sub-segment corresponding to the preset period according to a time corresponding to each preset period;

a fourth determining sub-module, configured to determine a first preset period corresponding to the time sub-segment including the most trajectory in the target time period;

The background image obtaining sub-module is configured to obtain, from the database, a first abstract background image corresponding to the first preset period.

Optionally, the device further includes:

a display module, configured to display a user interaction interface according to a user instruction;

a receiving module, configured to receive and save a target retrieval condition input by the user through the user interaction interface, a preset translation condition, and a preset period for generating a summary background image;

An execution module, configured to perform the step of searching the established database when receiving a startup request input by the user through the user interaction interface;

The ending module is configured to end the process of generating the video summary when receiving the interrupt request input by the user through the user interaction interface.

In a third aspect, an embodiment of the present application provides a computer device, including a processor and a memory, where

a memory, the memory, for storing a computer program;

The method of the first aspect of the embodiments of the present application is implemented when the processor is configured to execute a computer program stored in the memory.

In a fourth aspect, the embodiment of the present application provides a storage medium for storing executable code, where the executable code is used to execute at the runtime: the method steps described in the first aspect of the embodiment of the present application.

In a fifth aspect, an embodiment of the present application provides an application program for performing, at runtime, the method steps described in the first aspect of the embodiments of the present application.

A method and device for generating a video summary provided by an embodiment of the present application, by searching an established database, obtaining a trajectory that satisfies a target retrieval condition, and combining the trajectories that overlap in the trajectory into a combined trajectory, and then shifting differently The combined trajectory of the time period and the trajectory that does not overlap to the same target time period, and finally splicing the target original image and the abstract background image corresponding to the trajectory in the target time segment to generate a video summary; When the overlapping trajectories of the plurality of target objects are combined into one combined trajectory, the whole trajectory is shifted on the time axis to avoid losing some trajectories in the overlapping trajectories during translation, thereby improving the visual effect of generating the video summary.

DRAWINGS

In order to more clearly illustrate the embodiments of the present application and the technical solutions of the prior art, the following description of the embodiments and the drawings used in the prior art will be briefly introduced. Obviously, the drawings in the following description are only Some embodiments of the application may also be used to obtain other figures from those of ordinary skill in the art without departing from the scope of the invention.

FIG. 1 is a schematic flowchart diagram of a method for generating a video summary according to an embodiment of the present application;

2 is a schematic flowchart of a method for generating a video summary according to another embodiment of the present application;

3 is a schematic diagram of a specific process of S205 in the embodiment shown in FIG. 2;

4 is a schematic diagram of a specific process of S105 in the embodiment shown in FIG. 2;

FIG. 5 is a schematic diagram of a specific process of S103 in the embodiment shown in FIG. 2;

FIG. 6 is a schematic flowchart diagram of a method for generating a video summary according to still another embodiment of the present application;

FIG. 7 is a schematic structural diagram of a device for generating a video summary according to an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a device for generating a video summary according to another embodiment of the present application;

FIG. 9 is a schematic diagram showing a specific structure of a third extraction module 850 in the embodiment shown in FIG. 8;

FIG. 10 is a schematic diagram showing a specific structure of the translation module 730 in the embodiment shown in FIG. 8;

FIG. 11 is a schematic structural diagram of a second acquiring module 750 in the embodiment shown in FIG. 8;

FIG. 12 is a schematic structural diagram of a device for generating a video summary according to still another embodiment of the present application.

detailed description

In order to make the objects, technical solutions, and advantages of the present application more comprehensible, the present application will be further described in detail below with reference to the accompanying drawings. It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

In order to improve the visual effect of the video summary generated in the case where the target trajectory is complicated, the embodiment of the present application provides a method and an apparatus for generating a video summary.

A method for generating a video digest provided by the embodiment of the present application is first introduced.

An execution body of a method for generating a video summary provided by an embodiment of the present application may be a video summary controller having a function of generating a video summary. The manner of implementing the method for generating a video digest provided by the embodiment of the present application may be at least one of software, a hardware circuit, and a logic circuit that are disposed in the video summary controller. The video summary controller can be applied to a video surveillance system or to a server side of a video website.

As shown in FIG. 1 , a method for generating a video summary provided by an embodiment of the present application may include the following steps:

S101. Acquire a target search condition, and perform a search on the established database to obtain a first track set including a track that meets the target search condition.

The trajectory information of each trajectory extracted from the video frame containing the target object and the target original image are stored in the database, and the trajectory information of each trajectory includes overlapping state information with other trajectories; the database may further include: Attribute information of each target object, a video frame original image, a mask map and/or a background image of the target object in each frame original image; the overlap state information may be an identifier of the trajectory of the overlapping target object, the identifier It may be the name of the track, or the track number, or other feature symbols used to characterize the track; the track information may include: an identifier of the track of the target object, the number of track points of the target object, and each track point of the target object. The frame number, the time information of the trajectory of the target object, the spatial information of the trajectory of the target object, and/or the overlapping state information with other trajectories; the attribute information may include: the appearance time of the target object, the moving direction of the target object, the license plate Number, model, vehicle brand, vehicle color, person's dress color, person's age, person's height, person is Wearing glasses and / or whether the person backpack bag and other information. The target original image may be an original image of the video frame, or may be an image obtained by combining the mask image with the original image of the video frame.

The target retrieval condition may be any combination of all the information in the attribute information of the target object, or may be a certain time period. The established database is retrieved, and the trajectory of the target object that meets the target retrieval condition is retrieved from the database.

Optionally, the target retrieval condition includes: retrieving the time period and/or the retrieval attribute information of the target object.

When the retrieval condition includes only the retrieval time period, the steps of searching the established database may include:

According to the retrieval time period, the database is retrieved, and the trajectory of each target object in the retrieval time period is obtained.

When the target retrieval condition is a certain retrieval time period, the trajectory of each target object in the retrieval time period is acquired, for example, there is a video with a duration of 5 hours from 7:00 to 12:00, and the target retrieval condition is set. For the retrieval time period: 8:00 to 9:00, the trajectory of each target object in 8:00 to 9:00 is extracted. In this embodiment, the time interval of the original video is intercepted, and the target can be extracted. The video of the object's active time period reduces the input data generated by the subsequent video summary and reduces the amount of calculation.

When the search condition includes only the search attribute information of the target object, the step of searching the established database may include:

The database is searched according to the retrieval attribute information of the target object, and the trajectory of each target object matching the retrieval attribute information is obtained.

When the target search condition is any combination of all the information in the attribute information of the target object, the established database is searched, and the trajectory of the target object having the same target search condition is retrieved from the database. For example, the target search condition is: a man who is 1.75 meters in height and between 40 years old and 45 years old, wearing a white down jacket, according to the target search condition, can extract from the database each target object that satisfies the target search condition. Track. In this embodiment, the target retrieval condition defines the target object in the video, can ensure the accuracy of the target object, and is easier to extract the trajectory of the target object that satisfies the requirement.

When the retrieval condition includes the retrieval time period and the retrieval attribute information of the target object, the step of searching the established database may include:

The database is searched according to the retrieval time period and the retrieval attribute information of the retrieval target object, and the trajectory of each target object matching the retrieval attribute in the retrieval time period is obtained.

In combination with the above two embodiments, the trajectory of the target object that matches the attribute information in the retrieval time period can be obtained in this embodiment. This embodiment not only ensures the activity level of the target object but also defines the attributes of the target object. In the above two embodiments, the obtained trajectory is more accurate.

The target search condition may be preset, that is, the condition of the target search has been set before the video is obtained, and is mostly used for the case of fixing the scene, for example, the same time period of the same time period is repeated for many days in a row. Target objects, for the case of a fixed scene, using the preset target search condition can avoid repeatedly setting the same target search condition; the target search condition can also be input by the user according to the actual situation, for example, the user needs to retrieve a certain time period. The trajectory of the specific target object within the user, the user can set the target search condition according to the attribute information of the target object. This is all reasonable.

When searching the established database, a SQL (Structured Query Language) query can be generated according to the target retrieval condition; the established database is retrieved. Among them, the SQL query statement is the most commonly used statement in the SQL query language. The SQL query language is a database query and programming language for accessing data and querying, updating, and managing databases. SQL query statements are selected by selecting commands. The data of the target search condition is selected in the table of the database, and the data may be a track identifier of the target object, and the track information such as time information and spatial information of the track of the target object may be determined by the track identifier.

S102. Divide at least two trajectories in the first trajectory set that are overlapped into one group according to the overlapping state information, and each group is determined as one combined trajectory.

The overlapping state information may be an identifier of a track of the overlapping target object, the identifier may be a name of the track, or a track number, or other feature symbols used to represent the track; the first track set is a slave A set of trajectories in which a plurality of trajectories of a target object satisfying the target retrieval condition extracted in the database are combined.

In order to improve the effect of the video summary in the case where the trajectory of the target object is complicated, in the step of shifting the trajectory along the time axis, the overlapping trajectories are taken as a whole, and the trajectories that overlap in the first trajectory set are combined to obtain a combined trajectory. . Moreover, the order of the trajectories on the time axis in the combined trajectory needs to remain unchanged.

S103. Translate the trajectory of the to-be-translated trajectory that meets the preset translation condition to the same target time period along the time axis.

The trajectory to be translated includes: a combined trajectory composed of at least two trajectories in the first trajectory set and/or a trajectory in the first trajectory set that does not overlap; the target time period may be preset by the user, It can also be a certain period of time during the playback time of the entire video. The video summary to be generated is not simply spliced by motion segments, but the resulting video summary is concentrated by shifting the trajectory of the target object appearing at different time segments to the same time period. The translation of the trajectory of the target object is a translation of the time of the trajectory and does not include the translation of the spatial position.

S104. Acquire, from the database, a first target original image corresponding to each track in the target time period.

The target original image may be the original image of the video frame, or may be an image obtained by the mask image and the original image of the video frame, and the first target original image is the target original image of any track, in the first target original When the picture is the original image of the video frame, the first target original image may be obtained according to the trajectory information of the target object, or may be obtained according to the attribute information of the target object.

S105. Obtain a first abstract background image for generating a video summary.

In addition to the first target original image of the target object in the video frame, the image formed by the other content is the background image of the video, and the generated video summary cannot include only the first target original image, and should also include the first abstract background image.

In this embodiment, the first abstract background image may be a static abstract background image calculated by a static background determining method, or may be a dynamic abstract background image determined by a dynamic background determining method. When the special scene or video time is too long, the background image of the video at different times will be very different. In order to reduce the influence of the background image on the video summary when the video summary is generated, the effect of the video summary is improved, and different time needs to be saved. The background image of the segment's video is taken as a summary background image.

S106: splicing each first target original image with the first abstract background image to generate a video summary.

The first target original image is spliced with the first abstract background image, and the first target original image may be copied to a position where the target object is located in the first abstract background image. However, since there are overlapping trajectories and unoverlapping trajectories in the translated trajectory, the first target original image cannot be simply copied and pasted to obtain a video summary.

Applying the embodiment, by searching the established database, the trajectories satisfying the target retrieval condition are obtained, and the overlapping trajectories in the trajectories are combined into a combined trajectory, and then the combined trajectories of different time periods are translated and overlapped without overlapping. Tracking to the same target time segment, and finally splicing the target original image and the abstract background image corresponding to the trajectory in the target time segment to generate a video summary; in the embodiment of the present application, when the video summary is generated, overlapping multiple target objects will occur. The trajectories are combined into a combined trajectory, and the whole trajectory is shifted on the time axis to avoid losing some trajectories in the overlapping trajectories during translation, thereby improving the visual effect of generating a video summary.

Optionally, the method for generating the video summary further includes:

Upon receiving the interrupt request input by the user, the flow of the video summary generation is ended; when the startup request input by the user is received, the step of acquiring the target retrieval condition and performing the retrieval on the established database is performed.

In order to improve the user experience during the generation of the video summary, the user can input an interrupt request at any time. For example, the user finds that the target retrieval condition is set incorrectly, and after receiving the interrupt request, the process of generating the video summary is ended; Then, the user can reset the target retrieval condition according to the requirement, and input a startup request, and after receiving the startup request, re-search the established database according to the target retrieval condition.

As shown in FIG. 2, in the method for generating a video summary provided by the embodiment of the present application, before the step of acquiring the target search condition, the method for generating the video summary may further include:

S201, extracting each target object from the input video.

Among them, the target object is a target with characteristic information, such as a character, a car, a ship, and the like.

S202. Extract track information and attribute information of each target object.

The trajectory information may include: an identifier of a trajectory of the target object, a number of trajectory points of the target object, a frame number of each trajectory point of the target object, time information of the trajectory of the target object, spatial information of the trajectory of the target object, and / or information such as overlapping state information with other tracks; attribute information may include: the time of occurrence of the target object, the direction of movement of the target object, the license plate number, the model, the brand of the vehicle, the color of the vehicle, the color of the person's dress, the person Information such as the age, the height of the person, whether the person wears glasses and/or whether the person has a backpack or a bag.

After the input video is obtained, the video structured target extraction is performed on the video. The video structured target extraction includes target object extraction and target attribute extraction, and the target trajectory description file is obtained by the target object extraction; The target attribute description file, the target track description file and the target attribute description file are included in the video structured object description file. The target object extraction and the target attribute extraction can be performed synchronously. The target attribute extraction is to extract one or more video frame images containing the target object by using a preset video frame extraction method, and then combine the attribute classifier to obtain the target of the comprehensive attribute classifier. The target attribute description file for the object. The attribute classifier is used to identify a certain type of attribute of the target object, and the information of the class attribute can be obtained through internal analysis of the attribute classifier. The target object extraction is a specific attribute and a motion attribute of the target object extracted according to the target attribute, combined with the specific attribute and the motion attribute of the target object, and multi-target tracking is performed, and the specific attribute of the tracked target object is associated with the motion attribute. Get the target trajectory description file of the target object. It is guaranteed that the target object and the target object's trajectory correspond one-to-one when multiple targets occur at the same time, preventing the trajectory of the other target object from affecting the trajectory of the target object, and improving the accuracy of the target object extraction.

S203. Store the trajectory information and attribute information of each target object into the video structured target description file.

The video structured target description file is used to store attribute information and track information of the target object. The existing video structured object description file is usually generated on the industrial computer or server, and can also be implemented through an embedded platform, such as DSP (Digital Signal Processor) and ARM (Advanced Reduced Instruction Set). Computer Machines, a reduced instruction set microprocessor).

S204: Generate a database according to the video structured target description file.

After obtaining the video structured target description file, the database is used to establish a database, and all the attribute information is managed by the database, including: the appearance time of the target object, the moving direction of the target object, the license plate number, the model, and the brand of the vehicle. , the color of the vehicle, the color of the person's dress, the age of the person, the height of the person, whether the person wears glasses and/or whether the person has a backpack or the like.

Before establishing the database, the trajectory of the target object is analyzed, and the trajectory information of the target object is extracted. The trajectory information of the target object can be used to extract the overlapping state information of the target object, so that the overlapping state information of the target object is saved in the database. .

S205. Extract, from the video frame that includes the target object, an original image and a mask map of each frame corresponding to the trajectory information, and determine a target original image of each frame corresponding to the trajectory information according to the original image and the mask map.

The target original image is an image of the target object. In this embodiment, the target original image may be obtained by matching the original image of each video frame through a mask map. Since the mask map reflects the contour of the target object, the mask map is only Represents the outline of the target object, not the image content. After the original image of the video frame is matched, the image of the target object of the area of the mask map is obtained, compared to directly extracting the target object from the original image of the video frame. The image is more accurate. The mask map can be extracted by a preset background modeling method, and the preset background modeling method can be any of the color background model method, the average background model method, the Gaussian background model method or the CodeBook background model method. One.

As shown in FIG. 3, a method for generating a video digest in the embodiment of the present application, the step of extracting a mask map of each frame corresponding to the trajectory information from the video frame that includes the target object may include:

S2051: Extract a motion mask of the target object from the video frame that includes the target object.

S2052. Determine an initial mask map according to the motion mask.

The motion mask is two-dimensional data constituting the mask map, and the target can be determined by extracting the motion mask of the target object according to the two-dimensional data of the motion mask and the preset background modeling method. The mask map of the object. The mask map characterizes the outline of the target object and distinguishes the target original image from the background image in the original video frame.

S2053. Determine an edge point set of the initial mask map.

S2054: Extract a convex set in the edge point set to form a convex punctual point set of the mask map.

S2055: Fill the convex hull corresponding to the convex hull point set to obtain a final mask map.

In a complicated scenario, the mask map extraction is incomplete. In order to improve the mask map and further improve the effect of the video summary, after the mask map of the target object is extracted, the mask map can be post-processed. That is, according to the edge point set of the mask map, the convex punctual point set of the mask map is formed, and the convex hull corresponding to the convex hull point set is filled, thereby perfecting the mask map to maximize the contour of the target object.

S206: Store the original image, the mask map, and the target original image in a database.

The original image, the mask map and the target original image corresponding to the target object are stored in the database, so that in the step of generating the video summary, the corresponding target original image can be quickly searched from the database according to the attribute information of the target object.

Optionally, before the step of storing the original image, the mask image, and the target original image in the database, the method for generating the video summary may further include:

In the first step, the summary background image is obtained according to the preset period.

The preset period is a period for saving the summary background image, and may be set by the user according to actual needs, or may be an empirical value preset by a person skilled in the art.

The target original image can be obtained by the mask map and the original image of the video frame. Since the mask map reflects the outline of the target object, the mask map only represents the outline of the target object, and does not contain the image content, and the original video frame. After the image phase and the image of the target object of the region obtained by the mask map, it is more accurate than extracting the image of the target object directly from the original image of the video frame, and the target object is completely extracted, and the image is removed. The background part of the target frame enhances the stitching effect of the original image and the abstract background image. In the process of video analysis, a background image is maintained from beginning to end. This background image is updated every frame. When the preset cycle time is reached, the background image is automatically saved once. The updating method of the background image is: extracting a motion mask map for each frame image, and the pixels are motion foreground, and the pixels of the corresponding background image are not updated, otherwise the pixels of the background image are updated according to the update formula, wherein the update formula is :a=b×k+c×(1-k), a is the background image pixel value of the current frame, b is the background image pixel value of the previous frame, k is the preset value, and k is in the range of 0 to Any number of 1, c is the current video frame pixel value.

In the second step, the summary background image of each cycle obtained is stored in the database.

Since the background image changes during the playback of the video, the background image changes little compared to the target object. Therefore, it is only necessary to periodically save the background image, which not only ensures the authenticity of the background but also increases the background. Too much amount of calculation.

As shown in FIG. 4, a method for generating a video summary in the embodiment of the present application, the step of obtaining a first summary background image for generating a video summary may include:

S1051: The target time segment is divided into time sub-segments corresponding to the preset period according to the time corresponding to each preset period.

When the background image is periodically saved, the target time segment is divided into corresponding time sub-segments, which can ensure that the corresponding abstract background image can be obtained when the subsequent summary background image for generating the video summary is obtained, without requiring more A multi-algorithm algorithm determines a better summary background image, which can effectively save computational effort.

S1052: Determine a first preset period corresponding to a time sub-segment containing the most track in the target time period.

S1053. Obtain a first summary background image corresponding to the first preset period from the database.

If the number of tracks included in a certain time subsection is the most, indicating that the target object has the most frequent motion during that time period, the background image in the time period should be used as the abstract background image of the video summary, with only a small part of the track and the actual background. Does not match, compared to the static background image can more realistically reflect the actual trajectory of the target object, improve the effect of the video summary.

Applying the embodiment, by searching the established database, the trajectories satisfying the target retrieval condition are obtained, and the overlapping trajectories in the trajectories are combined into a combined trajectory, and then the combined trajectories of different time periods are translated and overlapped without overlapping. Tracking to the same target time segment, and finally splicing the target original image and the abstract background image corresponding to the trajectory in the target time segment to generate a video summary; in the embodiment of the present application, when the video summary is generated, overlapping multiple target objects will occur. The trajectories are combined into a combined trajectory, and the whole trajectory is shifted on the time axis to avoid losing some trajectories in the overlapping trajectories during translation, thereby improving the visual effect of generating a video summary. And by establishing a database, saving the attribute information of the target object, the original image of the target, and the like, and increasing the speed of generating the video summary when generating the video summary; the target original image is obtained by matching the mask image with the original image of the video frame, due to the mask The figure embodies the outline of the target object. The mask map only represents the outline of the target object, and does not contain the image content. After the original image is combined with the original image of the video frame, the image of the target object in the area of the mask map is obtained. It is more accurate to extract the image of the target object directly from the original video frame.

As shown in FIG. 5, a method for generating a video summary according to an embodiment of the present disclosure, the step of shifting a trajectory of a to-be-translated trajectory that meets a preset translation condition to the same target time segment along the time axis may include:

S1031: Establish a to-be-translated queue and a summary queue.

The queue to be translated is a queue for storing undistributed target trajectories, and the trajectory that can be stored in the queue to be translated is not yet determined in the database whether the preset condition is satisfied, or the target is not in the first trajectory set. The trajectory in the time period. The summary queue is a queue for storing tracks for generating video summaries.

S1032: A track that is not in the target time segment in the first track set is used as a track to be translated, and is stored in the queue to be translated.

The to-be-translated trajectory is a combined trajectory in the first trajectory set that is not in the target time segment and a trajectory that does not overlap.

S1033. Store the trajectory in the target time segment in the first trajectory set into the summary queue.

S1034: Extract the current to-be-translated trajectory from the queue to be translated, and obtain a rectangular frame of each target object in the video frame corresponding to the current to-be-translated trajectory according to the original image in the database.

The position of the target object in the video frame is actually contained in a rectangular sub-picture. The size of the rectangular sub-picture is related to the target extraction method. For example, if the head target is extracted, the size of the rectangular sub-picture is based on the size of each head target. As far as possible, including the range of the area of any individual head target; if, for example, the pedestrian target needs to be extracted, the size of the rectangular sub-picture is determined based on the size of each pedestrian target and the range of any pedestrian target as much as possible. The trajectory of the target object is formed by a plurality of consecutive video frames, and the trajectory of one target object has a plurality of rectangular sub-pictures, forming a rectangular frame to be translated, and the rectangular frame is a rectangular sub-graph containing the target object and having the smallest area. Box.

S1035. Calculate an overlapping area between the rectangular frame of each target object and the rectangular frame of the target object in the video frame corresponding to each track that has been stored in the summary queue.

S1036: When the overlapping area is less than or equal to the preset overlapping parameter threshold, the current to-be-translated trajectory is translated to the target time segment and stored in the summary queue.

When the overlapping area of the rectangular frames of the two tracks is too large, it means that there are too many overlapping parts of the two tracks, and the two tracks can be approximated as the same track. Therefore, when the track is translated, the overlap of the rectangular frames is not translated. A track with too much area. In this embodiment, an overlapping parameter is preset, and the preset overlapping parameter threshold may be set according to an actual demand situation and a specific attribute of the target object. When the overlapping area of the rectangular frame is greater than the preset overlapping parameter threshold, If the overlapping portion is too large, the track is not stored in the summary queue; the track to be translated to the summary queue is stored only when the overlapping area of the rectangular frame is less than or equal to the preset overlapping parameter threshold.

In this embodiment, by establishing two queues of the to-be-translated queue and the summary queue, the track in the first track set that is not in the target time segment is stored in the queue to be translated, and the track in the target time segment in the first track set is stored to The summary queue stores the track in the queue to be translated that meets the preset overlap condition to the summary queue. With the embodiment, it is possible to avoid a phenomenon in which a disorder occurs when the trajectory is translated.

Optionally, the step of obtaining the first target original image corresponding to each track of the target time segment from the database may include:

Acquire a first target original image corresponding to each track in the summary queue.

The trajectory information has a corresponding relationship with the target object and the trajectory of the target object. Of course, the trajectory information also has a corresponding relationship with the target original image of the target object. Therefore, the first target original image corresponding to the trajectory information can be extracted from the database according to the trajectory information of the trajectory to be translated.

Optionally, the first target original image is spliced with the first abstract background image to generate a video summary, including:

In the first step, the first position of the first target original image in the first summary background image is determined according to the target frame information set of the target object in the trajectory information.

The track information of each track further includes: a target frame information set of the target object, and the target frame information includes coordinates and a width and height of a top left corner of the rectangular frame of the target object, for example, the target frame information is (x, y, w, h), where x is the abscissa of the upper left corner of the rectangular frame of the target object, y is the ordinate of the upper left corner of the rectangular frame of the target object, w is the width of the rectangular frame of the target object, and h is the target object The height of the rectangular frame, of course, the target frame information may also include the coordinates of the center point of the rectangular frame of the target object and the width and height. For example, the target frame information is (m, n, p, q), where m is the rectangle of the target object. The abscissa of the center point of the frame, n is the ordinate of the center point of the rectangle of the target object, p is the width of the rectangle of the target object, and q is the height of the rectangle of the target object. The target forms a set of target frame information in the moving process, and the set includes information such as coordinates, length, direction, and the like of the first target original image in the first abstract background image, and therefore, according to the target frame information of the target object in the trajectory information The set determines the first position of the first target original image in the first summary background image.

In the second step, the first target original image is copied to the corresponding first position in the first abstract background image to generate a video summary.

When splicing the target original image and the abstract background image, if the target original image and the abstract background image are directly spliced, the method used for splicing may cause the target original image to have an error in the position and actual position in the abstract background image. Or, the target original image includes a partial background image, which is different from the background image to be stitched, so that the position of the target original image in the abstract background image does not match the original image; therefore, in this embodiment, the target original image may be The original image of the video frame and the mask image are combined with the obtained image. The target original image does not include the background image, and the true position of the target object in the abstract background image is reflected according to the mask map, and the target original image and the abstract background image are displayed. The splicing can accurately copy the target original image to the contour area outlined by the mask map, which can ensure the effect of splicing the target original image and the abstract background image, thereby improving the visual effect of the video summary.

Optionally, the step of copying each of the first target original images to the corresponding first location in the first summary background image may include:

In the first step, if the target objects in the first target original image overlap, the pixel value of the overlapping portion of the corresponding trajectory is set to be the mean value of the target original image pixel values of each target object, and the pixels of the non-overlapping portion. The value is the pixel value of the target original image of each target object, and the image to be copied is obtained;

In the second step, each of the first target original images that are not overlapped with the target object and the target object are copied to the corresponding first position in the first abstract background image to generate a video summary.

When the video summary is generated, the trajectory of the target object that generates the video summary overlaps and does not overlap. When no overlap occurs, the target original image can be directly copied into the summary background image, and overlap occurs. Then, according to the pixel value of the overlapping portion, the average value of the pixel values of the target original image of each target object is taken as the pixel value of the image of the overlapping portion, and then spliced to generate a video summary. Of course, the overlapping portion can also take the weight value of the pixel value of the target original image of each target object, which is reasonable.

As shown in FIG. 6 , a method for generating a video summary provided by an embodiment of the present application may further include: before the step of acquiring a target search condition, the method for generating a video summary may further include:

S601: Display a user interaction interface according to a user instruction.

The user interaction interface is an interface for realizing interaction between the user and the system, and the user interaction interface may be a dialog box or a selection screen in the webpage. It is used to prompt the user to input a target search, a preset panning condition, and a preset period, wherein the preset period is used to generate a summary background image.

S602. Receive and save a target retrieval condition input by the user through the user interaction interface, a preset translation condition, and a preset period for generating a summary background image.

Applying the embodiment, by searching the established database, the trajectories satisfying the target retrieval condition are obtained, and the overlapping trajectories in the trajectories are combined into a combined trajectory, and then the combined trajectories of different time periods are translated and overlapped without overlapping. Tracking to the same target time segment, and finally splicing the target original image and the abstract background image corresponding to the trajectory in the target time segment to generate a video summary; in the embodiment of the present application, when the video summary is generated, overlapping multiple target objects will occur. The trajectories are combined into a combined trajectory, and the whole trajectory is shifted on the time axis to avoid losing some trajectories in the overlapping trajectories during translation, thereby improving the visual effect of generating a video summary. And through the user interaction interface, the user is allowed to set the target retrieval condition and setting parameters, thereby improving the flexibility of the application and bringing convenience to the user.

Optionally, the method for generating the video summary may further include:

When receiving the startup request input by the user through the user interaction interface, the step of retrieving the established database is performed; and when the interrupt request input by the user through the user interaction interface is received, the process of generating the video summary is ended.

In order to improve the user experience during the generation of the video summary, the user can input an interrupt request at any time. For example, the user finds that the target retrieval condition is set incorrectly, and after receiving the interrupt request, the process of generating the video summary is ended; Then, the user can reset the target retrieval condition according to the requirement, and input a startup request, and after receiving the startup request, re-search the established database according to the target retrieval condition.

Corresponding to the foregoing method embodiment, the embodiment of the present application provides a device for generating a video summary. As shown in FIG. 7, the device for generating a video summary may include:

The retrieval module 710 is configured to acquire a target retrieval condition, and perform a retrieval on the established database to obtain a first trajectory set including a trajectory that meets the target retrieval condition, where the database stores a video frame from the target object. The trajectory information of each trajectory extracted in the trajectory and the target original image, wherein the trajectory information of each trajectory includes: overlapping state information with other trajectories;

The combining module 720 is configured to divide at least two tracks that overlap in the first track set into a group according to the overlapping state information, and each group is determined as a combined track;

a panning module 730, configured to translate, according to a time axis, a trajectory that satisfies a preset panning condition in the trajectory to be translated to the same target time segment, where the trajectory to be translated includes: the combined trajectory and/or the first trajectory set There is no overlapping trajectory in the middle;

a first acquiring module 740, configured to acquire, from the database, a first target original image corresponding to each track in the target time period;

a second obtaining module 750, configured to obtain a first abstract background image for generating a video summary;

The splicing module 760 is configured to splicing the first target original image and the first abstract background image to generate a video summary.

Applying the embodiment, by searching the established database, the trajectories satisfying the target retrieval condition are obtained, and the overlapping trajectories in the trajectories are combined into a combined trajectory, and then the combined trajectories of different time periods are translated and overlapped without overlapping. Tracking to the same target time segment, and finally splicing the target original image and the abstract background image corresponding to the trajectory in the target time segment to generate a video summary; in the embodiment of the present application, when the video summary is generated, overlapping multiple target objects will occur. The trajectories are combined into a combined trajectory, and the whole trajectory is shifted on the time axis to avoid losing some trajectories in the overlapping trajectories during translation, thereby improving the visual effect of generating a video summary.

Optionally, the target search condition may include: retrieving a time period and/or retrieval attribute information of the target object;

When the retrieval condition includes only the retrieval time period, the retrieval module may specifically be used to:

Searching the database according to the retrieval time period to obtain a trajectory of each target object in the retrieval time period;

When the search condition includes only the search attribute information of the target object, the search module may specifically be used to:

Searching the database according to the retrieval attribute information of the target object, and obtaining a trajectory of each target object that matches the retrieval attribute information;

When the retrieval condition includes the retrieval time segment and the retrieval attribute information of the target object, the retrieval module may specifically be used to:

And searching the database according to the retrieval time period and the retrieval attribute information of the retrieval target object, and obtaining a trajectory of each target object that matches the retrieval attribute in the retrieval time period.

Further, based on the inclusion of the retrieval module 710, the combination module 720, the translation module 730, the first acquisition module 740, the second acquisition module 750, and the splicing module 760, as shown in FIG. 8, the embodiment of the present application provides A device for generating a video summary may further include:

a first extraction module 810, configured to extract each target object from the input video;

The second extraction module 820 is configured to extract the trajectory information and the attribute information of each target object, where the trajectory information includes: movement information of the trajectory and overlapping state information with other trajectories;

a first storage module 830, configured to store the trajectory information and the attribute information of each target object into the video structured target description file;

a database generating module 840, configured to generate the database according to the video structured target description file;

a third extraction module 850, configured to extract, from a video frame that includes the target object, an original image and a mask map of each frame corresponding to the trajectory information, and determine the trajectory information according to the original image and the mask map. Corresponding target image of each frame;

The second storage module 860 is configured to store the original image, the mask map, and the target original image into the database.

Applying the embodiment, by searching the established database, the trajectories satisfying the target retrieval condition are obtained, and the overlapping trajectories in the trajectories are combined into a combined trajectory, and then the combined trajectories of different time periods are translated and overlapped without overlapping. Tracking to the same target time segment, and finally splicing the target original image and the abstract background image corresponding to the trajectory in the target time segment to generate a video summary; in the embodiment of the present application, when the video summary is generated, overlapping multiple target objects will occur. The trajectories are combined into a combined trajectory, and the whole trajectory is shifted on the time axis to avoid losing some trajectories in the overlapping trajectories during translation, thereby improving the visual effect of generating a video summary. And by establishing a database, saving the attribute information of the target object, the original image of the target, and the like, and increasing the speed of generating the video summary when generating the video summary; the target original image is obtained by matching the mask image with the original image of the video frame, due to the mask The figure embodies the outline of the target object. The mask map only represents the outline of the target object, and does not contain the image content. After the original image is combined with the original image of the video frame, the image of the target object in the area of the mask map is obtained. It is more accurate to extract the image of the target object directly from the original video frame.

As shown in FIG. 9, the third extraction module 850 may include:

a first extraction submodule 851, configured to extract a motion mask of the target object from a video frame that includes the target object;

a first determining submodule 852, configured to determine an initial mask according to the motion mask;

a second determining submodule 853, configured to determine an edge point set of the initial mask map;

a second extraction sub-module 854, configured to extract a convex set in the set of edge points, and form a convex punctual point set of the mask map;

The filling sub-module 855 is configured to fill the convex hull corresponding to the convex punctual point set to obtain a final mask map.

As shown in FIG. 10, the translation module 730 can include:

a queue establishment sub-module 731, configured to establish a queue to be translated and a summary queue;

a first storage sub-module 732, configured to use a trajectory that is not in the target time period in the target trajectory as a to-be-translated trajectory, and store the trajectory to be translated into a queue to be translated, where the to-be-translated trajectory is the a combined trajectory in the first trajectory set that is not in the target time period and a trajectory that does not overlap;

a second storage submodule 733, configured to store, in the first time set, a track in the target time period into the summary queue;

a third extraction sub-module 734, configured to sequentially extract a current to-be-translated trajectory from the to-be-translated trajectory, and obtain a rectangle of each target object in the video frame corresponding to the current to-be-translated trajectory according to an original image in the database frame;

The operation sub-module 735 is configured to calculate an overlapping area between the rectangular frame of each target object and the rectangular frame of the target object in the video frame corresponding to each track that has been stored in the summary queue;

The third storage sub-module 736 is configured to: when the overlapping area is less than or equal to a preset overlapping parameter threshold, translate the current to-be-translated trajectory to the target time segment, and store the trajectory to the summary queue;

The first obtaining module 740 is specifically configured to:

Obtaining a first target original image corresponding to each track in the summary queue.

Optionally, the track information of each track may further include: a target box information set of the target object;

The splicing module 760 can include:

a third determining submodule, configured to determine, according to the target box information set of the target object in the trajectory information, a first position of the first target original image in the first abstract background image;

The video summary generation sub-module is configured to copy each first target original image to a corresponding first position in the first summary background image to generate a video summary.

Optionally, the third determining submodule is specifically configured to:

If the target objects in the first target original image overlap, the pixel value of the overlapping portion of the corresponding trajectory is set to be the mean value of the target original image pixel value of each target object, and the pixel value of the non-overlapping portion is each target. The pixel value of the target original image of the object, and the image to be copied is obtained;

The splicing module 760 can be specifically configured to:

Copying each of the first target original images that do not overlap each of the to-be-copied image and the target object to a corresponding first position in the first summary background image to generate a video summary.

Optionally, the device may further include:

An operation module, configured to obtain a summary background image according to a preset period;

a third storage module, configured to store the acquired summary background image of each period into the database;

As shown in FIG. 11, the second obtaining module 750 may include:

The dividing sub-module 751 is configured to divide the target time segment into time sub-segments corresponding to the preset period according to the time corresponding to each preset period;

a fourth determining sub-module 752, configured to determine a first preset period corresponding to the time sub-segment including the most track in the target time period;

The background image obtaining sub-module 753 is configured to obtain, from the database, a first summary background image corresponding to the first preset period.

As shown in FIG. 12, the apparatus may further include:

The display module 1210 is configured to display a user interaction interface according to a user instruction;

The receiving module 1220 is configured to receive and save a target retrieval condition input by the user through the user interaction interface, a preset translation condition, and a preset period for generating a summary background image.

Optionally, the device shown may further include:

An execution module, configured to perform the step of searching the established database when receiving a startup request input by the user through the user interaction interface;

The ending module is configured to end the process of generating the video summary when receiving the interrupt request input by the user through the user interaction interface.

It can be understood that the video summary generating apparatus in another embodiment of the embodiment of the present application may include: a retrieval module 710, a combination module 720, a translation module 730, a first acquisition module 740, a second acquisition module 750, and a splicing module. 760, the first extraction module 810, the second extraction module 820, the first storage module 830, the database generation module 840, the third extraction module 850, the second storage module 860, the operation module, the third storage module, the display module 1210, and the receiving Module 1220, an execution module, and an end module.

The embodiment of the present application further provides a computer device, including a processor and a memory, where

a memory for storing a computer program;

The processor, when used to execute the computer program stored in the memory, implements all the steps of the method for generating the video summary provided by the embodiment of the present application.

The above image collector may include an IPC (IP Camera), a smart camera, and the like.

The above memory may include a RAM (Random Access Memory), and may also include NVM (Non-Volatile Memory), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The processor may be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), or the like; or a DSP (Digital Signal Processing) or an ASIC (Application) Specific Integrated Circuit, FPGA (Field-Programmable Gate Array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.

In this embodiment, by reading the computer program stored in the memory and running the computer program, the processor can realize: by searching the established database, obtaining a trajectory that satisfies the target retrieval condition, and the trajectories are generated in the trajectory. The stacked trajectories are combined into a combined trajectory, and then the combined trajectories of different time periods and the trajectories that do not overlap are translated to the same target time period, and finally the target original image and the abstract background image corresponding to the trajectory in the target time segment are spliced and generated. The video summary; in the embodiment of the present application, when the video summary is generated, the trajectories of the overlapping multiple target objects are combined into one combined trajectory, and the whole trajectory is shifted on the time axis to avoid losing some of the overlapping trajectories during translation. Tracks to improve the visuals of generating video summaries.

In addition, corresponding to the method for generating a video summary provided by the foregoing embodiment, the embodiment of the present application provides a storage medium for storing a computer program, and when the computer program is executed by the processor, the method for generating the video summary is implemented. All the steps.

In this embodiment, the storage medium stores an application that executes the method for generating the video digest provided by the embodiment of the present application at runtime, and thus can implement: by searching the established database, obtaining a trajectory that satisfies the target retrieval condition. Combine the trajectories that overlap in these trajectories into combined trajectories, then translate the combined trajectories of different time periods and the trajectories that have not overlapped to the same target time period, and finally the target original images and abstracts corresponding to the trajectories in the target time period. The background image is spliced to generate a video summary. In the embodiment of the present application, when the video summary is generated, the trajectories of the overlapping multiple target objects are combined into one combined trajectory, and the whole time is shifted on the time axis to avoid losing the overlap during translation. Some of the trajectories in the trajectory enhance the visual effect of generating a video summary.

In addition, corresponding to the method for generating the video summary provided by the foregoing embodiment, the embodiment of the present application provides an application program for performing the following steps of the method for generating the video summary provided by the embodiment of the present application.

In this embodiment, the application performs the method for generating the video summary provided by the embodiment of the present application at runtime, so that the trajectory that satisfies the target retrieval condition is obtained by searching the established database, and the trajectory occurs in the trajectory. The overlapping trajectories are combined into a combined trajectory, and then the combined trajectories of different time periods and the trajectories that do not overlap are translated to the same target time period, and finally the target original image and the abstract background image corresponding to the trajectory in the target time segment are spliced. Generating a video summary. When generating a video summary, the embodiment of the present application combines the trajectories of multiple overlapping target objects into a combined trajectory, and performs overall translation on the time axis to avoid losing one of the overlapping trajectories during translation. These tracks improve the visual effect of generating a video summary.

For the computer device, the application program and the storage medium embodiment, since the method content involved is basically similar to the foregoing method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

It should be noted that, in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply such entities or operations. There is any such actual relationship or order between them. Furthermore, the term "comprises" or "comprises" or "comprises" or any other variations thereof is intended to encompass a non-exclusive inclusion, such that a process, method, article, or device that comprises a plurality of elements includes not only those elements but also Other elements, or elements that are inherent to such a process, method, item, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device that comprises the element.

The various embodiments in the present specification are described in a related manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The above is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., which are made within the spirit and principles of the present application, should be included in the present application. Within the scope of protection.

Claims (21)

A method for generating a video summary, the method comprising: Obtaining a target retrieval condition, and retrieving the established database to obtain a first trajectory set including a trajectory that meets the target retrieval condition, wherein the database stores each trajectory extracted from a video frame containing the target object Trajectory information and target original image, wherein the trajectory information of each trajectory includes: overlapping state information with other trajectories; And dividing, according to the overlapping state information, at least two tracks that overlap in the first set of tracks into a group, each group being determined as a combined track; Translating a trajectory of the to-be-translated trajectory that meets the preset translation condition to the same target time period along the time axis, wherein the trajectory to be translated includes: the combined trajectory and/or the overlapping of the first trajectory set Trajectory Acquiring, from the database, a first target original image corresponding to each track in the target time period; Obtaining a first summary background image for generating a video summary; The first target original image is spliced with the first abstract background image to generate a video summary. The method according to claim 1, wherein the target retrieval condition comprises: retrieving a time period and/or retrieval attribute information of the target object; When the retrieval condition includes only the retrieval time period, the searching for the established database includes: Searching the database according to the retrieval time period to obtain a trajectory of each target object in the retrieval time period; When the retrieval condition includes only the retrieval attribute information of the target object, the searching for the established database includes: Searching the database according to the retrieval attribute information of the target object, and obtaining a trajectory of each target object that matches the retrieval attribute information; When the retrieval condition includes the retrieval time period and the retrieval attribute information of the target object, the searching for the established database includes: And searching the database according to the retrieval time period and the retrieval attribute information of the retrieval target object, and obtaining a trajectory of each target object that matches the retrieval attribute in the retrieval time period. The method according to claim 1, wherein before the obtaining the target retrieval condition, the method further comprises: Extract each target object from the input video; Extracting trajectory information and attribute information of each target object, where the trajectory information includes: movement information of the trajectory and overlapping state information with other trajectories; Storing the track information and attribute information of each target object into a video structured target description file; Generating the database according to the video structured target description file; Extracting, from a video frame that includes the target object, an original image and a mask map of each frame corresponding to the trajectory information, and determining, according to the original image and the mask map, a target original of each frame corresponding to the trajectory information. Figure The original image, the mask map, and the target original image are stored in the database. The method according to claim 3, wherein the extracting a mask map of each frame corresponding to the trajectory information from the video frame including the target object comprises: Extracting a motion mask of the target object from a video frame containing the target object; Determining an initial mask map according to the motion mask; Determining an edge point set of the initial mask map; Extracting a convex set in the set of edge points to form a convex set of points of the mask map; Filling the convex hull corresponding to the convex hull point set to obtain a final mask map. The method according to claim 3, wherein the shifting the trajectory of the to-be-translated trajectory that satisfies the preset translation condition to the same target time period along the time axis comprises: Establish a queue to be translated and a summary queue; And a track that is not in the target time segment in the first track set is used as a to-be-translated track, and is stored in a queue to be translated, wherein the to-be-translated track is not in the first track set. a combined trajectory in the target time period and a trajectory in which no overlap occurs; Storing the trajectory in the target time period in the first trajectory set into the summary queue; Extracting the current to-be-translated trajectory from the to-be-translated queue, and obtaining a rectangular frame of each target object in the video frame corresponding to the current to-be-translated trajectory according to the original image in the database; Calculating an overlapping area between a rectangular frame of each target object and a rectangular frame of the target object in the video frame corresponding to each track that has been stored in the summary queue; When the overlapping area is less than or equal to a preset overlapping parameter threshold, the current to-be-translated trajectory is translated to the target time period, and stored in the summary queue; And the step of acquiring the first target original image corresponding to each track of the target time period from the database, including: Obtaining a first target original image corresponding to each track in the summary queue. The method according to claim 3, wherein the track information of each track further comprises: a target frame information set of the target object; The splicing the first target original image and the first abstract background image to generate a video summary, including: Determining, according to the target frame information set of the target object in the trajectory information, a first position of the first target original image in the first abstract background image; Copying each first target original image to a corresponding first position in the first abstract background image to generate a video summary. The method according to claim 6, wherein the copying each of the first target original images to the corresponding first position in the first abstract background image comprises: If the target objects in the first target original image overlap, the pixel value of the overlapping portion of the corresponding trajectory is set to be the mean value of the target original image pixel value of each target object, and the pixel value of the non-overlapping portion is each target. The pixel value of the target original image of the object, and the image to be copied is obtained; Copying each first target original image to a corresponding first location in the first summary background image to generate a video summary, including: Copying each of the first target original images that do not overlap each of the to-be-copied image and the target object to a corresponding first position in the first summary background image to generate a video summary. The method according to claim 3, wherein before the storing the original image, the mask map and the target original image in the database, the method further comprises: Obtain a summary background image according to a preset period; Storing a summary background image of each cycle acquired into the database; The obtaining a first abstract background image for generating a video summary includes: Dividing the target time period into time sub-segments corresponding to preset periods according to time corresponding to each preset period; Determining, in the target time period, a first preset period corresponding to a time sub-segment containing the most track; Obtaining, from the database, a first summary background image corresponding to the first preset period. The method according to claim 8, wherein before the obtaining the target retrieval condition, the method further comprises: Display the user interaction interface according to user instructions; Receiving and saving a target retrieval condition input by the user through the user interaction interface, a preset translation condition, and a preset period for generating a summary background image; The method further includes: Performing the step of retrieving the established database when receiving a startup request input by the user through the user interaction interface; When the interrupt request input by the user through the user interaction interface is received, the process of generating the video summary is ended. A device for generating a video summary, the device comprising: a retrieval module, configured to acquire a target retrieval condition, and retrieve the established database to obtain a first trajectory set including a trajectory that meets the target retrieval condition, wherein the database stores the video frame from the target object Extracting the trajectory information of each trajectory and the target original image, wherein the trajectory information of each trajectory includes: overlapping state information with other trajectories; a combination module, configured to divide at least two tracks that overlap in the first set of tracks into a group according to the overlapping state information, and each group is determined as a combined track; a panning module for translating a trajectory of the to-be-translated trajectory that satisfies the preset panning condition to the same target time segment along the time axis, wherein the to-be-translated trajectory comprises: the combined trajectory and/or the first trajectory set No overlapping trajectories occur; a first acquiring module, configured to acquire, from the database, a first target original image corresponding to each track in the target time period; a second obtaining module, configured to obtain a first abstract background image for generating a video summary; a splicing module, configured to splicing each of the first target original images and the first abstract background image to generate a video summary. The apparatus according to claim 10, wherein said target retrieval condition comprises: retrieving a time period and/or retrieval attribute information of the target object; When the retrieval condition includes only the retrieval time period, the retrieval module is specifically configured to: Searching the database according to the retrieval time period to obtain a trajectory of each target object in the retrieval time period; When the search condition includes only the search attribute information of the target object, the search module is specifically configured to: Searching the database according to the retrieval attribute information of the target object, and obtaining a trajectory of each target object that matches the retrieval attribute information; When the retrieval condition includes the retrieval time period and the retrieval attribute information of the target object, the retrieval module is specifically configured to: And searching the database according to the retrieval time period and the retrieval attribute information of the retrieval target object, and obtaining a trajectory of each target object that matches the retrieval attribute in the retrieval time period. The device according to claim 10, wherein the device further comprises: a first extraction module, configured to extract each target object from the input video; a second extraction module, configured to extract trajectory information and attribute information of each target object, where the trajectory information includes: movement information of the trajectory, and overlapping state information with other trajectories; a first storage module, configured to store track information and attribute information of each target object into a video structured target description file; a database generating module, configured to generate the database according to the video structured target description file; a third extraction module, configured to extract an original image and a mask map of each frame corresponding to the trajectory information from the video frame that includes the target object, and determine, according to the original image and the mask map, the trajectory information corresponding to the trajectory information The original image of each frame of the frame; And a second storage module, configured to store the original image, the mask map, and the target original image into the database. The apparatus according to claim 12, wherein the third extraction module comprises: a first extraction submodule, configured to extract a motion mask of the target object from a video frame that includes the target object; a first determining submodule, configured to determine an initial mask according to the motion mask; a second determining submodule, configured to determine an edge point set of the initial mask map; a second extraction submodule, configured to extract a convex set in the set of edge points, and form a convex punctual point set of the mask map; The filler submodule is configured to fill the convex hull corresponding to the convex hull point set to obtain a final mask map. The device of claim 10, wherein the translation module comprises: a queue creation sub-module for establishing a queue to be translated and a summary queue; a first storage sub-module, configured to use a trajectory in the target trajectory that is not in the target time segment as a trajectory to be translated, and store the trajectory in the to-be-translated trajectory, where the trajectory to be translated is the a combined trajectory in a set of trajectories that is not in the target time period and a trajectory that does not overlap; a second storage submodule, configured to store, in the first time set, a track in the target time period into the summary queue; a third extraction sub-module, configured to sequentially extract a current to-be-translated trajectory from the to-be-translated queue, and obtain a rectangular frame of each target object in the video frame corresponding to the current to-be-translated trajectory according to the original image in the database ; An operation submodule, configured to calculate an overlapping area between a rectangular frame of each target object and a rectangular frame of the target object in a video frame corresponding to each track that has been stored in the summary queue; a third storage submodule, configured to: when the overlapping area is less than or equal to a preset overlapping parameter threshold, translate the current to-be-translated trajectory to the target time segment, and store the trajectory to the summary queue; The first acquiring module is specifically configured to: Obtaining a first target original image corresponding to each track in the summary queue. The apparatus according to claim 12, wherein the track information of each track further comprises: a target frame information set of the target object; The splicing module includes: a third determining submodule, configured to determine, according to the target box information set of the target object in the trajectory information, a first position of the first target original image in the first abstract background image; The video summary generation sub-module is configured to copy each first target original image to a corresponding first position in the first summary background image to generate a video summary. The device according to claim 15, wherein the third determining submodule is specifically configured to: If the target objects in the first target original image overlap, the pixel value of the overlapping portion of the corresponding trajectory is set to be the mean value of the target original image pixel value of each target object, and the pixel value of the non-overlapping portion is each target. The pixel value of the target original image of the object, and the image to be copied is obtained; The splicing module is specifically configured to: Copying each of the first target original images that do not overlap each of the to-be-copied image and the target object to a corresponding first position in the first summary background image to generate a video summary. The device according to claim 10, wherein the device further comprises: An operation module, configured to obtain a summary background image according to a preset period; a third storage module, configured to store the acquired summary background image of each period into the database; The second acquiring module includes: a sub-module, configured to divide the target time segment into a time sub-segment corresponding to the preset period according to a time corresponding to each preset period; a fourth determining sub-module, configured to determine a first preset period corresponding to the time sub-segment including the most trajectory in the target time period; The background image obtaining sub-module is configured to obtain, from the database, a first abstract background image corresponding to the first preset period. The device according to claim 10, wherein the device further comprises: a display module, configured to display a user interaction interface according to a user instruction; a receiving module, configured to receive and save a target retrieval condition input by the user through the user interaction interface, a preset translation condition, and a preset period for generating a summary background image; An execution module, configured to perform the step of searching the established database when receiving a startup request input by the user through the user interaction interface; The ending module is configured to end the process of generating the video summary when receiving the interrupt request input by the user through the user interaction interface. A computer device, comprising: a processor and a memory, wherein The memory is configured to store a computer program; The processor, when executed to execute a computer program stored on the memory, implements the method steps of any of claims 1-9. A storage medium, characterized by storing executable code for execution at runtime: the method steps of any of claims 1-9. An application, characterized in that it is executed at runtime: the method steps of any of claims 1-9.

Images