CN116935275A - Scene segmentation method and device based on area of interest - Google Patents

Abstract

The application discloses a scene segmentation method and device based on a region of interest, wherein the method comprises the following steps: pre-analyzing the video to be segmented, determining an interested region of the video to be segmented, and extracting first information of the interested region; performing scene pre-segmentation treatment on the video to be segmented to obtain at least one pre-selected transition frame; detecting the region of interest of the pre-selected transition frame to obtain second information of the region of interest; and matching according to the first information and the second information to determine a transition frame meeting a preset transition condition for scene segmentation. After the scene is pre-segmented to obtain a pre-selected transition frame, the pre-selected transition frame can be subjected to region-of-interest detection, and the second information of the detected region-of-interest and the first information of the region-of-interest determined by pre-analysis are used for judging, so that false detection can be avoided, and accurate scene segmentation is achieved.

Description

Scene segmentation method and device based on area of interest

Technical field

The present application relates to the field of image processing technology, and specifically to a scene segmentation method and device based on regions of interest.

Background technique

The switching of scene segments in a video is called a transition, which includes hard transitions and soft transitions (such as fade in and out, dissolve, sweep, dissolve, etc.). Scene segmentation of videos is to segment a complete video into video processing units with complete semantics - shots (shots) based on transitions. Scene segmentation is a necessary condition for many video post-processing processes, and is also commonly used in automated video annotation, content-based video retrieval, etc. The development of video platforms has also led to an upsurge in users creating videos. However, video creation usually requires users to segment the required segments in the video into scenes, and then combine the video segments according to their own understanding to create new video content. Scene segmentation of video clips is very time-consuming. Automated scene segmentation can help users reduce creation costs.

Most of the existing automatic scene segmentation is based on the changes in image characteristics between adjacent frames of different scene segments in video frames, such as the huge differences in color, brightness, statistical characteristics, etc. For example, when performing scene segmentation based on the correlation between consecutive frames, the similarity between the previous and next frames is calculated, such as using image similarity calculation. Specifically, the image grayscale histogram can be used to calculate the grayscale histogram similarity. This determines the similarity between the previous and next frames, and performs scene segmentation based on the similarity between the previous and next frames. For some video clips with rapidly changing backgrounds, they are often mistakenly divided into different scenes, as shown in Figure 1a and Figure 1b. Figure 1a and Figure 1b show the before and after frames of playing basketball. When the camera moves quickly, the similarity between the front and back frames Contrast will be considered different scenes, resulting in scene segmentation errors. Alternatively, an algorithm based on edge contour detection determines whether to segment the scene based on the difference in object edges and contours corresponding to the previous and subsequent frames. Although it can effectively distinguish the movement of video objects and sudden changes in the scene, and detect blending or fading, but for Video shots with rapid background changes or camera tracking during a game can easily be misdetected, leading to incorrect scene segmentation.

Contents of the invention

In view of the above problems, embodiments of the present application are proposed to provide a scene segmentation method and device based on regions of interest that overcome the above problems or at least partially solve the above problems.

According to the first aspect of the embodiment of the present application, a scene segmentation method based on regions of interest is provided, which includes:

Pre-analyze the video to be segmented, determine the area of interest of the video to be segmented, and extract the first information of the area of interest;

Perform scene pre-segmentation processing on the video to be segmented to obtain at least one pre-selected transition frame;

Perform area of interest detection on the preselected transition frame to obtain second information of the area of interest;

Matching is performed based on the first information and the second information to determine transition frames that meet the preset transition conditions to perform scene segmentation.

Optionally, pre-analyzing the video to be segmented, determining the region of interest of the video to be segmented, and extracting the first information of the region of interest further includes:

Pre-analyze the video to be segmented, determine the area of interest of the video frame in the video to be segmented, and extract the first total number and coordinate information of the area of interest in the video frame; the coordinate information includes the angular coordinate information of the area of interest , height information and width information.

Optionally, the objects corresponding to the regions of interest are the same or different;

Extracting the first total number of regions of interest in the video frame is specifically: counting the regions of interest of different objects in the video frame to obtain the first total number of regions of interest.

Optionally, pre-analyzing the video to be segmented and determining the area of interest of the video frame in the video to be segmented further includes:

Pre-analyze the video to be segmented and determine the area of interest of one or more video frames in the video to be segmented.

Optionally, methods also include:

The detection template image corresponding to the area of interest is determined based on the coordinate information of the area of interest.

Optionally, performing area of interest detection on the preselected transition frame to obtain the second information of the area of interest further includes:

Perform area-of-interest detection on the pre-selected transition frame based on the detection template image to obtain the area of interest included in the pre-selected transition frame, and obtain a second total number of areas of interest included in the pre-selected transition frame.

Optionally, perform area of interest detection on the preselected transition frame according to the detection template image, obtain the area of interest included in the preselected transition frame, and obtain the second total number of areas of interest included in the preselected transition frame by statistics, further including: :

Perform area of interest detection on the preselected transition frame based on the detection template image to obtain multiple areas of interest contained in the preselected transition frame;

Count the areas of interest of each different object in the preselected transition frame to obtain the second total number of areas of interest; where, if multiple areas of interest are the same object, the number of recorded multiple areas of interest is 1;

The second total number of each region of interest contained in the preselected transition frame is obtained by statistics.

Optionally, matching according to the first information and the second information to determine the transition frame that meets the preset transition conditions for scene segmentation further includes:

Calculate the ratio of the second total number to the first total number based on the first total number of the first information and the second total number of the second information;

Determine whether the ratio exceeds the preset first transition threshold;

If so, it is determined that the preselected transition frame does not meet the preset transition conditions.

Optionally, pre-analyzing the video to be segmented, determining the region of interest of the video to be segmented, and extracting the first information of the region of interest further includes:

Pre-analyze the video to be segmented, determine the area of interest of the video to be segmented, and extract the first total number of areas of interest, coordinate information, and weight values of objects in each area of interest;

Matching the first information and the second information to determine the transition frame that meets the preset transition conditions for scene segmentation further includes:

According to the first information, calculate the sum of the weight values of the objects in each area of interest to obtain the first weight value;

According to the second information, determine the area of interest contained in the preselected transition frame, and accumulate the weight values of the objects in the corresponding area of interest to obtain the second weight value;

Calculate the ratio of the second weight value to the first weight value, and determine whether the ratio exceeds the preset second transition threshold;

If so, it is determined that the preselected transition frame does not meet the preset transition conditions.

Optionally, performing scene pre-segmentation processing on the video to be segmented to obtain at least one preselected transition frame further includes:

Based on scene detection, perform scene pre-segmentation processing on the video to be segmented to obtain a preselected transition frame number set containing at least one preselected transition frame number; wherein the preselected transition frame number corresponds to the preselected transition frame one-to-one.

Optionally, perform area of interest detection on the preselected transition frame to obtain second information of the area of interest, and perform matching based on the first information and the second information to determine the transition frame that meets the preset transition conditions to perform the scene Segments further include:

Traverse the set of preselected transition frame numbers, perform area of interest detection on the preselected transition frame corresponding to any preselected transition frame number, and obtain the second information of the area of interest of the preselected transition frame; according to the first information and the second information Match and determine whether the preselected transition frame is a transition frame that meets the preset transition conditions; if not, delete the preselected transition frame number; obtain the next preselected transition frame number of the preselected transition frame number set, and continue traversing, Until the traversal completes the preselected transition frame number set, the scene is segmented according to the remaining preselected transition frame numbers in the preselected transition frame number set.

According to the second aspect of the embodiment of the present application, a scene segmentation device based on a region of interest is provided, which includes:

The region of interest module is suitable for pre-analyzing the video to be segmented, determining the region of interest of the video to be segmented, and extracting the first information of the region of interest;

The pre-segmentation module is suitable for performing scene pre-segmentation processing on the video to be segmented to obtain at least one pre-selected transition frame;

The detection module is adapted to detect the area of interest on the pre-selected transition frame and obtain the second information of the area of interest;

The matching module is adapted to perform matching according to the first information and the second information to determine transition frames that meet the preset transition conditions for scene segmentation.

According to a third aspect of the embodiment of the present application, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface complete each other through the communication bus. communication between;

The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the above-mentioned region-of-interest-based scene segmentation method.

According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided. At least one executable instruction is stored in the storage medium. The executable instruction causes the processor to perform the above-mentioned scene segmentation based on the region of interest. The operation corresponding to the method.

According to the scene segmentation method and device based on the area of interest provided in this application, after pre-segmenting the scene to obtain the pre-selected transition frame, the area of interest can be detected on the pre-selected transition frame, and the third area of the detected area of interest can be used. The second information is judged with the first information of the area of interest determined by pre-analysis, so that false detection can be avoided and accurate scene segmentation can be achieved.

The above description is only an overview of the technical solutions of the present application. In order to have a clearer understanding of the technical means of the present application, they can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present application more obvious and understandable. , the specific implementation methods of the present application are specifically listed below.

Description of the drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be construed as limiting the application. Also throughout the drawings, the same reference characters are used to designate the same components. In the attached picture:

Figure 1a shows a schematic diagram of the previous frame of the basketball video;

Figure 1b shows a schematic diagram of the last frame of the basketball video;

Figure 2 shows a flow chart of a scene segmentation method based on regions of interest according to an embodiment of the present application;

Figure 3 shows a flow chart of a scene segmentation method based on regions of interest according to another embodiment of the present application;

Figure 4 shows a schematic structural diagram of a scene segmentation device based on regions of interest according to an embodiment of the present application;

Figure 5 shows a schematic structural diagram of a computing device according to an embodiment of the present application.

Detailed ways

Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a thorough understanding of the present application, and to fully convey the scope of the present application to those skilled in the art.

First, the terminology involved in one or more embodiments of this application is explained.

Transition: a transition or transition from one video scene to another;

Scene segmentation: Split the video into independent video clips based on scene or scene changes;

Image similarity calculation: mainly measures the similarity of content between two images, and judges the similarity of images based on the calculated score. There are a variety of measurement algorithms that can determine the similarity of images, such as CORR (CorrelationCoefficient) correlation;

Image grayscale histogram: The distribution pattern of grayscale in the image, intuitively showing the proportion of each grayscale in the image, reflecting the brightness and contrast information of the image;

Grayscale histogram similarity calculation: convert the image into a grayscale image, count its histograms, and perform similarity calculation on the histograms of the grayscale images of the two images. The existing histogram similarity measurement methods mainly include the distance measurement method. , such as Manhattan distance, Euclidean distance, Hausdorff distance, central moment method, X2 statistical distance, etc.

Template matching: Pattern recognition method, studying where the pattern of a specific object is located in the image, and then identifying the object.

Figure 2 shows a flow chart of a scene segmentation method based on regions of interest according to an embodiment of the present application. As shown in Figure 2, the method includes the following steps:

Step S201: Pre-analyze the video to be segmented, determine the region of interest of the video to be segmented, and extract first information of the region of interest.

For videos to be segmented, when the existing technology performs scene segmentation based on changes in image features between adjacent frames, when the camera moves quickly and the background changes rapidly, scene segmentation errors are likely to occur. This embodiment is based on the area of interest. When the scene is segmented, by detecting the area of interest on the transition frame, it can be determined whether it is a transition. This avoids the problem of rapid changes in the background while the characters, items, etc. in the foreground do not change. If there are scene segmentation errors, remove the preselected transition frame numbers that do not meet the preset transition conditions to ensure accurate scene segmentation.

Specifically, the video to be segmented is pre-analyzed to determine the roi (region of interest, region of interest) contained in it. For example, by performing image analysis on the video frames and determining the area of interest based on the implementation situation, or an interface can be provided to facilitate the user to select the area of interest in the video frame, etc., which are not limited here.

After the area of interest is determined, information can be extracted for each area of interest to obtain the first information of the area of interest. The first information may include, for example, the total number of regions of interest in the video frame, coordinate information of each region of interest, etc. The coordinate information can determine the position of the area of interest in the video frame, making it easy to extract objects in the area of interest, etc.

Step S202: Perform scene pre-segmentation processing on the video to be segmented to obtain at least one preselected transition frame.

For the video to be segmented, scene pre-segmentation processing can be performed on it. The scene pre-segmentation processing here is not the final scene segmentation. You can first perform scene pre-segmentation processing on the video to be segmented to obtain the following results: At least one preselected transition frame in sequence.

Furthermore, the execution order of step S202 and step S201 is not limited, and any step can be executed first according to the implementation situation.

Step S203: Perform area of interest detection on the preselected transition frame to obtain second information of the area of interest.

For the preselected transition frame, area of interest detection is first performed, and the corresponding area of interest in the preselected transition frame is detected. The region of interest detection can be performed by, for example, template matching, using the region of interest determined by pre-analysis as a template, performing template matching with the preselected transition frame, and detecting the region of interest contained in the preselected transition frame.

Based on the region of interest contained in the preselected transition frame, second information of the region of interest can be obtained. The second information may include, for example, each area of interest, statistics of the total number of each area of interest, etc.

Step S204: Match the first information and the second information to determine the transition frame that meets the preset transition conditions to perform scene segmentation.

By matching the first information and the second information, the preselected transition frame can be judged, thereby determining whether the preselected transition frame is a transition frame that meets the preset transition conditions, so that based on the transition that meets the preset transition conditions Frames are used to segment scenes. Specifically, for example, the ratio of the total number of regions of interest in the first information to the total number of regions of interest in the second information is calculated. Based on the ratio, it can be determined whether the preselected transition frame also contains multiple regions of interest. If If it contains more regions of interest, it means that the video scene has no transition, that is, the preselected transition frame does not meet the preset transition conditions. If it contains few or no regions of interest, it means that the video scene has transitioned, and the preselected transition frame It is a transition frame that meets the preset transition conditions, and the scene can be divided based on it.

According to the scene segmentation method based on the region of interest provided by this application, after pre-segmenting the scene to obtain the pre-selected transition frame, the region of interest can be detected on the pre-selected transition frame, and the second information of the detected region of interest is combined with the scene segmentation method. The first information of the area of interest determined by pre-analysis is used for judgment, so that false detection can be avoided and accurate scene segmentation can be achieved.

Figure 3 shows a flow chart of a scene segmentation method based on regions of interest according to an embodiment of the present application. As shown in Figure 3, the method includes the following steps:

Step S301: Pre-analyze the video to be segmented, determine the region of interest of one or more video frames in the video to be segmented, and extract first information of the region of interest.

When pre-analyzing the video to be segmented, you can select a video frame from the video to be segmented, such as the first video frame, to determine the area of interest in the video frame; or select multiple video frames, multiple video frames The area of interest of multiple video frames can be determined for video frames corresponding to different scenes, etc., which is not limited here. Pre-analysis can be carried out through image processing, plot analysis, etc. For example, the current video is determined to be playing basketball, and the area of interest is determined to be "basketball" through image analysis. An interface can also be provided for the user to select the area of interest, such as the user Enter "basketball" and analyze and determine the area of interest corresponding to "basketball" from the video frame.

After determining the region of interest of the video frame in the video to be segmented, extract the first total number and coordinate information of the region of interest in the video frame. The first total number is the total number of each region of interest contained in the video frame. Furthermore, the objects corresponding to the areas of interest may be the same object, for example, the objects in multiple areas of interest are all "gold coins", or the objects in the multiple areas of interest are different, including object 1, object 2, object 3... . When counting the first total number, the regions of interest of different objects in the video frame are counted to obtain the first total number of regions of interest. The coordinate information is used to determine the area of interest, including the angular coordinate information, height information and width information of the area of interest. The angular coordinate information can be any angular coordinate information of the area of interest, such as upper left corner coordinate information, etc. The coordinate information is such as (xi, yi, wi, hi), where (xi, yi) corresponds to the coordinate information of the upper left corner of the area of interest, wi is the width information of the area of interest, hi is the height information of the area of interest, and i is The sorting number of the region of interest in the video frame.

Further, after pre-analyzing the video to be segmented to determine the area of interest of the video to be segmented, in addition to extracting the first total number and coordinate information of the area of interest, the weight value of the objects in each area of interest can also be determined. , different weight values can be set for different objects, such as the "basketball" object weight value 0.6, the "basket" object weight value 0.3, etc. Through different weight values, the importance of objects in the area of interest can be identified, and more accurately distinguish whether Transitions. The above are examples and are not limited here.

Step S302: Based on scene detection, perform scene pre-segmentation processing on the video to be segmented to obtain a set of preselected transition frame numbers including multiple preselected transition frame numbers.

Based on scene detection, such as the scene detection algorithm in the x264 encoder, by calculating a metric value for each video frame to estimate the degree of difference from the previous frame, the video to be segmented is subjected to scene pre-segmentation processing to obtain multiple A collection of preselected transition frame numbers. The preselected transition frame numbers are sorted in order, and each preselected transition frame number corresponds to a preselected transition frame.

The execution order of step S301 and step S302 is not limited, and any step can be executed first according to the implementation situation.

Step S303: Determine the detection template image corresponding to the area of interest based on the coordinate information of the area of interest.

According to the coordinate information of the region of interest extracted in step S301, the coordinate position corresponding to the video frame can be located, and a detection template image corresponding to the region of interest can be obtained, which facilitates subsequent use of the detection template image corresponding to the region of interest to detect the region of interest.

Step S304: Traverse the set of preselected transition frame numbers, perform area of interest detection on the preselected transition frame corresponding to any preselected transition frame number, and obtain second information of the area of interest of the preselected transition frame.

There may be non-transitions in each preselected transition frame number included in the preselected transition frame number set. It is also necessary to traverse the preselected transition frame number set and further determine the preselected transition frame corresponding to each preselected transition frame number. .

When traversing, traverse in order, first obtain a preselected transition frame number, and perform area of interest detection on its corresponding preselected transition frame. For example, based on the detection template image, perform area of interest detection on the preselected transition frame through the template. Matching: Find whether there is the same or similar detection template image in the preselected transition frame, determine its corresponding position, and obtain the area of interest contained in the preselected transition frame.

By counting the regions of interest included in the preselected transition frames, the second total number of regions of interest included in the preselected transition frames can be obtained. The second total number is the total number of regions of interest included in the preselected transition frame. The objects in the multiple regions of interest contained in the preselected transition frame are the same or different, and the regions of interest of the different objects in the preselected transition frame are counted to obtain the second total number of regions of interest. Wherein, if multiple regions of interest are the same object, the number of multiple regions of interest recorded is 1, and finally the second total number of regions of interest included in the transition frame is obtained.

Step S305, based on the first information and the second information, determine whether the preselected transition frame meets the preset transition conditions; if not, delete the preselected transition frame number; obtain the next preselected transition frame number of the preselected transition frame number set , continue traversing until the traversal completes the preselected transition frame number set, so as to perform scene segmentation based on the remaining preselected transition frame numbers in the preselected transition frame number set.

After the region of interest is detected on the preselected transition frame, it is determined whether the preselected transition frame meets the preset transition conditions based on the first information and the obtained second information. Specifically, the ratio of the second total number to the first total number can be calculated based on the first total number of first information and the second total number of second information. If the first total number is 5 and the second total number is 3, the ratio is 3/5=0.6. Determine whether the ratio exceeds the preset first transition threshold. The preset first transition threshold can be set according to the implementation situation, such as 0.5. If the ratio exceeds the preset first transition threshold, that is, the preselected transition frame also contains more senses. The area of interest is judged not to meet the preset transition conditions. If there are few or no regions of interest, it means that the video scene has transitioned, and the preselected transition frame is a transition frame that meets the preset transition conditions, and there is no need to process the preselected transition frame number. Or, according to the first information, based on the weight values of the objects in each area of interest, the sum of the weight values of the objects in each area of interest is calculated, that is, the first weight value is obtained. According to the second information, each area of interest included in the preselected transition frame is determined, and the weight values of the objects in the corresponding area of interest are accumulated to obtain a second weight value. Calculate the ratio of the second weight value to the first weight value, and determine whether the ratio exceeds the preset second transition threshold. If so, it means that the area of interest included in the preselected transition frame is a more important area of interest, and the preselected transition If the frame still belongs to the video frame of the current scene, it is judged that it does not meet the preset transition conditions. If the ratio does not exceed the preset second transition threshold, it means that the video scene has transitioned, and the preselected transition frame is a transition frame that meets the preset transition conditions, and there is no need to process the preselected transition frame number. The above is an example. The specific settings are based on the implementation situation and are not limited here.

When it is determined that the preselected transition frame does not meet the preset transition conditions, the corresponding preselected transition frame number will be deleted, and subsequent scene segmentation will not be performed based on the preselected transition frame number. Continue to obtain the next preselected transition frame number of the preselected transition frame number set, and execute steps S304-S305 based on the preselected transition frame corresponding to the next preselected transition frame number until the traversal of the preselected transition frame number set is completed, that is, for all After the preselected transition frame is judged, the remaining preselected transition frame numbers in the preselected transition frame number set are the accurate transition frame numbers. Scene switching can be performed based on the remaining preselected transition frame numbers in the preselected transition frame number set. points to obtain accurate scene segmentation results.

According to the scene segmentation method based on the region of interest provided by this application, scene pre-segmentation is performed on the video to be segmented, and a set of preselected transition frame numbers of multiple preselected transition frame numbers can be obtained. Pre-analyze the video to be segmented, determine the area of interest of one or more video frames in the video to be segmented, and extract the first information of the area of interest; determine the detection template object based on the area of interest, and pre-select the transition frame The preselected transition frame corresponding to the number is detected for the area of interest to obtain the second information of the area of interest. According to the first total number of the first information and the second total number of the second information, the ratio of the second total number to the first total number is calculated. If it is determined that the ratio exceeds the preset first transition threshold, it means The preselected transition frame also contains many areas of interest, which do not meet the preset transition conditions, or the first weight value is obtained based on the sum of the weight values of the objects in each area of interest in the first information, and the first weight value is obtained according to the second information, determine each area of interest contained in the preselected transition frame, and accumulate the weight values of the objects in the corresponding area of interest to obtain a second weight value. Calculate the ratio of the second weight value to the first weight value. If it is determined that the ratio exceeds the preset second transition threshold, the preset transition condition is not met. For preselected transition frames that do not meet the preset transition conditions, delete the preselected transition frame number, obtain the next preselected transition frame number of the preselected transition frame number set, and continue traversing until the preselected transition frame number set is completed. The remaining pre-selected transition frame numbers in the transition frame number set are used for scene segmentation to ensure accurate scene segmentation.

Figure 4 shows a schematic structural diagram of a scene segmentation device based on regions of interest provided by an embodiment of the present application. As shown in Figure 4, the device includes:

The region of interest module 410 is adapted to pre-analyze the video to be segmented, determine the region of interest of the video to be segmented, and extract the first information of the region of interest;

The pre-segmentation module 420 is adapted to perform scene pre-segmentation processing on the video to be segmented to obtain at least one preselected transition frame;

The detection module 430 is adapted to detect the region of interest on the preselected transition frame to obtain the second information of the region of interest;

The matching module 440 is adapted to perform matching according to the first information and the second information to determine transition frames that meet the preset transition conditions for scene segmentation.

Optionally, the region of interest module 410 is further adapted to:

Pre-analyze the video to be segmented, determine the area of interest of the video frame in the video to be segmented, and extract the first total number and coordinate information of the area of interest in the video frame; the coordinate information includes the angular coordinate information of the area of interest , height information and width information.

Optionally, the objects corresponding to the regions of interest are the same or different;

The region of interest module 410 is further adapted to: count the regions of interest of different objects in the video frame to obtain a first total number of regions of interest.

Optionally, the region of interest module 410 is further adapted to:

Pre-analyze the video to be segmented and determine the area of interest of one or more video frames in the video to be segmented.

Optionally, the device further includes: a template image module 450, adapted to determine a detection template image corresponding to the region of interest based on the coordinate information of the region of interest.

Optionally, the detection module 430 is further adapted to:

Perform area-of-interest detection on the pre-selected transition frame based on the detection template image to obtain the area of interest included in the pre-selected transition frame, and obtain a second total number of areas of interest included in the pre-selected transition frame.

Optionally, the detection module 430 is further adapted to:

Perform area of interest detection on the preselected transition frame based on the detection template image to obtain multiple areas of interest contained in the preselected transition frame;

Count the areas of interest of each different object in the preselected transition frame to obtain the second total number of areas of interest; where, if multiple areas of interest are the same object, the number of recorded multiple areas of interest is 1;

The second total number of each region of interest contained in the preselected transition frame is obtained by statistics.

Optionally, the matching module 440 is further adapted to:

Calculate the ratio of the second total number to the first total number based on the first total number of the first information and the second total number of the second information;

Determine whether the ratio exceeds the preset first transition threshold;

If so, it is determined that the preselected transition frame does not meet the preset transition conditions.

Optionally, the region of interest module 410 is further adapted to:

Pre-analyze the video to be segmented, determine the area of interest of the video to be segmented, and extract the first total number of areas of interest, coordinate information, and weight values of objects in each area of interest;

Matching module 440 is further adapted to:

According to the first information, calculate the sum of the weight values of the objects in each area of interest to obtain the first weight value;

According to the second information, determine the area of interest contained in the preselected transition frame, and accumulate the weight values of the objects in the corresponding area of interest to obtain the second weight value;

Calculate the ratio of the second weight value to the first weight value, and determine whether the ratio exceeds the preset second transition threshold;

If so, it is determined that the preselected transition frame does not meet the preset transition conditions.

Optionally, the pre-slicing module 420 is further adapted to:

Based on scene detection, scene pre-segmentation is performed on the video to be segmented to obtain a set of preselected transition frame numbers containing multiple preselected transition frame numbers; among which, the preselected transition frame numbers correspond to the preselected transition frames one-to-one.

Optionally, the detection module 430 and the matching module 440 are further adapted to:

Traverse the set of preselected transition frame numbers, perform area of interest detection on the preselected transition frame corresponding to any preselected transition frame number, and obtain the second information of the area of interest of the preselected transition frame; according to the first Match the first information with the second information to determine whether the preselected transition frame is a transition frame that meets the preset transition conditions; if not, delete the preselected transition frame number; obtain the preselected transition frame Continue traversing the next preselected transition frame number in the set of preselected transition frame numbers until the traversal completes the set of preselected transition frame numbers, so as to perform scene segmentation based on the remaining preselected transition frame numbers in the set of preselected transition frame numbers.

For the description of each module above, refer to the corresponding description in the method embodiment, and will not be described again here.

According to the scene segmentation device based on the area of interest provided by this application, after pre-segmenting the scene to obtain a set of preselected transition frame numbers of multiple preselected transition frame numbers, the transition frames corresponding to the preselected transition frame numbers can be processed. Region of interest detection uses the second information of the detected region of interest and the first information of the region of interest determined by pre-analysis to make a judgment, thereby avoiding false detection and achieving accurate scene segmentation.

This application also provides a non-volatile computer storage medium. The computer storage medium stores at least one executable instruction. The executable instruction can execute the scene segmentation method based on the region of interest in any of the above method embodiments.

Figure 5 shows a schematic structural diagram of a computing device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the computing device.

As shown in FIG. 5 , the computing device may include: a processor 502 , a communications interface 504 , a memory 506 , and a communications bus 508 .

in:

The processor 502, the communication interface 504, and the memory 506 complete communication with each other through the communication bus 508.

The communication interface 504 is used to communicate with network elements of other devices such as clients or other servers.

The processor 502 is configured to execute the program 510. Specifically, it may execute the relevant steps in the above-mentioned scene segmentation method embodiment based on the region of interest.

Specifically, program 510 may include program code including computer operating instructions.

The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the present application. The one or more processors included in the computing device may be the same type of processor, such as one or more CPUs; or they may be different types of processors, such as one or more CPUs and one or more ASICs.

Memory 506 is used to store programs 510. The memory 506 may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 510 may be specifically used to cause the processor 502 to execute the scene segmentation method based on the region of interest in any of the above method embodiments. For the specific implementation of each step in the program 510, please refer to the corresponding steps and corresponding descriptions in the units in the above-mentioned region-of-interest-based scene segmentation embodiment, and will not be described again here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be described again here.

The algorithms or displays provided herein are not inherently associated with any particular computer, virtual system, or other device. Various general-purpose systems can also be used with teaching based on this. From the above description, the structure required to construct such a system is obvious. Furthermore, this application is not specific to any specific programming language. It should be understood that the subject matter described herein may be implemented using a variety of programming languages, and that the above descriptions of specific languages are for the purpose of disclosing preferred embodiments of the subject matter.

In the instructions provided here, a number of specific details are described. However, it is understood that embodiments of the present application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

Similarly, it will be understood that in the above description of exemplary embodiments of the present application, in order to streamline the present application and assist in understanding one or more of the various inventive aspects, various features of the present application are sometimes grouped together into a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this application.

Those skilled in the art will understand that modules in the devices in the embodiment can be adaptively changed and arranged in one or more devices different from that in the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method so disclosed may be employed in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of the equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features of different embodiments are meant to be within the scope of the present application. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will understand that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components according to the present application. The present application may also be implemented as an apparatus or device program (eg, computer program and computer program product) for performing part or all of the methods described herein. Such a program implementing the present application may be stored on a computer-readable medium, or may be in the form of one or more signals. Such signals may be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the element claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, third, etc. does not indicate any order. These words can be interpreted as names. Unless otherwise specified, the steps in the above embodiments should not be understood as limiting the order of execution.

Claims (14)

1. A scene cut method based on a region of interest, comprising:

pre-analyzing a video to be segmented, determining an interested region of the video to be segmented, and extracting first information of the interested region;

Performing scene pre-segmentation processing on the video to be segmented to obtain at least one pre-selected transition frame;

detecting the region of interest of the pre-selected transition frame to obtain second information of the region of interest;

and matching according to the first information and the second information to determine a transition frame meeting a preset transition condition for scene segmentation.

2. The method of claim 1, wherein the pre-analyzing the video to be segmented, determining a region of interest of the video to be segmented, and extracting first information of the region of interest further comprises:

pre-analyzing the video to be segmented, determining the region of interest of the video frame in the video to be segmented, and extracting to obtain first total number and coordinate information of the region of interest in the video frame; the coordinate information includes angular coordinate information, height information, and width information of the region of interest.

3. The method of claim 2, wherein the objects corresponding to the regions of interest are the same or different;

the first total number of the interested areas in the extracted video frame is specifically as follows: and counting the interested areas of different objects in the video frame to obtain a first total number of the interested areas.

4. The method of claim 2, wherein the pre-analyzing the video to be sliced, determining the region of interest of the video frames in the video to be sliced further comprises:

and pre-analyzing the video to be segmented, and determining an interested area of one or more frames of video frames in the video to be segmented.

5. The method of any of claims 1-4, wherein the method further comprises:

and determining a detection template image corresponding to the region of interest according to the coordinate information of the region of interest.

6. The method of claim 1, wherein said performing region of interest detection on said pre-selected transition frame to obtain second information of the region of interest further comprises:

and detecting the interested areas of the preselected transition frame according to the detection template image to obtain the interested areas contained in the preselected transition frame, and counting to obtain the second total number of the interested areas contained in the preselected transition frame.

7. The method of claim 6, wherein the detecting the region of interest of the preselected transition frame according to the detection template image, obtaining the region of interest contained in the preselected transition frame, and counting the second total number of the regions of interest contained in the preselected transition frame further comprises:

Detecting the region of interest of the preselected transition frame according to the detection template image to obtain a plurality of regions of interest contained in the preselected transition frame;

counting the interested areas of different objects in the preselected transition frame to obtain a second total number of the interested areas; if a plurality of regions of interest are the same object, recording the number of the regions of interest as 1;

and counting to obtain a second total number of each region of interest contained in the preselected transition frame.

8. The method of any of claims 1-7, wherein the matching according to the first information and the second information to determine a transition frame meeting a preset transition condition for scene cut further comprises:

calculating a ratio of the second total number to the first total number according to the first total number of the first information and the second total number of the second information;

judging whether the ratio exceeds a preset first transition threshold value or not;

if yes, determining that the pre-selected transition frame does not meet a preset transition condition.

9. The method according to any one of claims 1-7, wherein the pre-analyzing the video to be segmented, determining a region of interest of the video to be segmented, and extracting first information of the region of interest further comprises:

Pre-analyzing a video to be segmented, determining an interested region of the video to be segmented, and extracting and obtaining a first total number and coordinate information of the interested regions and weight values of objects of all the interested regions;

the matching according to the first information and the second information to determine a transition frame meeting a preset transition condition to perform scene segmentation further includes:

calculating the sum of the weight values of the objects in each region of interest according to the first information to obtain a first weight value;

determining the interested region contained in the preselected transition frame according to the second information, and accumulating the weight values of the objects of the corresponding interested region to obtain a second weight value;

calculating the ratio of the second weight value to the first weight value, and judging whether the ratio exceeds a preset second transition threshold value or not;

if yes, determining that the pre-selected transition frame does not meet a preset transition condition.

10. The method of claim 1, wherein the performing scene pre-segmentation processing on the video to be segmented to obtain at least one pre-selected transition frame further comprises:

based on scene detection, performing scene pre-segmentation processing on the video to be segmented to obtain a pre-selected transition frame number set containing at least one pre-selected transition frame number; the pre-selected transition frame numbers are in one-to-one correspondence with the pre-selected transition frames.

11. The method of claim 10, wherein the detecting the region of interest of the pre-selected transition frame to obtain second information of the region of interest, and matching according to the first information and the second information to determine a transition frame meeting a pre-set transition condition for scene cut further comprises:

traversing the pre-selected transition frame number set, and detecting the region of interest of a pre-selected transition frame corresponding to any pre-selected transition frame number to obtain second information of the region of interest of the pre-selected transition frame; according to the first information and the second information, matching is carried out, and whether the preselected transition frame is a transition frame meeting a preset transition condition or not is judged; if not, deleting the preselected transition frame number; and obtaining the next pre-selected transition frame number of the pre-selected transition frame number set, and continuing to traverse until the pre-selected transition frame number set is completed by traversing, so as to segment the scene according to the residual pre-selected transition frame numbers in the pre-selected transition frame number set.

12. A scene cut-off device based on a region of interest, comprising:

the region of interest module is suitable for pre-analyzing the video to be segmented, determining the region of interest of the video to be segmented, and extracting and obtaining first information of the region of interest;

The pre-segmentation module is suitable for carrying out scene pre-segmentation processing on the video to be segmented to obtain at least one pre-selected transition frame;

the detection module is suitable for detecting the region of interest of the pre-selected transition frame to obtain second information of the region of interest;

and the matching module is suitable for matching according to the first information and the second information so as to determine a transition frame meeting a preset transition condition to perform scene segmentation.

13. A computing device, comprising: the device comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus;

the memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform the operations corresponding to the region-of-interest-based scene segmentation method according to any one of claims 1-11.

14. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the region-of-interest based scene cut method of any of claims 1-11.