CN117336525A - Video processing method, device, computer equipment and storage medium - Google Patents

Abstract

This application discloses a video processing method, device, computer equipment and storage medium. The video processing method includes: identifying key frames in a video to be processed to obtain key frames in the video to be processed; dividing the video to be processed into Be a plurality of video clips; based on the key frame, extract a partial video frame from each video clip of the plurality of video clips; based on the target prompt information and the partial video frame, generate a third video frame for inputting a large language model An input information, the target prompt information is used to prompt the large language model to perform video understanding; input the first input information to the large language model to obtain the video understanding result corresponding to the video to be processed. This method can improve the accuracy of understanding long-term videos.

Description

Video processing method, device, computer equipment and storage medium

Technical field

The present application relates to the field of video processing technology, and more specifically, to a video processing method, device, computer equipment and storage medium.

Background technique

With the rapid development of the Internet and digital technology, a large amount of video data is generated in various application scenarios, such as security monitoring, intelligent transportation, medical health, etc. Video understanding technology is an essential foundation for a series of upper-level multi-modal artificial intelligence tasks such as question and answer, dialogue, and recommendation. However, related technologies have poor accuracy in video understanding of long videos.

Contents of the invention

This application proposes a video processing method, device, computer equipment and storage medium, which can improve the accuracy of understanding long-term videos.

In a first aspect, embodiments of the present application provide a video processing method. The method includes: performing key frame identification on a video to be processed to obtain key frames in the video to be processed; dividing the video to be processed into multiple Video clips; based on the key frame, extract a partial video frame from each video clip of the plurality of video clips; based on the target prompt information and the partial video frame, generate first input information for inputting a large language model , the target prompt information is used to prompt the large language model to perform video understanding; input the first input information to the large language model to obtain a video understanding result corresponding to the video to be processed.

In the second aspect, embodiments of the present application provide a video processing device, which includes: a key frame identification module, a video dividing module, a video frame extraction module, an information generation module, and a video understanding module, wherein the key frame identification module The module is used to identify key frames in the video to be processed to obtain the key frames in the video to be processed; the video dividing module is used to divide the video to be processed into multiple video segments; the video frame extraction module is used to Based on the key frame, a partial video frame is extracted from each video segment of the plurality of video segments; the information generation module is used to generate a large language model for input based on the target prompt information and the partial video frame. First input information, the target prompt information is used to prompt the large language model to perform video understanding; the video understanding module is used to input the first input information to the large language model to obtain the video to be processed Corresponding video understanding results.

In a third aspect, embodiments of the present application provide a computer device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and Configured to be executed by the one or more processors, the one or more application programs are configured to execute the video processing method provided by the above-mentioned first aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium. Program code is stored in the computer-readable storage medium. The program code can be called by a processor to execute the video provided in the first aspect. Approach.

The solution provided by this application is to obtain the key frames in the video to be processed by identifying the key frames in the video to be processed, and divide the video to be processed into multiple video segments. Based on the key frames, from each video segment of the multiple video segments Extract some video frames, generate first input information for inputting into the large language model based on the acquired partial video frames and target prompt information used to prompt the large language model for video understanding, and then input the first input information into the large language model model to obtain the video understanding results corresponding to the video to be processed. Therefore, when using a large language model to understand the video to be processed, after extracting some video frames based on the identified key frames and divided video segments, the input to the large language model is determined based on the extracted partial video frames. Input information, so when understanding long-term videos, there is no need to input the large language model multiple times after segmenting the video, so that the large language model can better understand the long-term videos and improve the understanding of long-term videos. accuracy.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 shows a schematic flowchart of a video processing method according to an embodiment of the present application.

Figure 2 shows a schematic flowchart of a video processing method according to another embodiment of the present application.

Figure 3 shows a schematic flowchart of a video processing method according to yet another embodiment of the present application.

Figure 4 shows a schematic flowchart of a video processing method according to yet another embodiment of the present application.

Figure 5 shows a schematic flowchart of a video processing method according to yet another embodiment of the present application.

Figure 6 shows a schematic flowchart of a video processing method according to yet another embodiment of the present application.

Figure 7 shows a block diagram of a video processing device according to an embodiment of the present application.

FIG. 8 is a block diagram of a computer device for executing a video processing method according to an embodiment of the present application.

Figure 9 is a storage unit used to save or carry the program code for implementing the video processing method according to the embodiment of the present application.

Detailed ways

In order to enable those in the technical field to better understand the solution of the present application, the technical solution in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application.

Video understanding is a technology covering multiple fields, including computer vision, natural language processing, speech recognition, etc. In video understanding, video data usually needs to be processed to identify and understand the content in the video, such as identifying and classifying objects, scenes, events, etc. in the video. In the early technology of video understanding, video understanding can be carried out through video data processing technology. Video data processing technology refers to converting the original video into an analyzable format, such as extracting the image frames in the video and preprocessing them so that Carry out subsequent tasks such as image recognition and target detection. In video understanding, this technology can help the computer accurately identify and understand various elements in the video, such as characters, scenes, actions, etc.

With the rise of artificial intelligence (AI) technology in recent years, video understanding has emerged through deep learning. Neural network models can be trained to extract useful features from a large amount of data. In video understanding, this Technology can help computers accurately identify and understand content in videos. Moreover, with the development of artificial intelligence, video understanding can be carried out through large language models (LLM). Large language models refer to complex artificial neural networks trained based on large amounts of data and computing resources, which can learn rich Language patterns and knowledge to produce accurate responses to natural language input.

In related technologies, when using a large language model for video understanding, you can perform feature extraction on the video frames of the original video, convert the extracted image features of the video frames into the feature space of the large language model, and then input into the large language model, so that the large language model outputs video understanding results based on the input content. However, when the duration of the original video is long, the number of video frames in the original video is large, and the large language model has limitations on the content of each input. Therefore, when the actual input original video is processed, the original video will be processed. The video is divided into multiple videos, and then feature extraction is performed on the video frames of each video, and then converted to the feature space of the large language model, and then input into the large language model to obtain the video understanding results corresponding to each video. In this way, since the video is divided into multiple segments and the input information is determined separately and then input into the large language model, it will cause the large language model to lose the overall information of the original video when it performs video understanding, which will further affect the long-term performance of the video. The video perception and understanding is limited, and the granularity of the understanding of video content is relatively coarse, and the content in the video cannot be understood in a refined manner.

In response to the above problems, the inventor proposed the video processing method, device, computer equipment and storage medium provided by the embodiments of the present application, which can realize video understanding of the video to be processed through a large language model based on the identified key frames and divisions. After extracting some video frames from the video clip, the input information for entering the large language model is determined based on the extracted partial video frames. Therefore, when understanding long-term videos, there is no need to input the large language model multiple times after segmenting the video. This enables the large language model to better understand long-term videos and improve the accuracy of understanding long-term videos. The specific video processing method will be described in detail in subsequent embodiments.

The video processing method provided by the embodiment of the present application will be introduced in detail below with reference to the accompanying drawings.

Please refer to Figure 1. Figure 1 shows a schematic flow chart of a video processing method provided by an embodiment of the present application. In a specific embodiment, the video processing method is applied to the video processing device 700 shown in FIG. 7 and the computer device 100 (FIG. 8) configured with the video processing device 700. The following will take a computer device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the computer device applied in this embodiment can be a server, a smart phone, a tablet computer, a smart watch, a notebook computer, etc., which will not be used here. Make limitations. The process shown in Figure 1 will be described in detail below. The video processing method may specifically include the following steps:

Step S110: Perform key frame identification on the video to be processed to obtain key frames in the video to be processed.

In the embodiment of the present application, when the computer device performs video understanding on the video to be processed, considering that the video to be processed is long, it needs to divide the video to be processed into multiple videos, and then convert the multiple videos into large language input formats. The information of the model is then input into the large language model to obtain the video understanding results corresponding to each video. This will cause the large language model to lose the overall information of the original video when performing video understanding, thus making the video to be processed difficult to process. The accuracy of understanding is insufficient, so the computer device can identify key frames for the video to be processed that requires video understanding, so that the video to be processed can be compressed based on the identified key frames in the video to be processed, thereby reducing the number of videos to be processed. video frames in the video frame, thereby eliminating the need to input a large language model multiple times after segmenting the video. Among them, key frames refer to video frames in which the degree of change of the video content in the video to be processed is greater than the target threshold. It can be understood that when the degree of change of the video content in the video to be processed is large, it is usually characters, objects, etc. in the video. The video frame where the key action in the motion changes is located, or the video frame where the scene content is switched, so the key frame can be better representative in its adjacent video frames.

In some implementations, the computer device can identify key frames in the video to be processed through an inter-frame difference algorithm. Among them, the inter-frame difference algorithm is a method of obtaining the outline of a moving target by performing a difference operation on two consecutive frames of a video image sequence. Specifically, this algorithm first needs to obtain two consecutive frames of images in the video sequence, and then The outline of the moving target is obtained by calculating the difference between the two frames of images. If there is an obvious difference between the two frames of images, then this difference will be recorded. This difference can be calculated by calculating the difference between the pixel values of the corresponding positions in the image. The absolute value of the difference is obtained.

In the above method, when identifying key frames in the video to be processed through the inter-frame difference algorithm, the degree of change between two adjacent video frames can be obtained for each two adjacent video frames; if the latter video frame If the change degree of the frame relative to the previous video frame is greater than the target threshold, the next video frame can be determined as a key frame; if the change degree of the next video frame relative to the previous video frame is not greater than the target threshold, then the subsequent video frame can be determined as a key frame. A video frame is not a keyframe.

In some implementations, a pre-trained key frame recognition model can also be used to perform key frame recognition on the video to be processed, thereby obtaining the key frames in the video to be processed. When identifying key frames for the video to be processed, each video frame of the video to be processed can be input into the key frame recognition model to obtain the recognition result of each video frame by the key frame recognition model; based on the recognition of each video frame As a result, key frames in the video to be processed can be determined. Among them, the specific type of the key frame recognition model is not limited. For example, the key frame recognition model can be a deep neural network (DeepNeural Network, DNN), a long short-term memory neural network (Long Short-Term Memory, LSTM), a Transformer network, etc. .

In a possible implementation, the recognition result of the key frame recognition model for each video frame may include whether the video frame is a key frame. For example, the recognition result may include 1 and 0. If the recognition result is 1, it means that the video frame is a key frame. If the recognition result is 0, it means that the video frame is not a key frame. In this way, based on the recognition results of each video frame by the key frame recognition model, it can be determined which video frames in the video to be processed are key frames.

In a possible implementation, the recognition result of each video frame by the key frame recognition model may include a key frame probability. The key frame probability is used to characterize the probability that the video frame is a key frame. The greater the key frame probability, the The more likely the video frame is a key frame. In this method, the key frame probability corresponding to each video frame can be compared with the key frame threshold; according to the comparison result, if the key frame probability is greater than the key frame threshold, the video frame can be determined to be a key frame; if the key frame probability is not greater than the key frame threshold, it can be determined that the video frame is not a key frame. In this way, the key frames existing in the video to be processed can be determined through further judgment on the recognition results corresponding to each video frame.

In a possible implementation, the above key frame recognition model can be trained in the following manner: obtain a sample video, and each video frame in the sample video can be marked with a key frame label. The key frame label is used to characterize whether the video frame is a key frame; when the key frame recognition model is obtained based on sample video training, each sample video frame of the sample video can be input into the initial recognition model to obtain the recognition result of the initial recognition model for the sample video frame, and then based on each frame of sample The recognition results of the video frames and the key frame labels marked on each sample video frame determine the loss value corresponding to the initial recognition model; after determining the loss value corresponding to the initial recognition model, the initial recognition model can be evaluated based on the loss value. Iterative training to obtain the final key frame recognition model. When determining the loss value corresponding to the initial recognition model, the loss value can be determined based on the difference between the recognition result corresponding to each sample video frame and the annotated key frame label.

Optionally, when iteratively training the initial recognition model based on the determined loss value, the model parameters of the initial recognition model can be adjusted based on the calculated loss value; the sample video frames of the sample video are repeatedly input to the initial recognition model, Obtain the recognition results of the input sample video frames output by the initial recognition model. Based on the recognition results of the sample video frames and the annotated key frame labels, determine the loss value corresponding to the initial device model, and adjust the model parameters of the initial recognition model based on the loss value. , until the training end conditions are met, and the trained key frame recognition model is obtained. The training end conditions of iterative training may include: the number of iterative training reaches the target number; or the total loss value of the initial recognition model meets the set conditions.

Of course, the specific method for identifying key frames in the video to be processed may not be limited in the embodiments of this application.

In some implementations, when the execution subject of the video processing method provided by the embodiments of the present application is a server, the video to be processed can be a video uploaded by an electronic device, or a video stored in the server according to a selection instruction sent by the electronic device. The target video selected in the application; when the execution subject of the video processing method provided by the embodiment of the present application is an electronic device, such as a smartphone, a tablet, a laptop, etc., the video to be processed can be a video stored locally on the electronic device, or it can be It is a video downloaded from the server.

Step S120: Divide the video to be processed into multiple video segments.

In this embodiment of the present application, when the computer device performs video understanding on the video to be processed, it can also divide the video to be processed into multiple video segments, so that the video to be processed can be processed based on the divided video segments and the key frames identified above. Compression, thereby reducing video frames in the video to be processed.

In some embodiments, when the computer device divides the video to be processed into video clips, it can divide the video to be processed according to different scenes, different characters, different objects, etc. in the video clips. For example, when the scene in the video to be processed changes. , the starting point of the next video clip is determined, and when the scene of the video to be processed changes again, the end point of the next video clip is determined, thereby dividing a video clip.

In some implementations, the computer device can also divide the video to be processed into multiple video segments according to the target time length. For example, the video to be processed can be divided into multiple video segments with a time length of 10 seconds.

Of course, the specific method of dividing the video to be processed into video segments may not be limited in the embodiments of this application.

Step S130: Based on the key frame, extract a partial video frame from each video segment of the plurality of video segments.

In this embodiment of the present application, after the key frames in the video to be processed are identified and the video to be processed is divided into multiple video segments, the video to be processed can be processed based on the divided video segments and the key frames identified above. Compression, thereby reducing video frames in the video to be processed. Among them, partial video frames can be extracted from each video segment of the divided multiple video segments based on key frames to reduce the number of video frames in each video segment, thereby reducing the input used to determine the input of the large language model Number of video frames of information. Understandably, since the key frame is representative among its adjacent video frames, partial video frames can be extracted from each video segment based on this, thereby reducing the number of video frames and at the same time. This ensures that the information in the video to be processed can be effectively extracted during subsequent video understanding.

In some implementations, for each of the above multiple video segments, it may be determined whether the video segment includes a key frame; if the video segment includes a key frame, the key frame and the key frame may be obtained from the video segment. The first target number of video frames adjacent to the key frame are used as part of the video frames in the video clip; if the video clip does not include key frames, the second target number of videos can be extracted from the video clip according to the specified extraction rules. frame. The specific values of the first target number and the second target number may not be limited. For example, they may be 5 frames, 10 frames, 20 frames, etc.

In a possible implementation, the second target number may be smaller than the first target number, that is, for a video segment including key frames, when extracting partial video frames, the number of extracted partial video frames may be more than not The number of partial video frames extracted from the video clip that includes keyframes. Understandably, since video clips that do not include key frames are clips with little picture change, the information contained in their video frames is relatively close, so fewer video frames can be extracted to further reduce the number of frames used in subsequent video understanding. The number of video frames.

In a possible implementation, when no key frame is included in any video segment, when a second target number of video frames are extracted from the video segment according to the specified extraction rule, the video frame may be randomly extracted from the video segment. A second target number of video frames.

In a possible implementation, when any video clip does not include key frames, when extracting a second target number of video frames from the video clip according to the specified extraction rules, it is also possible to extract the video frames that meet the target distribution conditions. The second target number of video frames are used as partial video frames in the video clip. Among them, the target distribution condition can be uniformly distributed in the video clips, thereby ensuring that information can be effectively extracted from the video clips during subsequent video understanding.

Of course, in the embodiment of the present application, the specific method of extracting some video frames from each video segment of multiple video segments based on key frames is not limited. It only needs to ensure that the necessary video frames are retained from the video to be processed based on the key frames. Video frames are required to effectively extract the information in the video to be processed.

Step S140: Generate first input information for inputting a large language model based on target prompt information and the partial video frames, where the target prompt information is used to prompt the large language model to perform video understanding.

In this embodiment of the present application, after extracting part of the video frames for each of the above video clips, the input information for inputting the large language model can be determined based on all the extracted video frames. Among them, the input information for inputting the large language model can be generated based on the target prompt information and all the video frames extracted above, and used as input information. Among them, the target prompt information can be a prompt word, that is, prompt, used to prompt the large language model to understand the video. The target prompt information can provide the user's intention. After the large language model subsequently understands the target prompt information, it can know the user's intention and Perform corresponding processing according to user intent. Understandably, since it is for the video to be processed, after extracting some video frames from each divided video segment through the identified key frames, the input information for inputting into the large language model is then generated, so the need for generating input information can be reduced. The number of video frames can ensure that these remaining video frames can provide the necessary information for video understanding. Therefore, for the video to be processed, the input information for inputting the large language model only needs to be generated once, and the input information will be retained in the to-be-processed video. Processing the overall information of the video makes the video understanding results output by the large language model more accurate.

In some implementations, feature extraction can be performed on some of the video frames obtained above, and then the extracted features are converted to the feature space of a large language model, that is, they are converted into text tokens that are usually input to the large language model, so that they can be used with After the above target prompt information (also text tokens) is spliced, the first input information used to input the large language model is obtained.

Step S150: Input the first input information to the large language model to obtain a video understanding result corresponding to the video to be processed.

In the embodiment of the present application, after obtaining the above first input information, the first input information can be input into the large language model, thereby obtaining a video understanding result corresponding to the video to be processed. Among them, the video understanding results may include a description of the video, answers to the above target prompt information after video understanding, etc. The specific content of the video understanding results may not be limited. The above large language model can be a pre-trained model capable of video understanding. The large language model can be a generative large language model based on AIGC, and the large language model can generate video understanding results based on the above input information. The large language model understands the above target prompt information in the first input information based on the above first input information. After obtaining the user's intention, it can output video understanding results based on the information extracted from the above part of the video frames in the above first input information. .

In some implementations, considering that the capabilities of large language models are relatively broad, and there may be many scenarios for video understanding, a Low-Rank Adaptation (LoRA) model can also be added to the large language model, and the corresponding Sample videos in the scene are used to train the LoRA model, so that the large language model can better understand the video in these scenes. Among them, the LoRA model is obtained by fine-tuning the cross-attention layer in the UNet module in the large language model; the LoRA model is applied to the cross-attention layer in the large language model, by freezing the parameters of the large language model and using the corresponding Sample videos in the scene are trained to obtain a LoRA model that can better understand the video in the scene. When used, the parameters of the LoRA model are injected into the large language model, so that the large language model outputs Video understanding results in this scenario.

In some implementations, when the execution subject of the video processing method provided by the embodiments of the present application is an electronic device, then part of the above steps S110 to step S150 may be executed locally on the electronic device, and part of the steps may be submitted by the electronic device to a corresponding request. Server, executed by the server and returns the result. For example, to perform key frame identification on video frames to be processed, the electronic device can submit a key frame identification request to the corresponding server and upload the video to be processed to the server, so that the server can perform key frame identification on the video to be processed and feed the identification results back to Electronic device; The electronic device locally divides the video to be processed into multiple video segments, and then extracts partial video frames from each of the multiple video segments based on key frames; then uploads the extracted partial video frames and target prompt information to The server, after generating the first input information for inputting the large language model based on the target prompt information and part of the video frames, inputs the first input information into the large language model to obtain the video understanding result corresponding to the video to be processed, And return the video understanding results to the electronic device.

The video processing method provided by the embodiment of the present application uses a large language model to perform video understanding of the video to be processed. After extracting some video frames based on the identified key frames and divided video segments, the video processing method is determined based on the extracted partial video frames. Enter the input information of the large language model. Therefore, when understanding long-term videos, there is no need to input the large language model multiple times after segmenting the video. This allows the large language model to better understand the long-term videos and improve the understanding of the long-term videos. Accuracy of long-duration video comprehension.

Please refer to Figure 2. Figure 2 shows a schematic flowchart of a video processing method provided by another embodiment of the present application. This video processing method is applied to the above-mentioned computer equipment. The process shown in Figure 2 will be described in detail below. The video processing method may specifically include the following steps:

Step S210: Perform key frame identification on the video to be processed to obtain key frames in the video to be processed.

In this embodiment of the present application, reference may be made to the content of the foregoing embodiments for step S210, which will not be described again here.

Step S220: Divide the video to be processed into multiple video segments.

In some embodiments, when dividing the video to be processed into multiple video segments, the video to be processed may be divided into multiple video segments based on key frames, wherein each video segment of the multiple video segments includes at least one frame the key frame. That is to say, when dividing the video clips for the video to be processed, it is done with reference to the identified key frames, and each divided video clip includes at least one key frame, so that each video clip includes At least one video frame with a change greater than the target threshold is obtained.

In a possible implementation, when dividing the video to be processed into multiple video segments based on key frames, the video to be processed can be marked with multiple target timestamps based on the above key frames, where two adjacent targets The videos between timestamps constitute a video clip, and the target timestamp is between two adjacent keyframes. That is to say, the video to be processed can be segmented by marking the timestamp, so that two adjacent target timestamps can be divided into one video segment, thereby dividing multiple video segments. When further processing the video to be processed, the video to be processed can be further processed according to the annotation. timestamp, extract part of the video frame from the video clip consisting of each two adjacent timestamps.

Optionally, the above target key frame can be located between two key frames, and the change degree of the video picture is less than the target threshold; the duration of each video segment divided above is greater than or equal to the specified duration, such as 10 seconds, 20 seconds, 30 seconds, 60 seconds, etc.

Optionally, the key frames identified above may include primary key frames and secondary key frames. The primary key frame is a video frame whose degree of change relative to the previous video frame is greater than the first threshold. The secondary key frame is a video frame relative to the previous frame. The degree of change of the video frame is greater than the second threshold of the video frame, and the first threshold is greater than the second threshold. That is to say, the degree of change of the main key frame relative to the previous video frame and the change of the secondary key frame relative to the previous video frame The degrees are all greater than a certain threshold, but the degree of change of the main key frame relative to the previous frame is greater. Since the change degree of the main key frame relative to the previous frame is greater than the change degree of the secondary key frame relative to the previous frame, the main key frame is more representative. Therefore, based on the above key frames, the video to be processed is marked with multiple target times. When stamping, you can select the target time point between the two main key frames and the change degree of the video picture is less than the target threshold based on the main key frame, and mark the target timestamp at the target time point.

Step S230: Based on the key frame, sample each video segment of the plurality of video segments to obtain partial video frames in each video segment.

In the embodiment of the present application, after identifying the key frames in the video to be processed and dividing the video to be processed into multiple video segments, sampling can be performed for each video segment of the multiple video segments based on the above key frames. , thereby obtaining partial video frames in each video clip. That is, a certain number of video frames are sampled from each video segment to obtain partial video frames in each video segment.

In some implementations, the above video segments may be divided based on key frames, and each video segment includes at least one key frame. When sampling each video segment of multiple video segments based on key frames, each video segment can be divided based on the key frames in each video segment to obtain multiple video sub-segments corresponding to each video segment. ;Sample each video sub-segment in the multiple video sub-segments corresponding to each video segment to obtain some video frames in each video segment. Among them, the sampling rate corresponding to the video sub-segment where the key frame is located is higher than that of other videos. The sampling rate corresponding to the sub-segment, and the other video sub-segments are the video sub-segments among the multiple video sub-segments except the video sub-segment where the key frame is located.

In the above implementation, based on the key frames in each video segment, each video segment can be divided into video sub-segments containing key frames and video sub-segments not containing key frames. That is to say, for each video segment, the video segment can also be further divided according to the position of the key frame in the video segment, and multiple video sub-segments are obtained, so that some video sub-segments include key frames, and some video sub-segments include key frames. The sub-segment does not include key frames, and then the video sub-segment where the key frame is located is sampled according to the sampling rate corresponding to the video sub-segment where the key frame is located, and the video sub-segment where the key frame is located is sampled according to the sampling rate corresponding to the video sub-segment excluding key frames. Sample video sub-clips. Moreover, since video clips that do not include key frames are clips with little picture change, the information contained in each video frame is relatively close, so a relatively low sampling rate can be used to extract relatively few video frames. For video clips that include key frames, a relatively higher sampling rate can be used to retain the necessary video frames to effectively extract information from the video to be processed.

In a possible implementation, after dividing each video segment into video sub-segments containing key frames and video sub-segments not containing key frames based on the key frames in each video segment, if the key frames are included The video sub-segments of the frame are two adjacent video sub-segments, then the duration between the key frames in the two video sub-segments can be determined. If the distance is less than the duration threshold, the two video sub-segments can be Merge into one video sub-segment so that after sampling, effective information for video understanding can be better retained.

In a possible implementation, the key frames identified above may include primary key frames and secondary key frames. The primary key frame is a video frame whose degree of change relative to the previous video frame is greater than the first threshold. The secondary key frame It is a video frame whose degree of change relative to the previous video frame is greater than the second threshold, and the first threshold is greater than the second threshold. That is to say, the degree of change of the main key frame relative to the previous video frame is the same as the degree of change of the secondary key frame relative to the previous video frame. The degree of change of each video frame is greater than a certain threshold, but the degree of change of the main key frame is greater than the previous frame. In the above embodiments, the sampling rate of the video sub-segment including the primary key frame may be greater than the sampling rate of the video sub-segment including the secondary key frame, so that effective information for video understanding can be better retained.

In some implementations, among the multiple video segments divided above, some video segments may include key frames, and some video segments may not include key frames.

For each video segment including a key frame, the video segment can be divided based on the key frames in the video segment according to the previous embodiment to obtain multiple video sub-segments corresponding to the video segment, and then multiple video sub-segments corresponding to the video segment are obtained. Each video sub-segment in the video sub-segment is sampled to obtain some video frames in the video segment. Among them, the sampling rate corresponding to the video sub-segment where the key frame is located is higher than the sampling rate corresponding to other video sub-segments. Other video sub-segments It is a video sub-segment among multiple video sub-segments except the video sub-segment where the key frame is located.

For each video segment that does not include key frames, it can be divided into multiple video sub-segments of the same duration, and then determine the duration between each video sub-segment and the nearest key frame in position, and then determine each video segment based on this duration. The sampling rate corresponding to the video sub-clip. The sampling rate is negatively correlated with the duration. That is to say, the further away from the nearest key frame, the lower the sampling rate. In this way, the number of retained video frames can be further reduced. At the same time, it better retains effective information for video understanding. For example, in any video segment that does not include key frames, including video sub-segment 1, video sub-segment 2 and video sub-segment 3, video sub-segment 1 is 10 seconds away from its nearest key frame, and video sub-segment 2 is 10 seconds away from its nearest key frame. The distance between key frames is 8 seconds, and the distance between video sub-segment 3 and its nearest key frame is 15 seconds. Then the sampling rate corresponding to video sub-segment 2 is the highest, the sampling rate corresponding to video segment 1 is second, and the sampling rate corresponding to video segment 3 is The sampling rate is the lowest.

In some implementations, considering that the video duration of the video to be processed that needs to be processed for video understanding is not fixed each time, in order to ensure that input information for inputting a large language model can be generated at one time based on all partial video frames extracted, in After sampling each video segment of multiple video segments, the total number of video frames obtained may be a fixed value, such as 400 frames, 500 frames, 700 frames, etc. When sampling each video segment of multiple video segments, the sampling rate can be dynamically adjusted according to the specific duration of the video to be processed to ensure that the total number of partial video frames corresponding to each video segment is the above fixed value.

Step S240: Generate first input information for inputting a large language model based on target prompt information and the partial video frames, where the target prompt information is used to prompt the large language model to perform video understanding.

Step S250: Input the first input information to the large language model to obtain a video understanding result corresponding to the video to be processed.

In the embodiment of the present application, the content of step S240 and step S250 can be referred to the previous embodiment, and will not be described again here.

The video processing method provided by the embodiment of the present application uses a large language model to perform video understanding on the video to be processed. The divided video segments are sampled according to the identified key frames. After obtaining partial video frames, the video is then processed based on the obtained partial video frames. Determine the input information used to input the large language model, so when understanding the long-term video, there is no need to input the large language model multiple times after segmenting the video, thus enabling the large language model to better understand the long-term video. , to improve the accuracy of understanding long-term videos; and, because some video frames are sampled based on key frames, necessary video frames can be retained, so that the input information determined based on the partial video frames can be input into the large language model. The large language model can effectively extract the information in the video to be processed, thereby ensuring the accuracy of video understanding.

Please refer to Figure 3, which shows a schematic flowchart of a video processing method provided by yet another embodiment of the present application. This video processing method is applied to the above-mentioned computer equipment. The process shown in Figure 3 will be described in detail below. The video processing method may specifically include the following steps:

Step S310: Perform key frame identification on the video to be processed to obtain the key frames in the video to be processed.

Step S320: Divide the video to be processed into multiple video segments, and each video segment includes at least one frame of the key frame.

In the embodiment of the present application, the content of steps S310 and S320 can be referred to other embodiments, and will not be described again here.

Step S330: For each video segment in the plurality of video segments, obtain a first number of video frames adjacent to the key frame, and obtain a second number of video frames from other video frames to obtain each of the video frames. The first number of video frames and the second number of video frames corresponding to each video segment, wherein the other video frames are the key frames in each video segment except the key frames and the adjacent key frames. Video frames other than the first number of video frames.

In the embodiment of the present application, after identifying the key frames in the video to be processed and dividing the video to be processed into multiple video segments, when extracting partial video frames from each video segment based on the key frames, the key frames can be extracted for each video segment. A video clip is obtained, a first number of video frames adjacent to the key frame are obtained, and a second number of video frames are obtained from other video frames. That is to say, a first number of continuous video frames containing key frames are extracted, and a second number of video frames are extracted from other video frames except the extracted video frames, thereby not only reducing the number of video frames, but Necessary video frames can also be preserved.

In some implementations, when obtaining the second number of video frames from the above other video frames, sampling can be performed in other video frames to obtain the second number of video frames. Wherein, the sampling rate for sampling may be determined according to the second number and the number of other video frames. Extracting the second number of video frames through sampling can ensure that the video frames extracted from other video frames are evenly distributed, thereby ensuring that information from the video clip can be effectively extracted during subsequent video understanding.

Step S340: Determine the key frame, the first number of video frames, and the second number of video frames corresponding to each video segment as partial video frames corresponding to each video segment.

In this embodiment of the present application, after extracting the above first number of video frames and the above second number of video frames for each video segment, the key frames in each video segment, the above first number of video frames can be and a second number of video frames, determined as partial video frames corresponding to each video segment.

Step S350: Generate first input information for inputting a large language model based on target prompt information and the partial video frames, where the target prompt information is used to prompt the large language model to perform video understanding.

Step S360: Input the first input information to the large language model to obtain a video understanding result corresponding to the video to be processed.

In the embodiment of the present application, the content of steps S350 and S360 can be referred to other embodiments, and will not be described again here.

The video processing method provided by the embodiment of the present application uses a large language model to perform video understanding of the video to be processed. According to the identified key frames, a first number of video frames adjacent to the key frames are obtained from each video segment, and from Obtain the second number of video frames from other video frames of each video segment. After obtaining the partial video frames corresponding to each video segment, the input information for inputting the large language model is determined based on the obtained partial video frames. Therefore, after When understanding long-term videos, there is no need to input the large language model multiple times after segmenting the video, so that the large language model can better understand the long-term videos and improve the accuracy of understanding the long-term videos; and, Since some video frames at least contain key frames and their adjacent video frames, the effective information in the video clips is retained, so that after the input information determined based on the partial video frames is input to the large language model, the large language model can effectively extract the to-be-processed information. The information in the video thus ensures the accuracy of video understanding.

Please refer to Figure 4, which shows a schematic flowchart of a video processing method provided by yet another embodiment of the present application. This video processing method is applied to the above-mentioned computer equipment. The process shown in Figure 4 will be described in detail below. The video processing method may specifically include the following steps:

Step S410: Perform key frame identification on the video to be processed to obtain the key frames in the video to be processed.

Step S420: Divide the video to be processed into multiple video segments.

Step S430: Based on the key frame, extract a partial video frame from each video segment of the plurality of video segments.

In this embodiment of the present application, the contents of steps S410 to S430 may be referred to other embodiments, and will not be described again here.

Step S440: Perform feature extraction on the partial video frames to obtain image features corresponding to the partial video frames.

In the embodiment of this application, after extracting part of the video frames for each video clip based on the identified key frames, based on the target prompt information and the above part of the video frames corresponding to all video clips, a third video frame for inputting the large language model is generated. When information is input, feature extraction can be performed on all the extracted video frames (that is, the above part of the video frames corresponding to all the above video clips) to obtain the image features corresponding to these video frames, so that these image features can be subsequently converted into large-scale input images. The information of the language model is converted into the feature space of the large language model, so that it can be input into the large language model. The image features extracted for each video frame may include global features and patch features of multiple image blocks. Global features usually refer to features extracted from the entire image, such as color, texture, and shape. These features can describe the content and contextual information of the entire image and are often used in tasks such as image classification, target detection, and scene recognition; patch features are The features of local small blocks or patches extracted from the image are usually used to express local detailed information of the image. These features can be color, texture or shape, etc.

In some implementations, partial video frames corresponding to all video clips can be input to a pre-trained feature extraction model, thereby obtaining image features corresponding to each video frame in these video frames. The specific type of the feature extraction model used for image feature extraction is not limited. For example, it can be a visual encoder, such as a low-resolution variant of the visual classifier (Vision Transformer for Low Resolution Classification, ViT-L).

Step S450: Pool the image features corresponding to the partial video frames, and splice the pooled image features to obtain features to be input.

In the embodiment of this application, after extracting the image features corresponding to the above video frames, the image features corresponding to the video frames can be pooled to reduce the amount of data, subsequent calculation amount and memory consumption, and then pooling The final image features are spliced to obtain the features to be input.

In some embodiments, the image features corresponding to the above video frames may include global features and block features corresponding to multiple image blocks. The image features corresponding to the above video frames are pooled, and the pooled image features are spliced. , when the features to be input are obtained, the image features can be input into the spatial pool and the temporal pool, so that the image features corresponding to the above video frames are spatially pooled through the spatial pool, and the image features corresponding to the above video frames are temporally pooled through the temporal pool. ization, and then splicing the pooled image features obtained by spatial pooling and temporal pooling to obtain the above input features.

In this method, in the temporal pool, the global features corresponding to some video frames of all the above video clips can be spliced according to the order of the video frames in the above video frames to obtain the first feature; in the spatial pool, the first feature can be obtained The block features corresponding to some video frames of all the above video clips are average pooled to obtain the second feature; then the first feature and the second feature are spliced to obtain the feature to be input. In this way, since the time pool is processed according to the time order of all the video frames retained, that is, the order of the video frames in the video to be processed, time information can be retained and the accuracy of video understanding can be ensured.

In the above method, when the block features corresponding to some video frames of all the above video clips are average pooled, the multiple block features of all video frames in each time period can be average pooled to obtain the corresponding block features of each video frame. The second feature corresponding to all video frames in the time period. The time period may be determined based on the above video segments, and specifically may be the time period consisting of the timestamp of the earliest video frame in the partial video frames extracted from each video segment to the timestamp of the latest video frame. .

Step S460: Splice the features to be input and the target prompt information to obtain the first input information for inputting the large language model. The target prompt information is used to prompt the large language model to perform video understanding.

Step S470: Input the first input information to the large language model to obtain a video understanding result corresponding to the video to be processed.

In the embodiment of the present application, the contents of steps S460 and S470 may be referred to other embodiments, and will not be described again here.

The video processing method provided by the embodiment of the present application uses a large language model to perform video understanding of the video to be processed, and then extracts image features from the extracted partial video frames based on the identified key frames and divided video segments. Finally, the image features are pooled and then spliced with the prompt words to obtain the input information for inputting the large language model. Therefore, when understanding the long-term video, the video to be input can be determined at one time for inputting the large language. The input information of the model can enable the large language model to better understand long-term videos and improve the accuracy of understanding long-term videos.

Please refer to FIG. 5 , which shows a schematic flowchart of a video processing method provided by yet another embodiment of the present application. This video processing method is applied to the above-mentioned computer equipment. The process shown in Figure 5 will be described in detail below. The video processing method may specifically include the following steps:

Step S510: Obtain the video duration corresponding to the video to be processed.

In the embodiment of the present application, when performing video understanding on the video to be processed, the video duration corresponding to the video to be processed can be determined, so as to determine whether the video to be processed that needs to be video understood this time is a long-term video.

Step S520: If the video duration is less than or equal to the target duration, generate second input information for inputting a large language model based on the video frames in the video to be processed and the target prompt information.

In the embodiment of this application, after obtaining the video duration of the video to be processed, the video duration can be compared with the target duration; according to the comparison result, if the video duration is less than or equal to the target duration, it means that the video duration needs to be understood this time. The processed video is not a long-term video, so according to the conventional processing method, the input information determined for the video to be processed can be input into the large language model at one time, without the need to input the large language multiple times after segmenting the video to be processed. model, it can directly generate the second input information for inputting the large language model based on the video frames and target prompt information in the video to be processed.

Among them, feature extraction can be performed on the video frames in the video to be processed to obtain the image features corresponding to the video frames in the video to be processed; the video frames in the video to be processed are pooled, and the pooled image features are spliced. Obtain the features to be input; splice the features to be input and the target prompt information to obtain the second input information for inputting the large language model. For the pooling method of the image features of the video frame, please refer to the content of the previous embodiment, and will not be described again here.

Step S530: Input the second input information to the large language model to obtain a video understanding result corresponding to the video to be processed.

In this embodiment of the present application, after obtaining the above second input information, the second input information can be input into the large language model, thereby obtaining a video understanding result corresponding to the video to be processed. In this way, the input information can be input into the large language model once, and the video understanding of the video to be processed can be completed.

Step S540: If the video duration is greater than the target duration, perform key frame identification on the video to be processed to obtain the key frames in the video to be processed.

In the embodiment of this application, after obtaining the video duration of the video to be processed, the video duration is compared with the target duration; according to the comparison result, if the video duration is greater than the target duration, it means that the video to be processed this time needs to be video understood. For long-term videos, it is possible to identify key frames in the video to be processed, obtain the key frames in the video to be processed, and perform subsequent steps, thereby improving the accuracy of video understanding of long-term videos.

Step S550: Divide the video to be processed into multiple video segments.

Step S560: Based on the key frame, extract a partial video frame from each video segment of the plurality of video segments.

Step S570: Generate first input information for inputting a large language model based on target prompt information and the partial video frames, where the target prompt information is used to prompt the large language model to perform video understanding.

Step S580: Input the first input information to the large language model to obtain a video understanding result corresponding to the video to be processed.

In the embodiment of this application, the contents of steps S550 to S580 may be referred to other embodiments, and will not be described again here.

The video processing method provided by the embodiment of the present application uses a large language model to perform video understanding of the video to be processed, based on the video duration of the video to be processed, and when the video duration is greater than the target duration, based on the identified key frames and divided videos After extracting some video frames from the clip, the input information used to input the large language model is determined based on the extracted partial video frames. Therefore, when understanding the long-term video, the input information used to input the large language model can be determined for the video to be input at once. Input information, so that the large language model can better understand long-term videos and improve the accuracy of understanding long-term videos; and when the video duration is not greater than the target duration, it is directly based on the video to be processed The video frame determines the input information used to input the large language model, so that not only long-term videos can be accurately understood, but also short videos can be accurately understood.

The video processing method provided by the foregoing embodiment will be described below with an example.

Please refer to Figure 6. When performing video understanding on the video to be processed, the video to be processed can be divided into key frame videos and video segments, so that the video to be processed is marked with the timestamp used to divide the video segments and key frames; and then based on the annotation timestamps and key frames, extract part of the video frames from each video clip; then input the extracted video frames into the visual encoder to obtain the image features corresponding to each video frame; then input the extracted image features into the spatial pool and time pool , In the spatial pool, the global features in the image features corresponding to all video frames are spliced according to the time sequence of the video frames. In the temporal pool, the patch features in the image features corresponding to all video frames are average pooled; Then the features obtained from the spatial pool and the temporal pool are spliced and mapped to the feature space of the large language model. After being spliced with the target prompt information, they are input to the large language model, so that the large language model outputs video understanding results based on the input information, such as , in 3S-5.6S, two people were happily communicating; in 5min30sec-8min15sec, four people were racing in a melee, experiencing scenes of streets, bridges, and rivers.

Please refer to FIG. 7 , which shows a structural block diagram of a video processing device 700 provided by an embodiment of the present application. The video processing device 700 applies the above-mentioned computer equipment. The video processing device 700 includes: a key frame identification module 710, a video segmentation module 720, a video frame extraction module 730, an information generation module 740, and a video understanding module 750. Among them, the key frame identification module 710 is used to perform key frame identification on the video to be processed to obtain the key frames in the video to be processed; the video dividing module 720 is used to divide the video to be processed into multiple video segments. ; The video frame extraction module 730 is configured to extract a partial video frame from each video segment of the plurality of video segments based on the key frame; the information generation module 740 is configured to extract a partial video frame based on the target prompt information and the partial The video frame is used to generate the first input information for inputting the large language model, and the target prompt information is used to prompt the large language model to perform video understanding; the video understanding module 750 is used to input the first input information to The large language model obtains the video understanding results corresponding to the video to be processed.

In some implementations, the video frame extraction module 730 may be specifically configured to sample each video segment of the plurality of video segments based on the key frame to obtain partial video frames in each video segment.

In a possible implementation, each video segment includes at least one frame of the key frame. The video frame extraction module 730 may also be specifically configured to divide each video segment based on the key frame in each video segment to obtain multiple video sub-segments corresponding to each video segment; Each video sub-segment in the plurality of video sub-segments corresponding to each video segment is sampled to obtain partial video frames in each video segment, where the sampling rate corresponding to the video sub-segment where the key frame is located is Higher than the corresponding sampling rate of other video sub-segments, which are video sub-segments of the plurality of video sub-segments other than the video sub-segment where the key frame is located.

Optionally, the video frame extraction module 730 may also be specifically configured to divide the video sub-segments containing the key frames from each video segment based on the key frames in each video segment, and not The video subsegment containing the keyframe.

In some embodiments, each video segment includes at least one frame of the key frame, and the video frame extraction module 730 may be specifically configured to obtain, for each video segment in the plurality of video segments, information related to the key frame. frame adjacent first number of video frames, and obtain a second number of video frames from other video frames to obtain the first number of video frames and the second number of video frames corresponding to each video segment. , wherein the other video frames are video frames in each video segment other than the key frame and the first number of video frames adjacent to the key frame; Key frames, the first number of video frames, and the second number of video frames are determined as partial video frames corresponding to each video segment.

In some embodiments, the video dividing module 720 may be specifically configured to divide the video to be processed into multiple video segments based on the key frame, wherein each video segment of the multiple video segments includes at least one frame the key frame.

In a possible implementation, the video dividing module 720 can also be specifically configured to mark multiple target timestamps on the video to be processed based on the key frame, wherein the two adjacent target timestamps The video between them constitutes a video clip, and the target timestamp is located between two adjacent key frames.

In some embodiments, the information generation module 740 may be specifically configured to perform feature extraction on the partial video frames to obtain image features corresponding to the partial video frames; pool the image features corresponding to the partial video frames, and The pooled image features are spliced to obtain features to be input; the features to be input are spliced with the target prompt information to obtain first input information for inputting a large language model.

In a possible implementation, the image features corresponding to each video frame in the partial video frames include global features and block features corresponding to multiple image blocks. The information generation module 740 may also be specifically configured to generate The order of the video frames in the frame, splicing the global features corresponding to the partial video frames to obtain the first feature; performing average pooling on the block features corresponding to the partial video frames to obtain the second feature; The first feature and the second feature are spliced together to obtain the feature to be input.

In some embodiments, the video processing device may further include a duration acquisition module. The duration acquisition module may be used to acquire the to-be-processed video before performing key frame identification on the to-be-processed video to obtain the key frames in the to-be-processed video. Process the video duration corresponding to the video; the key frame identification module can be specifically used to perform key frame identification on the video to be processed if the video duration is greater than the target duration, and obtain the key frames in the video to be processed.

In a possible implementation, the information generation module may also be configured to, after obtaining the video duration corresponding to the video to be processed, if the video duration is less than or equal to the target duration, based on the video duration to be processed. video frames and the target prompt information to generate second input information for inputting into the large language model; the video understanding module can also be used to input the second input information into the large language model to obtain the to-be-processed Video understanding results corresponding to the video.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software function modules.

To sum up, the solution provided by this application obtains the key frames in the video to be processed by identifying the key frames in the video to be processed, divides the video to be processed into multiple video segments, and based on the key frames, extracts the key frames from the multiple video segments. Partial video frames are extracted from each video clip, and based on the obtained partial video frames and the target prompt information used to prompt the large language model for video understanding, the first input information for inputting the large language model is generated, and then the first input is The information is input to the large language model to obtain the video understanding results corresponding to the video to be processed. Therefore, when using a large language model to understand the video to be processed, after extracting some video frames based on the identified key frames and divided video segments, the input to the large language model is determined based on the extracted partial video frames. Input information, so when understanding long-term videos, there is no need to input the large language model multiple times after segmenting the video, so that the large language model can better understand the long-term videos and improve the understanding of long-term videos. accuracy.

Please refer to FIG. 8 , which shows a structural block diagram of a computer device provided by an embodiment of the present application. The computer device 100 may be a smartphone, a tablet computer, a smart watch, an e-book, or other device capable of running application programs. The computer device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more application programs, wherein one or more application programs may be stored in the memory 120 and configured to One or more processors 110 execute, and one or more application programs are configured to execute the method as described in the foregoing method embodiments.

Processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire computer device 100, and executes by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Various functions of computer device 100 and processing data. Optionally, the processor 110 may adopt at least one of digital signal processing (Digital Signal Processing, DSP), field-programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). implemented in hardware form. The processor 110 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the display content; and the modem is used to handle wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110 and may be implemented solely through a communication chip.

The memory 120 may include random access memory (RAM) or read-only memory (Read-Only Memory). Memory 120 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 120 may include a program storage area and a data storage area, where the program storage area may store instructions for implementing an operating system and instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing each of the following method embodiments, etc. The storage data area can also store data created during use of the computer device 100 (such as phone book, audio and video data, chat record data), etc.

Please refer to FIG. 9 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code is stored in the computer-readable medium 800, and the program code can be called by the processor to execute the method described in the above method embodiment.

Computer-readable storage medium 800 may be electronic memory such as flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any method steps in the above-described methods. These program codes can be read from or written into one or more computer program products. Program code 810 may, for example, be compressed in a suitable form.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: it can still Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (14)

1. A video processing method, characterized in that the method includes: Perform key frame identification on the video to be processed to obtain the key frames in the video to be processed; Divide the video to be processed into multiple video segments; extracting a partial video frame from each of the plurality of video segments based on the key frame; Generate first input information for inputting a large language model based on target prompt information and the partial video frame, where the target prompt information is used to prompt the large language model to perform video understanding; The first input information is input into the large language model to obtain a video understanding result corresponding to the video to be processed. 2. The method according to claim 1, characterized in that, based on the key frame, extracting a partial video frame from each video segment of the plurality of video segments includes: Based on the key frame, sampling is performed for each video segment of the plurality of video segments to obtain partial video frames in each video segment. 3. The method according to claim 2, characterized in that each video segment includes at least one frame of the key frame, and based on the key frame, for each video of the plurality of video segments, Sampling the clips to obtain partial video frames in each video clip, including: Based on the key frames in each video segment, divide each video segment to obtain multiple video sub-segments corresponding to each video segment; Sampling each video sub-segment in the plurality of video sub-segments corresponding to each video segment is performed to obtain partial video frames in each video segment, where the samples corresponding to the video sub-segment where the key frame is located are The rate is higher than the corresponding sampling rate of other video sub-segments, which are video sub-segments among the plurality of video sub-segments other than the video sub-segment where the key frame is located. 4. The method according to claim 3, characterized in that, based on the key frames in each video segment, dividing each video segment to obtain the corresponding Multiple video subclips, including: Based on the key frame in each video segment, video sub-segments containing the key frame and video sub-segments not containing the key frame are divided from each video segment. 5. The method of claim 1, wherein each video segment includes at least one frame of the key frame, and based on the key frame, each video segment from the plurality of video segments Extract some video frames from the clip, including: For each video segment in the plurality of video segments, obtain a first number of video frames adjacent to the key frame, and obtain a second number of video frames from other video frames to obtain each video segment. The corresponding first number of video frames and the second number of video frames, wherein the other video frames are the first number of adjacent key frames in each video segment except the key frames. video frames other than video frames; The key frames corresponding to each video segment, the first number of video frames, and the second number of video frames are determined as partial video frames corresponding to each video segment. 6. The method of claim 1, wherein dividing the video to be processed into multiple video segments includes: Based on the key frame, the video to be processed is divided into multiple video segments, wherein each video segment of the multiple video segments includes at least one frame of the key frame. 7. The method of claim 6, wherein dividing the video to be processed into multiple video segments based on the key frames includes: Based on the key frame, multiple target timestamps are marked on the video to be processed, wherein the video between two adjacent target timestamps constitutes a video segment, and the target timestamps are located between two adjacent target timestamps. between keyframes. 8. The method according to any one of claims 1 to 7, characterized in that generating first input information for inputting a large language model based on target prompt information and the partial video frame includes: Perform feature extraction on the partial video frames to obtain image features corresponding to the partial video frames; Pool the image features corresponding to the partial video frames, and splice the pooled image features to obtain the features to be input; The features to be input are spliced with the target prompt information to obtain first input information for inputting a large language model. 9. The method of claim 8, wherein the image features corresponding to each video frame in the partial video frames include global features and block features corresponding to multiple image blocks, and the partial video frames The corresponding image features are pooled, and the pooled image features are spliced to obtain the features to be input, including: Splicing the global features corresponding to the partial video frames according to the order of the video frames in the partial video frames to obtain the first feature; Perform average pooling on the block features corresponding to the partial video frames to obtain the second features; The first feature and the second feature are spliced to obtain the feature to be input. 10. The method according to any one of claims 1 to 7, characterized in that, before performing key frame identification on the video to be processed to obtain the key frames in the video to be processed, the method further includes: Obtain the video duration corresponding to the video to be processed; If the video duration is greater than the target duration, the step of identifying key frames in the video to be processed is performed to obtain the key frames in the video to be processed. 11. The method according to claim 10, characterized in that after obtaining the video duration corresponding to the video to be processed, the method further includes: If the video duration is less than or equal to the target duration, generate second input information for inputting a large language model based on the video frames in the video to be processed and the target prompt information; The second input information is input into the large language model to obtain a video understanding result corresponding to the video to be processed. 12. A video processing device, characterized in that the device includes: a key frame identification module, a video dividing module, a video frame extraction module, an information generation module and a video understanding module, wherein, The key frame identification module is used to identify key frames in the video to be processed and obtain the key frames in the video to be processed; The video dividing module is used to divide the video to be processed into multiple video segments; The video frame extraction module is configured to extract a partial video frame from each video segment of the plurality of video segments based on the key frame; The information generation module is configured to generate first input information for inputting a large language model based on target prompt information and the partial video frame, and the target prompt information is used to prompt the large language model to perform video understanding; The video understanding module is used to input the first input information into the large language model to obtain a video understanding result corresponding to the video to be processed. 13. A computer device, characterized by comprising: one or more processors; memory; one or more programs, wherein said one or more programs are stored in said memory and configured to be executed by said one or more processors, said one or more programs are configured to perform as claimed The method described in any one of 1-11. 14. A computer-readable storage medium, characterized in that program code is stored in the computer-readable storage medium, and the program code can be called and executed by a processor as described in any one of claims 1-11 Methods.