JP7538574B1 - Video creation device, video creation method, video creation program, and video creation system - Google Patents

Abstract

The present invention provides a technique for generating video in a format desired by a user.
SOLUTION
The video generation device comprises an acquisition unit that acquires a video file, an analysis unit that acquires an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in chronological order, an editing plan unit that transmits the time-series speech information and command information that instructs an editing plan to be output for editing the time-series speech information in accordance with desired editing policy information to an interactive AI using a language model and receives the editing plan from the interactive AI, and a video editing unit that edits the video file in accordance with the editing plan and generates an edited video.
[Selected Figure] Figure 1

Description

The present invention relates to a video generation device, a video generation method, a video generation program, and a video generation system.

Patent document 1 describes a video editing device that includes an edited video generation unit that generates edited video data in accordance with a user's request, the edited video data having a length desired by the user and including the content of the portion that the user wishes to view.

JP 2021-087180 A

The above technology involves inputting index information to identify each section of video data divided into predetermined lengths, identifying the part the user wants to watch, and generating edited video data that includes that content, so the structure or appearance of the generated video is not necessarily taken into consideration.

The objective of the present invention is to generate videos in the format desired by the user.

The present application includes multiple means for solving at least part of the above problems, examples of which are as follows: A video generation device according to one aspect of the present invention is characterized by having an acquisition unit that acquires a video file, an analysis unit that acquires an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in a time series, an editing plan unit that transmits the time-series speech information and command information that instructs an editing plan to be output for editing the time-series speech information according to desired editing policy information to an interactive AI using a language model and receives the editing plan from the interactive AI, and a video editing unit that edits the video file according to the editing plan and generates an edited video.

In addition, in the above video generation device, the editing plan unit may include, in the command information, constraints on the edited video obtained by the editing plan.

In addition, in the video generation device described above, the editing plan unit may include, in the command information, configuration information about the edited video obtained by the editing plan.

In addition, in the video generation device described above, the editing plan unit may include in the command information a specification of a video, still image, or audio to be added to the edited video obtained by the editing plan.

In addition, in the video generation device described above, the editing plan unit may include in the command information a specification of visual effects to be used in the edited video obtained by the editing plan.

In addition, in the above video generation device, the editing plan may include information for constructing the edited video by connecting together partial videos whose start and end positions on a time axis within the video file are specified, and the video editing unit may generate the edited video by extracting the partial videos from the video file and connecting them together.

In the video generation device described above, the editing plan may include information for linking together partial videos, each of which has a specified start position and end position on a time axis within the video file, to create the edited video, and designation of video, still images, or audio to be added before and after the partial videos, and the video editing unit may generate the edited video by cutting out the partial videos from the video file, linking them together, and adding the video, still images, or audio to be added.

In the video generation device described above, the editing plan may include information for stitching together partial videos, each of which has a specified start position and end position on a time axis within the video file, to create the edited video, and a specification of visual effects to be used at the seams of the partial videos, and the video editing unit may generate the edited video by cutting out the partial videos from the video file, stitching them together, and applying the specified visual effects to the seams.

In addition, in the above video generation device, the analysis unit may fast-forward edit the speech included in the video file while maintaining the time series, and pass it to a predetermined speech-to-text conversion unit to obtain the time series speech information.

In the video generation device described above, the analysis unit may identify the speaker of the speech sound included in the video file, extract the speech sound for each speaker while maintaining the time series, and transfer the text information obtained to a predetermined speech-to-text conversion unit to integrate the text information to obtain the time series speech information.

In the above video generation device, the editing plan may be described in a predetermined format language, and the editing plan unit may include definition information about the format language that describes the editing plan in the command information.

In addition, a video generation method according to another aspect of the present invention is a video generation method using a video generation device, the video generation device having a processor, the processor is characterized by carrying out the following steps: an acquisition step of acquiring a video file; an analysis step of acquiring an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in a time series; an editing plan step of transmitting the time-series speech information and command information instructing an editing plan for editing the time-series speech information according to desired editing policy information to an interactive AI using a language model and receiving the editing plan from the interactive AI; and a video editing step of editing the video file according to the editing plan and generating an edited video.

In addition, a video generation program according to another aspect of the present invention is a video generation program that causes an information processing device to generate a video, the information processing device having a processor, and causes the processor to execute the following steps: an acquisition step of acquiring a video file; an analysis step of acquiring an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in a time series; an editing plan step of transmitting the time-series speech information and command information instructing the processor to output an editing plan for editing the time-series speech information according to desired editing policy information to an interactive AI using a language model and receiving the editing plan from the interactive AI; and a video editing step of editing the video file according to the editing plan and generating an edited video.

In addition, a video generation system according to another aspect of the present invention is a video generation system including a user terminal and a video generation device communicatively connected to the user terminal, the video generation device having an acquisition unit that acquires a video file from the user terminal via communication, an analysis unit that acquires an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in a time series, an editing plan unit that transmits the time-series speech information and command information that instructs the time-series speech information to be written and output as an editing plan for editing the time-series speech information according to desired editing policy information to an interactive AI using a language model and receives the editing plan from the interactive AI, and a video editing unit that edits the video file according to the editing plan and generates an edited video.

The present invention provides technology that allows users to generate videos in the format they desire.

Problems, configurations and advantages other than those mentioned above will become clear from the description of the embodiments below.

1 is a diagram showing an overview of a moving image generation system according to an embodiment. 1 is a configuration diagram of a video production system according to an embodiment. FIG. 4 is a diagram illustrating an example of a data structure of material information. FIG. 11 is a diagram illustrating an example of a data structure of time-series speech information. FIG. 11 is a diagram illustrating an example of a data structure of editing policy information. FIG. 13 illustrates an example of a data structure of command information. FIG. 13 is a diagram illustrating an example of a data structure of an editing plan. FIG. 2 is a diagram illustrating an example of a hardware configuration of a moving image generating device. FIG. 13 is a diagram illustrating an example of a video generation flow (video material registration). FIG. 13 is a diagram illustrating an example of a video generation flow (editing policy registration). FIG. 13 is a diagram showing an example of a video material registration screen. FIG. 13 is a diagram showing an example of a new material registration screen. FIG. 13 is a diagram showing an example of an editing policy registration screen. FIG. 13 is a diagram showing an example of a new editing policy registration screen.

A video generation system 1 to which an embodiment according to one aspect of the present invention is applied will be described below with reference to the drawings. In the following embodiment, when necessary for convenience, the description will be divided into multiple sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is a partial or complete modification, detail, supplementary explanation, etc. of the other.

In addition, in the following embodiments, when referring to the number of elements (including the number, numerical value, amount, range, etc.), unless otherwise specified or clearly limited in principle to a specific number, the number is not limited to that specific number and may be more than or less than the specific number.

Furthermore, it goes without saying that in the following embodiments, the components (including element steps, etc.) are not necessarily essential unless specifically stated otherwise or considered to be clearly essential in principle.

Similarly, in the following embodiments, when referring to the shapes, positional relationships, etc. of components, etc., it is intended to include shapes that are substantially similar or similar to those, unless otherwise specified or considered to be clearly different in principle. The same applies to the above numerical values and ranges.

In addition, in all drawings used to explain the embodiments, the same components are generally given the same reference numerals, and repeated explanations will be omitted.

In recent years, with the spread of networks and various electronic devices (personal computers, tablet devices, smartphones, etc.), an environment is being created in which videos can be created and published anywhere, anytime. For example, it is becoming possible for anyone to easily shoot videos using a smartphone or the like and post them anywhere, anytime to SNS (Social Networking Services) or video sharing sites that anyone can access. However, high-quality videos that attract attention are often the result of the time and effort of editors with specialized knowledge.

Therefore, in an embodiment of the present invention, a video creation system 1 is made available that accepts a video editing policy desired by a user and automatically creates a video in line with the policy. With video creation system 1, a video can be created in the format desired by the user even if the user does not have video editing skills or does not have the equipment and environment required for video creation.

Figure 1 is a diagram showing an overview of a video generation system according to this embodiment. In the video generation system 1, a user uses a user terminal 400 that the user uses, and a group of devices that are communicatively connected to the user terminal 400 via a communication path. The group of devices includes a video generation device 100, a group of devices that provide an interactive AI service 200, and a group of devices that provide a voice analysis service 300.

For example, the group of devices providing the interactive AI service 200, the group of devices providing the voice analysis service 300, and the video generating device 100 may be a cloud computer connected via the Internet, or a server device managed by the owner of the video generating device 100, the group of devices providing the interactive AI service 200, and the group of devices providing the voice analysis service 300. Furthermore, without being limited to this, a wearable device such as a user's smart watch may be used as the user terminal 400.

When the user terminal 400 and the device group (including the video generating device 100, the device group providing the interactive AI service 200, and the device group providing the voice analysis service 300) communicate with each other, they are connected via a communication path such as a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, a mobile phone network, or a short-range wireless communication such as Bluetooth (registered trademark), or a combination of these. The communication path 50 may be a VPN (Virtual Private Network) on a wireless communication network such as a mobile phone network.

By using the video creation system 1, a video can be created in the format desired by the user. Specifically, the user uses the user terminal 400 to register a video material file in which speech and environmental sounds are recorded and a video editing policy in the video creation system 1 (1 and 2), and then applies for video creation. The video creation device 100 extracts only the audio components from the material video and requests the audio analysis service 300 to analyze it as an audio file (3) for the material video. The audio analysis service 300 analyzes the audio file of the material video and returns an analyzed text file (4) that associates the timing of speech with spoken words and sentences to the video creation device 100.

The video production device 100 sends the analyzed text file obtained from the audio analysis service 300 and command information (5) including the video editing policy to the interactive AI service 200 to request the creation of an editing plan. At this time, the video production device 100 does not send actual video material files or audio files to the interactive AI service 200, but sends the analyzed text file. The interactive AI service 200 creates an editing plan that satisfies the constraints specified in the command information, the constraints on the edited video obtained by the editing plan, the configuration information, the designation of the video, still images or audio to be added, the designation of visual effects, etc., and returns it to the video production device 100 as an editing plan (6).

When the video generating device 100 receives the editing plan from the interactive AI service 200, it performs video editing processing according to the editing plan, creates an edited video (7), and provides it to the user terminal 400. This allows the user to utilize the provided edited video.

Figure 2 is a configuration diagram of a video generation system according to an embodiment. The video generation system 1 includes a video generation device 100, an interactive AI service 200 capable of communicating with the video generation device 100 via a communication path 50, a voice analysis service 300, and a user terminal 400.

The video generating device 100 includes a memory unit 110, a processing unit 120, an input/output unit 140, and a communication unit 150, which are connected to each other via a bus or the like so that they can communicate with each other.

The memory unit 110 includes material information 111, time-series speech information 112, editing policy information 113, command information 114, editing plan 115, and edited video 116.

Figure 3 is a diagram showing an example data structure of material information. Material information 111 stores information on multiple material videos to be used for generating videos. Material information 111 includes user 111A, video title 111B, video file path 111C, description 111D, analyzed flag 111E, and analysis result 111F.

User 111A is information that distinguishes a user from other users. Video title 111B is the title of the video to be registered as material. Video file path 111C is the storage location on the file system of the video to be registered as material, or a URI (Uniform Resource Identifier). Description 111D is information that explains in natural language the content of the video to be registered as material. Analyzed flag 111E is information that indicates whether or not analysis by the voice analysis service 300 has been completed. Analysis result 111F is analyzed text that is information on the results of analysis by the voice analysis service 300.

Figure 4 is a diagram showing an example data structure of time-series speech information. Time-series speech information 112 is information that stores the text of utterances made within a video in order, with the time that has elapsed within the video being a time series. Time-series speech information 112 includes utterance start time 112A, utterance end time 112B, and utterance text (words) 112C.

The speech start time 112A and the speech end time 112B are information that specifies the start and end timings of speech made within the video by the time that has elapsed since the start time of the video (time within the video). The speech text (words) 112C are words that are spoken between the speech start time 112A and the speech end time 112B. However, they are not limited to words, and may be sentences or clauses of a certain length.

Figure 5 is a diagram showing an example data structure of editing policy information. Editing policy information 113 is information on the editing policy of the video to be generated. Editing policy information 113 includes a title 113A, a content goal 113B, constraints 113C, a content structure 113D, a resource file 113E, and an editing plan format 113F.

Title 113A is the title of the editing policy or the title of the video to be generated. Content goal 113B is information such as the image that the video to be generated aims to achieve and the aim of psychological change in the viewer (to make the viewer feel happy or calm when watching). Constraints 113C is information on constraints for video creation such as the length (playback time) of the generated video. Content configuration 113D is information on the configuration of the video to be generated, for example, connecting three consecutive videos with a visual effect transition. Resource file 113E is information on video materials to be used for the video to be generated. Editing plan format 113F is information specifying the format of the editing plan for video generation. The format of the editing plan may be a known format or may be defined in an extension language conforming to SGML (Standard Generalized Markup Language) or the like.

Figure 6 is a diagram showing an example of the data structure of command information. Command information 114 is a command (prompt) for causing the interactive AI service 200 to perform processing. A command for generating a video according to this embodiment, for example, specifies editing policy information 113 and instructs the creation of an editing plan in accordance with the editing policy, and is written in natural language.

Figure 7 shows an example of the data structure of an editing plan. The editing plan 115 describes editing information in a specified format, such as by tagging components to be assigned to times within the video to be generated, to create planning information for video production.

The format of the editing plan according to this embodiment will be outlined below. First, the editing plan can include three main types of elements: "shot", "view", and "attach". The "shot" tag is a collection of multiple "views". "View" specifies the material file and related information. Material files include videos (including the start and end times of the parts to be used in the video material) and images (magnification rate and on-screen layout of the image file), and related information includes color and gradation designations. "Attach" specifies an element to be displayed by adding it to the material specified by "view" (if it is an image, this includes the size, layout, and start and end times in the video to be generated. If it is audio, this includes the audio volume and start and end times in the video to be generated).

For example, cut editing information such as allocating an excerpt from a source video extracted by specifying a time within the source video to one of the components ("views"), playing multiple such excerpts continuously with transitions in between, and then attaching a time to display a QR code (registered trademark) for accessing other videos in the channel ("attaches") is described.

For example, the editing plan may include, as an editing plan, information for constructing an edited video by joining partial videos with specified start and end positions on the elapsed time axis in the video file to be used as the material. The editing plan may also include, as an editing plan, information for constructing an edited video by joining partial videos with specified start and end positions on the elapsed time axis in the video file to be used as the material, and specification of videos, still images, or audio to be added before and after the partial videos. The editing plan may also include, as an editing plan, information for constructing an edited video by joining partial videos with specified start and end positions on the elapsed time axis in the video file to be used as the material, and specification of visual effects to be used at the joins of the partial videos.

Returning to the explanation of FIG. 2, the processing unit 120 includes an acquisition unit 121, an analysis unit 122, an editing planning unit 123, and a video editing unit 124.

The acquisition unit 121 acquires a video file. The analysis unit 122 acquires an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in time series, from the voice analysis service. The analysis unit 122 may also fast-forward edit the speech included in the video file while maintaining the time series, and pass it on to a specified voice-to-text conversion unit (voice analysis service 300) to obtain time-series speech information. Alternatively, the analysis unit 122 may identify the speaker of the speech included in the video file, extract it while maintaining the time series for each speaker, and integrate the text information obtained by passing it on to a specified voice-to-text conversion unit (voice analysis service 300) to obtain time-series speech information.

The editing plan unit 123 transmits the time-series speech information and command information instructing the output of an editing plan for editing the time-series speech information according to desired editing policy information to an interactive AI (interactive AI service 200) using a language model, and receives an editing plan from the interactive AI. The editing plan unit may also include, in the command information, constraint conditions for the edited video obtained by the editing plan. The editing plan unit 123 may also include, in the command information, configuration information for the edited video obtained by the editing plan. The editing plan unit 123 may also include, in the command information, a designation of a video, a still image, or a sound to be added to the edited video obtained by the editing plan. The editing plan unit 123 may also include, in the command information, a designation of a visual effect to be used in the edited video obtained by the editing plan. The editing plan unit 123 may also include, in the command information, definition information for a format language for describing the editing plan.

The video editing unit 124 edits the video file in accordance with the editing plan included in the editing plan to generate an edited video. Specifically, the video editing unit 124 generates the edited video by cutting out partial videos from video material files and splicing them together. The video editing unit 124 may also generate the edited video by adding videos, still images, or audio to be added. The video editing unit 124 may also generate the edited video by cutting out partial videos from video files and splicing them together, and applying a specified visual effect to the splices.

The input/output unit 140 controls input/output to/from the video generating device 100. For example, the input/output unit 140 accepts various types of input, such as various contact inputs, such as typing, touch, and flick input, or various types of inputs, such as gaze input. The input/output unit 140 also performs output to the user. The information that is output is various types of output information, such as a screen, presentation information, advertisements, and videos.

The communication unit 150 communicates via the communication path 50 with a group of devices that provide the interactive AI service 200, a group of devices that provide the voice analysis service 300, a user terminal 400, and other terminals that communicate via the Internet.

The interactive AI service 200 is a service that provides the functions of so-called generation AI, such as GPT and Gemini, via an API (Application Programming Interface) or the like. The interactive AI service 200 gives instructions (prompts) in natural language to the generation AI to generate the desired results. In this embodiment, the generation AI is made to generate an editing plan for generating a video.

The voice analysis service 300 performs voice analysis using known technology such as the Google TTS API. When the voice analysis service 300 receives an audio file, it transcribes the speech in the audio file into text, and for each utterance contained in the audio file, it outputs the text of the utterance and information specifying the start and end times of the utterance as analyzed text.

The user terminal 400 is a terminal used by the user. The user terminal 400 may be a smartphone terminal of the user, a PC (Personal Computer), or the like. Furthermore, without being limited to this, a wearable device such as a smart watch of the user may be used as the user terminal 400.

Figure 8 is a diagram showing an example of the hardware configuration of a video generation device. The video generation device 100 has a hardware configuration realized by the housing of a so-called server device, workstation, personal computer, smartphone, or tablet terminal. The video generation device 100 has a processor 101, memory 102, storage 103, input device 104, display device 105, communication device 106, and a bus connecting each device.

The processor 101 is a computing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

Memory 102 is a memory device such as a RAM (Random Access Memory).

Storage 103 is a non-volatile storage device capable of storing digital information, such as a hard disk drive, a solid state drive (SSD), or a flash memory.

The input device 104 is a device that accepts input from one or more of a keyboard, mouse, touch panel, and microphone. The display device 105 is a device that displays one or more of various output devices such as an organic EL (Electro-Luminescence) display.

The communication device 106 is a network interface card (NIC) or the like that communicates with other devices via a network.

The device providing the interactive AI service 200, the device providing the voice analysis service 300, and the user terminal 400 also have substantially the same hardware configuration as the video generation device 100.

The processing unit 120, acquisition unit 121, analysis unit 122, editing planning unit 123, and video editing unit 124 of the video generating device 100 described above are realized by a program that causes the processor 101 to perform processing. This program is stored in the memory 102, storage 103, or a ROM device (not shown), and is loaded onto the memory 102 for execution and executed by the processor 101.

The memory unit 110 of the video generating device 100 is realized by the memory 102 and the storage 103. The input/output unit 140 is realized by the input device 104 and the display device 105. The communication unit 150 is realized by the communication device 106. The above is an example of the hardware configuration of the video generating device 100.

The configuration of the video generating device 100 can be further divided into more components depending on the processing content. Also, a single component can be divided into more processes.

In addition, each processing unit (processing unit 120, acquisition unit 121, analysis unit 122, editing planning unit 123, and video editing unit 124) may be constructed using dedicated hardware (ASIC, GPU, etc.) that realizes each function. In addition, the processing of each processing unit may be executed by a single piece of hardware, or may be executed by multiple pieces of hardware.

Next, the operation of the video generation system 1 in this embodiment will be described.

Figure 9 shows an example of a video generation flow (video material registration). The video generation flow (video material registration) starts when a user requests the start of the flow in the web browser or application software (hereinafter sometimes simply referred to as the browser) of the user terminal 400.

The acquisition unit 121 of the video production device 100 generates a video material registration screen and displays it on the user terminal 400 (step S001). Specifically, the acquisition unit 121 generates a video material registration screen that allows the user to manage a list of videos that have been previously registered. The acquisition unit 121 then transmits display information for the generated video material registration screen to the user terminal 400.

Then, the browser of the user terminal 400 displays a video material registration screen and sends a video material registration request to the video generating device 100 together with information including the video material file to be registered, the video title, and descriptive information (step S002).

The acquisition unit 121 acquires video material files, etc. (step S003). Specifically, the acquisition unit 121 registers the user, video title, video file, and description in the material information 111.

Then, the analysis unit 122 performs video analysis (audio portion extraction) (step S004). Specifically, the analysis unit 122 separates and acquires audio components from the acquired video file.

Then, the analysis unit 122 performs video analysis (fast-forward audio generation) (step S005). Specifically, the analysis unit 122 fast-forward edits the audio components contained in the acquired video file while maintaining the chronological order. For example, when processing a video file of an utterance from 0 minutes 15 seconds when the utterance starts to 0 minutes 27 seconds when the utterance ends (the utterance lasts 12 seconds), the analysis unit 122 edits the video file at 4 times the normal speed, reducing the amount of data in the audio file so that the time from the start of the utterance to the end of the utterance is 3 seconds.

Then, the analysis unit 122 performs video analysis (request for audio analysis) (step S006). Specifically, the analysis unit 122 requests the audio analysis service 300 to analyze the audio file of the fast-forwarded audio created in step S005 by sending it via an API or the like.

The audio analysis service 300 performs audio analysis processing on the transmitted audio file of the fast-forwarded audio (step S007). Specifically, the audio analysis service 300 generates a speech timing analyzed text of the raw video in which the speech timing and speech content of the raw video are associated and recorded, and transmits the text to the video production device 100.

Then, the analysis unit 122 performs video analysis (creation of time-series information) (step S007). Specifically, the analysis unit 122 stores the received analyzed text in the time-series speech information 112, sets the analyzed flag 111E to "Done", and stores reference information to the time-series speech information 112 in the analysis result 111F. At that time, if the data structures of the analyzed text and the time-series speech information 112 are different, the analysis unit 122 may convert the information of the analyzed text so that the time information is returned from the fast-forward state to the normal speed state and store it as the time-series speech information 112, or may convert the time information so that the time information is returned from the fast-forward state to the normal speed state and then convert it into the data structure of the time-series speech information 112 and store it.

The above is an example of the video generation flow (video material registration). According to the video generation flow (video material registration), for videos registered as video material, it is possible to obtain text information of utterances and time-series utterance information that analyzes the timing of the utterances in the video.

Figure 10 is a diagram showing an example of a video creation flow (editing policy registration). The video creation flow (editing policy registration) starts when a user requests the start of the flow in the browser of the user terminal 400.

The editing planning unit 123 of the video generating device 100 generates an editing policy registration screen and displays it on the user terminal 400 (step S101). Specifically, the editing planning unit 123 generates an editing policy registration screen that manages a list of editing policies that the user has previously registered. Then, the editing planning unit 123 transmits display information for the generated editing policy registration screen to the user terminal 400.

Then, the browser of the user terminal 400 displays an editing policy registration screen and transmits an editing policy registration request to the video production device 100 together with information including the editing policy title to be registered, the video material to be registered, and the order (step S102).

The editing planning unit 123 accepts the editing policy and the like (step S103). Specifically, the editing planning unit 123 registers, in the editing policy information 113, the editing policy title, the content goal, constraint conditions, content structure, and editing plan format based on the order, and resource files based on the registered video material. The editing planning unit 123 interprets the natural language written in the order to identify the content goal, constraint conditions, content structure, and editing plan format included in the order.

Then, the editing planning unit 123 performs editing preparation (creating command information) (step S104). Specifically, the editing planning unit 123 creates command information 114. For example, the editing planning unit 123 replaces the specified portion of the editing policy data in the command information 114 described above with the contents of the editing policy information 113, and generates a prompt to be passed to the interactive AI service 200.

Then, the editing planning unit 123 performs editing preparation (planning request) (step S105). Specifically, the editing planning unit 123 transmits the command information 114 created in step S104 and the speech timing analyzed text of the raw video to the interactive AI service 200 via an API or the like.

Then, the interactive AI service 200 performs editing planning processing according to the transmitted command information (step S106). Specifically, the interactive AI service 200 uses the speech timing analyzed text of the raw video to perform cut editing, focusing on important parts and comments evaluated as interesting or interesting, taking into consideration the speech content (meaning) and speech timing, and creates a plan to edit the video to meet the specified length by incorporating transitions and attachments according to the order. The interactive AI service 200 generates the planned editing content in a specified format as an editing plan and transmits it to the video generation device 100.

Then, the video editing unit 124 performs video editing (edited video creation) according to the editing plan 115 (step S107). Specifically, when the video editing unit 124 receives the transmitted editing plan, it stores it in the editing plan 115 in the storage unit 110. Then, the video editing unit 124 performs video editing (edited video creation) according to the editing plan 115, stores the edited video obtained as a result of the video editing in the edited video 116 in the storage unit 110, and transmits it to the user terminal 400. Note that the video editing unit 124 may post the edited video obtained as a result of the video editing on a website so that it can be downloaded and transmit a link to the user terminal 400, or may upload the edited video obtained as a result of the video editing to a video sharing site designated in advance from the user terminal 400.

The above is an example of the video generation flow (editing policy registration). According to the video generation flow (editing policy registration), the video material can be edited to obtain an edited video according to the time-series speech information obtained by analyzing the video registered as video material and the editing plan created using the editing policy. Therefore, it can be said that a video in the format desired by the user can be generated.

Figure 11 is a diagram showing an example of a video material registration screen. The example video material registration screen 600 displays information including at least a video title 611 and explanatory information 615 for each registered video material file 610. In addition, the example video material registration screen 600 includes an editing policy display button 601 that accepts an instruction to transition to an editing policy list screen, a new registration button 602 that accepts an instruction to newly register a video material, and, for each registered video material file 610, a video file name 612, a content analysis status 613, and a delete button 614 that cancels the registration of the video material.

The content analysis status 613 is information indicating whether or not time-series speech information obtained by analyzing the speech text information and the timing of the speech in the video for the registered video material has been obtained. When an input is received from the editing policy display button 601, the screen transitions to an example of an editing policy registration screen, which will be described later. When an input is received from the new registration button 602, the screen transitions to an example of a new material registration screen, which will be described later.

Figure 12 is a diagram showing an example of a new material registration screen. The example new material registration screen 650 includes at least a video title 651, a video file name 652, a reference button 653 for referencing and inputting a file path indicating the storage location of the video file specified by the video file name 652, a material file description input field 654, a close button 655 for receiving an instruction to transition to the video material registration screen, and a registration button 656 for receiving an instruction to register the video material.

The material file description input field 654 accepts a description of the content of the material in free text. For example, in the case of video material, the material file description input field 654 accepts a synopsis and a description of the scene for each time in the video. When the registration button 656 accepts an instruction to register video material, it carries out the registration process of step S003 of the video generation flow (video material registration).

Figure 13 is a diagram showing an example of an editing policy registration screen. The example 700 of the editing policy registration screen displays at least, for each registered editing policy 710, an editing policy name 711, a delete button 712 for canceling the registration of the editing policy, an order 713 that is the specific content of the editing policy, an editing plan creation button 714 for receiving an instruction to create an editing plan, information 715 explaining the plot of the video to be created according to the editing plan, and a video creation button 716 for receiving an instruction to generate an edited video according to the editing plan.

Order 713 is text information that describes the editing policy (including constraints and composition conditions) in natural language. For example, order 713 may include restrictions or guidelines for the length of the video to be generated, designation of video, still images, and audio to be added to the edited video, or designation of visual effects to be used in the edited video.

When input is received by the edit plan creation button 714, it accepts the input as an instruction to create an edit plan, and executes steps S104 to S107 of the video creation flow (editing policy registration). Synopsis 715 displays the synopsis of the edited video indicated by the edit plan (e.g., chapter structure, video playback time, etc.). When input is received by the video creation button 716, it accepts the input as an instruction to create a video in accordance with the created edit plan, and executes step S107 of the video creation flow (editing policy registration).

The example editing policy registration screen 700 also includes a registered video material list display button 701 and a new registration button 702. When input is received from the registered video material list display button 701, the screen transitions to the example video material registration screen 600. When input is received from the new registration button 702, the screen transitions to the example new editing policy registration screen described below.

Figure 14 is a diagram showing an example of a new editing policy registration screen. The example 750 of the new editing policy registration screen includes at least an editing policy name 751 for the editing policy to be registered by the user, a video file name 752 of the video material to be edited, a reference button 753 for referencing and inputting a file path indicating the storage location of the video file specified by the video file name 752, an order input field 754 for receiving specific details of the editing policy, a close button 755 for receiving an instruction to transition to the example 700 of the editing policy registration screen, and a register button 756 for receiving an instruction to register the editing policy.

The order input field 754 accepts free text instructions (additional information to the prompt) regarding the content of the editing policy. Specifically, the order input field 754 accepts restrictions or guidelines for the length of the video to be generated, designation of video, still images, and audio to be added to the edited video, or designation of visual effects to be used in the edited video. For example, the order input field 754 accepts free text instructions as the content of the editing policy, such as "It will consist of three scenes, and visual effects will be added to the transitions between each scene to avoid sudden changes in subject and brightness. The background music will be a bright song, and there will be 10 seconds at the end of the video to display a QR code. The entire video should be within 5 minutes."

When the registration button 656 receives an instruction to register an editing policy, it performs the registration process of step S103 of the video generation flow (editing policy registration).

The above is an explanation of the video creation system 1 as one embodiment of the present invention. As in the above embodiment, the video creation system 1 allows a user to create a video in the format desired by the user, even if the user does not have video editing skills or does not have the equipment and environment required for video creation.

The present invention is not limited to the above-described embodiment. The above-described embodiment can be modified in various ways within the scope of the technical concept of the present invention. For example, in the above-described embodiment, the video generating device 100 obtains a video editing plan using the interactive AI service 200, but this is not limited thereto. For example, the video generating device 100 itself may operate a generation AI specialized for video generation to generate an edited video.

Alternatively, in the above embodiment, the video production device 100 uses the audio analysis service 300 to analyze the speech in the raw video, but this is not limited thereto. For example, the video production device 100 itself may operate a generation AI specialized in audio analysis to generate time-series speech information.

Furthermore, the functions of the video generating device 100 may be realized by a cloud service consisting of one or more computers.

In addition, the technical elements of the above-mentioned embodiments may be applied independently, or may be divided into multiple parts, such as program parts and hardware parts, and then applied.

The present invention has been described above, focusing on the embodiments.

1: Video generation system, 50: Communication path, 100: Video generation device, 110: Memory unit, 111: Material information, 112: Time-series speech information, 113: Editing policy information, 114: Command information, 115: Editing plan, 116: Edited video, 120: Processing unit, 121: Acquisition unit, 122: Analysis unit, 123: Editing plan unit, 124: Video editing unit, 140: Input/output unit, 150: Communication unit, 200: Interactive AI service, 300: Voice analysis service, 400: User terminal.

Claims (14)

An acquisition unit that acquires a video file;
an analysis unit that acquires an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in a time series;
An editing plan unit that transmits the time-series speech information and command information instructing the user to output an editing plan for editing the time-series speech information according to desired editing policy information to an interactive AI using a language model, and receives the editing plan from the interactive AI;
a video editing unit that edits the video file in accordance with the editing plan and generates an edited video;
A video generating device having the above configuration. The video generating device according to claim 1 ,
The editing plan unit includes, in the command information, a constraint condition for the edited video obtained according to the editing plan.
A moving image generating device comprising: The video generating device according to claim 1 ,
The editing plan unit includes, in the command information, configuration information about the edited video obtained according to the editing plan.
A moving image generating device comprising: The video generating device according to claim 1 ,
The editing planning unit includes, in the command information, a designation of a video, a still image, or a sound to be added to the edited video obtained according to the editing plan.
A moving image generating device comprising: The video generating device according to claim 1 ,
The editing plan unit includes, in the command information, a designation of a visual effect to be used in the edited video obtained according to the editing plan.
A moving image generating device comprising: The video generating device according to claim 1 ,
The editing plan includes information for constructing the edited video by connecting partial videos each having a start position and an end position on a time axis in the video file,
the video editing unit generates the edited video by extracting the partial video from the video file and connecting the extracted partial video.
A moving image generating device comprising: The video generating device according to claim 1 ,
The editing plan includes information for linking together partial videos, each of which has a start position and an end position on a time axis in the video file, to form the edited video, and information for specifying videos, still images, or audio to be added before and after the partial videos;
the video editing unit extracts the partial video from the video file, joins them together, and generates the edited video by adding the video, still image, or audio to be added;
A moving image generating device comprising: The video generating device according to claim 1 ,
The editing plan includes information for constructing the edited video by joining together partial videos, each of which has a start position and an end position on a time axis in the video file, and a designation of visual effects to be used at the joins between the partial videos;
the video editing unit extracts the partial videos from the video file, joins them together, and applies the designated visual effect to the joins to generate the edited video;
A moving image generating device comprising: The video generating device according to claim 1 ,
The analysis unit performs fast-forward editing of the speech included in the video file while maintaining the time series, and transfers the speech to a predetermined speech-to-text conversion unit to obtain the time-series speech information.
A moving image generating device comprising: The video generating device according to claim 1 ,
The analysis unit identifies speakers of speech sounds included in the video file, extracts the speech sounds while maintaining a time series for each speaker, and transfers the speech sounds to a predetermined speech-to-text conversion unit to obtain the time-series speech information by integrating the obtained text information.
A moving image generating device comprising: The video generating device according to claim 1 ,
The editing plan is described in a predetermined format language;
The editing plan unit includes, in the command information, definition information about the format language in which the editing plan is described.
A moving image generating device comprising: A moving image generating method using a moving image generating device, comprising:
The video production device includes a processor,
The processor,
An acquisition step of acquiring a video file;
An analysis step of acquiring an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in a time series;
An editing planning step of transmitting the time-series speech information and command information instructing the user to output an editing plan for editing the time-series speech information according to desired editing policy information to an interactive AI using a language model, and receiving the editing plan from the interactive AI;
a video editing step of editing the video file in accordance with the editing plan to generate an edited video;
A video generation method that implements the above. A moving image generating program for causing an information processing device to generate a moving image,
The information processing device includes a processor,
The processor,
An acquisition step of acquiring a video file;
An analysis step of acquiring an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in a time series;
An editing planning step of transmitting the time-series speech information and command information instructing the user to output an editing plan for editing the time-series speech information according to desired editing policy information to an interactive AI using a language model, and receiving the editing plan from the interactive AI;
a video editing step of editing the video file in accordance with the editing plan to generate an edited video;
A video generation program that performs the above. A video production system including a user terminal and a video production device communicably connected to the user terminal,
The video generating device comprises:
An acquisition unit that acquires a video file from the user terminal via communication;
an analysis unit that acquires an analysis result including time-series speech information, which is text information obtained by transcribing speech included in the video file in a time series;
An editing plan unit that transmits the time-series speech information and command information instructing the user to output an editing plan for editing the time-series speech information according to desired editing policy information to an interactive AI using a language model, and receives the editing plan from the interactive AI;
a video editing unit that edits the video file in accordance with the editing plan and generates an edited video;
A video generation system comprising:

Images