WO2025048288A1 - Apparatus and method for sequential semantic generation communication in communication system - Google Patents

Abstract

The present disclosure relates generally to a communication system and, more particularly, to an apparatus and a method for sequential semantic generation communication in a communication system. An operation method of a terminal in a communication system may comprise the steps of: receiving a first image; on the basis of a first model, generating a text prompt from the first image; evaluating the importance of a word included in the text prompt; and on the basis of the importance of the word, transmitting the word to another terminal.

Description

Device and method for sequential semantic generation communication in a communication system

The present disclosure relates generally to communication systems, and more specifically to devices and methods for sequential semantic generative communication in communication systems.

The recent development of artificial intelligence (AI) technology has had a significant impact on communication systems. In particular, AI technology is being actively utilized to solve problems such as limited bandwidth, various channel conditions, and various user requirements.

3GPP communication standards are also preparing for the next-generation 6G communication beyond 5G by applying deep learning technology to communication systems. The advancement of these technologies requires rapid data exchange and information conversion in various industrial fields such as autonomous vehicles, mobile edge computing, and robotics.

In particular, generative models such as DALL-E, CLIP, and BLIP have recently achieved great results in the task of converting texts and images. These multimodal generative models can understand and generate common meanings between two different data domains, namely texts and images. This plays an important role in utilizing the ability to interpret and explain data of various modalities in communication systems and to generate interactive cognitive meanings.

Based on the above discussion, the present disclosure provides a device and method for sequential semantic generation communication in a communication system.

In addition, the present disclosure provides a device and method for maximizing communication efficiency through conversion between text and images by utilizing a multimodal generative model in a communication system.

In addition, the present disclosure provides a device and method for reducing communication load by conveying the meaning of image data using text in a communication system.

In addition, the present disclosure provides a device and method for fast and efficient communication by sequentially conveying the meaning of an image using text in a communication system.

In addition, the present disclosure provides a device and method for optimizing a transmission order according to the information importance of each word of a text in a communication system.

According to various embodiments of the present disclosure, a method of operating a terminal in a communication system may include a process of receiving a first image, a process of generating a text prompt based on a first model from the first image, a process of evaluating the importance of a word included in the text prompt, and a process of transmitting the word to another terminal based on the importance of the word.

According to various embodiments of the present disclosure, in a communication system, a terminal includes a transceiver and a control unit operably connected to the transceiver, wherein the control unit can receive a first image, generate a text prompt based on a first model from the first image, evaluate the importance of a word included in the text prompt, and transmit the word to another terminal based on the importance of the word.

Devices and methods according to various embodiments of the present disclosure enable fast and efficient data transmission while maintaining high similarity to the original image by reducing communication load and minimizing information loss through efficient conversion between text and images using a multimodal generative model.

In addition, the devices and methods according to various embodiments of the present disclosure can significantly improve data transmission efficiency and quality in various environments and conditions by optimizing the transmission order according to the information importance of each word through a sequential meaning transmission method, thereby reaching the target similarity level with a minimum number of transmission steps.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

FIG. 1 illustrates a schematic diagram for sequential semantic communication according to one embodiment of the present disclosure.

FIG. 2 illustrates an example of a word transmission method in sequential semantic communication according to one embodiment of the present disclosure.

FIG. 3 illustrates simulation results according to transmission methods according to various embodiments of the present disclosure.

FIG. 4 is a diagram showing a device configuration according to various embodiments of the present disclosure.

FIG. 5 illustrates an example applied to vehicle-to-vehicle communication according to one embodiment of the present disclosure.

The terms used in this disclosure are only used to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly indicates otherwise. The terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person having ordinary skill in the art described in this disclosure. Among the terms used in this disclosure, terms defined in general dictionaries may be interpreted as having the same or similar meaning as the meaning they have in the context of the related technology, and shall not be interpreted in an ideal or excessively formal meaning unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

The terms used in this disclosure are only used to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly indicates otherwise. The terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person having ordinary skill in the art described in this disclosure. Among the terms used in this disclosure, terms defined in general dictionaries may be interpreted as having the same or similar meaning as the meaning they have in the context of the related technology, and shall not be interpreted in an ideal or excessively formal meaning unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

In the various embodiments of the present disclosure described below, a hardware-based approach is described as an example. However, since the various embodiments of the present disclosure include techniques using both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

Additionally, in the detailed description and claims of this disclosure, “at least one of A, B, and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B, and C.” Additionally, “at least one of A, B, or C” or “at least one of A, B, and/or C” can mean “at least one of A, B, and C.”

The present disclosure relates to a device and method for sequential semantic generative communication in a communication system. Specifically, the present disclosure describes a technique for reducing communication load and minimizing information loss through efficient conversion between text and images using a multimodal generative model in a communication system, thereby quickly and efficiently transmitting data while maintaining high similarity to the original image.

The terms referring to signals, channels, control information, network entities, and components of devices used in the following description are examples for convenience of explanation. Therefore, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used.

In addition, although the present disclosure describes various embodiments using terms used in some communication standards (e.g., 3rd Generation Partnership Project (3GPP)), this is only an example for explanation. The various embodiments of the present disclosure can be easily modified and applied to other communication systems.

FIG. 1 illustrates a schematic diagram for sequential semantic communication according to one embodiment of the present disclosure.

Referring to FIG. 1, a transmitter for sequential semantic communication (hereinafter, transmitter) can convert an image into a text prompt (101). Specifically, the transmitter (e.g., Alice) can convert an original image into a text prompt using an image-to-text model ('Img2Txt Gen' model). For example, the transmitter can input an 'image of a cat running through a field of flowers' and generate a text prompt of 'A while cat running through a field of flowers'. The operation (101) can be an operation to convert a main visual element of an image into text, thereby reducing the size of the transmission data and compressing the meaning.

In one embodiment, the image-to-text model can be an image-based text decoder using a transformer-based text prompt generation technique.

The transmitter can prioritize words for sequential transmission (103). Specifically, words can be prioritized in order to sequentially transmit the words in the generated text prompt. For example, the transmitter can use the 'Txt2Img Gen' model and the LPIPS (learned perceptual image patch similarity) metric to evaluate how important each word is for image reconstruction.

In one embodiment, the LPIPS metric measures the visual similarity between the original image and the reconstructed image, so that words with high similarity can be transmitted first. For example, in the sentence 'A white cat running through a field of flowers', 'cat' is evaluated as the most important word and can be transmitted first. In operation (103), the transmitter can analyze the correlation between words using 'LM (language model)' and determine the order of words to be transmitted sequentially, starting from the word with the largest amount of information.

In one embodiment, the LPIPS metric represents values from 0 to 1 and can express visual similarity.

The transmitter can transmit words with high importance to the receiver according to the order of the determined words (105). For example, the word "cat" can be transmitted as the first word.

In one embodiment, each word is transmitted independently, which reduces the communication load and allows the image to be reconstructed incrementally for each action.

A receiver (a "transmitter for sequential semantic communication") can reconstruct an image based on the received word (107). Specifically, the receiver can generate an image based on the received word using the 'Txt2Img Gen' model. For example, after receiving the word 'cat', the receiver can reconstruct an image of a cat based on the word. In operation (107), the receiver can generate an image visually similar to the original image using the transmitted word.

According to the embodiment of Fig. 1, the present disclosure enables efficient communication between a transmitter and a receiver, and can reconstruct a high-quality image similar to an original image by compressively transmitting meaningful information through text. In addition, it can realize efficient data transmission while reducing the communication load and maintaining high similarity.

FIG. 2 illustrates an example of a word transmission method in sequential semantic communication according to an embodiment of the present disclosure. FIG. 2 presents, but is not limited to, a lowest LPIPS Transmission (201), a Most Attentive Transmission (202), and a Least Attentive Transmission (203) as examples.

In order for the transmitter to transmit the word that the receiver needs to generate as similarly as possible during the initial communication phase, the transmitter can assume that it knows information about the receiver's text-to-image conversion processor. Therefore, assuming that both the transmitter and the receiver cache and store the necessary models, the transmitter can store and cache the receiver's model. By doing so, the transmitter can predict in advance the image that Bob will generate based on the transmitted word.

In some embodiments, it may be appropriate for the transmitter to store only a certain amount of additional model parameters in addition to the required image-to-text transformation model. For example, the transmitter may cache or store all or only some of the receiver's parameters. Depending on these cases, the way the transmitter selects the order of words to transmit may vary.

If the transmitter caches (stores) the entire model of the receiver, the transmitter can fully utilize all the capabilities of the receiver. The transmitter can predict the generated image of the receiver with the text it has generated. Before the transmitter transmits a word, the transmitter can first predict Bob's expected result with all the words of the generated text prompt. That is, according to the lowest LPIPS transmission (201) method of FIG. 2, one embodiment of the present disclosure can be performed.

When the transmitter caches (stores) part of the receiver's entire model, the transmitter can store only the language model (LM) and the attention module to determine the importance of words and the order in which they are transmitted.

Specifically, the text-to-image model of the receiver may include text embedding (language model (LM)), image generation (U-Net), and post-processing (UAE) processes. The transmitter may cache only the LM among the models of the receiver to evaluate the importance of words, and the attention module may interpret the text and evaluate the association between words. That is, by using this method, an embodiment of the present disclosure may be performed according to at least one of the most attentive transmission (203) or the least attentive transmission (205) of FIG. 2.

According to one embodiment, the attention value is calculated as the product of the attention weight and the value, and the attention weight can be expressed in the form of a matrix representing the relationship between each word.

Referring to FIG. 2, the objective image is an image (205), which can be expressed as a generated text prompt as a box (207) ('a white cat running through a field of flowers').

For Lowest LPIPS Transmission (201), the transmitter can simulate the expected image reconstruction results when each word is transmitted using the entire model of the receiver. In one embodiment, the process of simulating the image reconstruction results can be a process of performing simulations for all candidate words.

Afterwards, after simulating all words, the word that generates the image most similar to the original image can be selected and transmitted sequentially. In other words, minimum LPIPS transmission can be transmitted by selecting the word that minimizes the LPIPS value.

Afterwards, the transmitter repeatedly transmits the words one by one, and the receiver that receives them repeatedly transmits the words one by one to reconstruct the final image.

With this, the minimum LPIPS transmission can ensure that the reconstructed image becomes progressively more similar to the original image as each word is transmitted.

For example, a minimal LPIPS transmission could be the sequential words 'cat', 'white', 'running', 'flower'. That is, by transmitting 'cat' as the first word, the main elements of the cat can be reconstructed. Then, by sequentially transmitting the words "white", "running", and "flower", the color, motion, and background elements of the cat can be progressively added.

Most attentive transmission (203) corresponds to the case where the transmitter can only cache a part of the receiver's text-to-image generation model, and in this case, LM is mainly used to analyze the association between words. Specifically, most attentive transmission can calculate the attention value between each word and select and transmit the word with the highest relevance. This can be a method to quickly convey the meaning by preferentially transmitting the word with the most important meaning in the sentence.

Therefore, most attentive transmission (203) can calculate the attention value between words using a language model, select the most important word first and transmit it, and then sequentially transmit the word most related to the transmitted word.

For example, the most attentive transmission (203) can transmit words in the order of 'cat', 'white', 'running', 'through', 'field', 'flowers', 'of', 'a'. That is, the most important word 'cat' is transmitted first to reconstruct the main elements of the cat, and then the words 'white' and 'running', which are most related to 'cat', are transmitted to add the color and movement of the cat. After that, the remaining words are sequentially transmitted in the order of their high relevance to the transmitted words so that background elements can be added.

Least attentive transmission (205) corresponds to the case where the transmitter can only cache a part of the receiver's text-to-image generation model, and in this case, LM is mainly used to analyze the association between words. Specifically, most attentive transmission calculates the attention value between each word, and transmits the most important word first, and then selects and transmits the word with the least correlation to the first word. This can be a way to transmit unimportant information at the beginning of the transmission by preferentially transmitting words with less important meanings in the sentence.

Therefore, most attentive transmission (205) can calculate the attention value between words using a language model, select the most important word first and transmit it, and then sequentially transmit the word with the least correlation to the transmitted word.

For example, the least careful transmission can transmit words in the order of 'cat', 'of', 'white', 'running', 'through', 'a', 'field', 'flower'. Specifically, by transmitting the most important word 'cat' as the first word, the main elements of the cat can be reconstructed, and then less important words such as 'of', 'white', etc. can be transmitted to slowly reconstruct the image.

FIG. 3 illustrates simulation results according to transmission methods according to various embodiments of the present disclosure.

Referring to Fig. 3, the lowest LPIPS transmission (301) has the lowest LPIPS value and can reach the target similarity level the fastest. The most attentive transmission (303) has a low LPIPS value, but may have a value slightly higher than the lowest LPIPS transmission method. The least attentive transmission (305) has the highest LPIPS value and may require many communication processes.

FIG. 4 is a diagram showing a device configuration according to various embodiments of the present disclosure. Referring to FIG. 4, the device of the present disclosure may include at least one processor (410), a memory (420), and a communication device (430) that is connected to a network and performs communication. In addition, the communication node (400) may further include an input interface device (440), an output interface device (450), a storage device (460), etc. Each component included in the device (400) may be connected by a bus (470) and communicate with each other.

However, each component included in the device (400) may be connected through an individual interface or individual bus centered around the processor (410), rather than a common bus (470). For example, the processor (410) may be connected to at least one of a memory (420), a communication device (430), an input interface device (440), an output interface device (450), and a storage device (460) through a dedicated interface.

The processor (410) can execute a program command stored in at least one of the memory (420) and the storage device (460). The processor (410) may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which methods according to embodiments of the present invention are performed. Each of the memory (420) and the storage device (460) may be configured with at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory (420) may be configured with at least one of a read only memory (ROM) and a random access memory (RAM).

FIG. 5 illustrates an example applied to vehicle-to-vehicle communication according to one embodiment of the present disclosure.

Referring to FIG. 5, vehicles (20, 10, 30) can transmit real-time data through communication with each other.

In the case of autonomous vehicles, the present disclosure requires rapid exchange of information on road conditions, traffic signals, obstacles, etc. At this time, a method of converting image data into text and transmitting it can be useful for reducing the amount of data transmitted and increasing communication efficiency. For example, an image describing a specific road condition (e.g., an accident, road construction, etc.) can be converted into text and transmitted. That is, a text such as "A car accident at the intersection of Main St. and 1 ^st Ave" can be transmitted by converting the image into text.

In addition, important information in inter-vehicle communication needs to be transmitted with priority. According to an embodiment of the present disclosure, the most important information can be transmitted first, and secondary information can be transmitted later. For example, "Accident Ahead" can be transmitted as the first text, and then "At the intersection of Main St. and 1 ^st Ave", "Two vehicles involved" can be transmitted as secondary information.

Additionally, in vehicle-to-vehicle communication, each vehicle may be of a different make and model, and may use different sensors and data formats. In such cases, using a common text-based semantic representation can improve interoperability.

In summary, the present disclosure can be applied to vehicle-to-vehicle communication, and can provide great advantages, especially in terms of data transmission efficiency, real-time information sharing, and improved interoperability. Applying this technology to vehicle-to-vehicle communication will greatly improve the safety and efficiency of autonomous vehicles.

The methods according to the embodiments described in the claims or specification of the disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

In the case of software implementation, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors in an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to the embodiments described in the claims or specification of the present disclosure.

These programs (software modules, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, they may be stored in a memory composed of a combination of some or all of these. In addition, each configuration memory may be included in multiple numbers.

Additionally, the program may be stored in an attachable storage device that is accessible via a communications network, such as the Internet, an Intranet, a local area network (LAN), a wide area network (WAN), or a storage area network (SAN), or a combination thereof. The storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communications network may be connected to a device performing an embodiment of the present disclosure.

In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in the singular or plural form depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for the convenience of explanation, and the present disclosure is not limited to the singular or plural components, and even if a component is expressed in the plural form, it may be composed of the singular form, or even if a component is expressed in the singular form, it may be composed of the plural form.

Meanwhile, although the detailed description of the present disclosure has described specific embodiments, it is obvious that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be determined not only by the scope of the claims described below, but also by equivalents of the scope of the claims.

Claims (10)

In the method of operating a terminal in a communication system, The process of receiving the first image, A process of generating a text prompt based on the first model from the first image above, The process of evaluating the importance of words included in the above text prompt, A method comprising the process of transmitting said word to another terminal based on the importance of said word. In claim 1, the second model is a model for predicting the first image by the word by the other terminal, The above second model is partially or completely stored in the terminal, In claim 2, A process of predicting a second image based on the second model stored in the terminal, and A process for identifying the similarity between the first image and the second image, The process of selecting the word with the highest similarity, A method comprising a process of sequentially transmitting the highest word to the other terminal. In claim 2, The process of predicting a second image by the word based on the second model stored in the terminal is as follows: The process of selecting and transmitting the most important word from the first image above, A method comprising the process of sequentially transmitting the most important words in order of high relevance to the above In claim 2, The process of predicting a second image by the word based on the second model stored in the terminal is as follows: The process of selecting and transmitting the most important word from the first image above, A method comprising the process of sequentially transmitting the most important words in order of decreasing relevance to the above words. In a terminal in a communication system, Transmitter and receiver, A control unit operably connected to the above transmitter and receiver, The above control unit, Receive the first image, Generate a text prompt based on the first model from the first image above, Evaluate the importance of words included in the above text prompt, A device that transmits said word to another terminal based on the importance of said word. In claim 6, the second model is a model for predicting the first image by the word by the other terminal, The above second model is a device, partly or wholly stored in the terminal. In claim 7, the control unit predicts the second image by the word based on the second model stored in the terminal. Identifying the similarity between the first image and the second image, Select the word with the highest similarity, A device that sequentially transmits the highest word to the other terminal. In claim 7, The above control unit, based on the second model stored in the terminal, predicts the second image by the word. Select the most important word from the first image above and send it, A device comprising a process of sequentially transmitting the most important words in order of high relevance to the above In claim 7, the control unit predicts the second image by the word based on the second model stored in the terminal. Select the most important word from the first image above and send it, A device that sequentially transmits the most important words and the words with the lowest relevance.

Images