WO2024078018A1 - Data transmission method, apparatus and system - Google Patents

Abstract

The present application belongs to the technical field of networks. Disclosed are a data transmission method, apparatus and system. For an original message, the sender of which is an SFU server, after a first node or an intermediate node that supports data deduplication on a transmission path of the original message acquires the original message, the first node or the intermediate node determines whether same has sent, to the next hop of the original message, a historical message, the load portion of which has duplicate content with a load portion of the original message; and if the first node or the intermediate node has sent, to the next hop of the original message, a historical message, the load portion of which has duplicate content with the load portion of the original message, the node performs deduplication processing on the load portion of the original message, so as to obtain a deduplicated message, and sends the deduplicated message to the next hop of the original message. Since the data volume of a deduplicated message is smaller than the data volume of the original message, the data volume of message transmission can be reduced, thereby reducing the overhead of a network bandwidth.

Description

Data transmission method, device and system

This application claims priority to Chinese patent application No. 202211233819.3, filed on October 10, 2022, with invention name “Traffic grouping and deduplication technology based on SFU architecture”, and priority to Chinese patent application No. 202211643989.9, filed on December 20, 2022, with invention name “Data transmission method, device and system”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of network technology, and in particular to a data transmission method, device and system.

Background technique

In communication scenarios such as video conferencing or live broadcasting, the audio and video data of the same user usually needs to be distributed to multiple recipients, which results in a large amount of data transmission and thus a large network overhead.

Summary of the invention

The present application provides a data transmission method, device and system, which can reduce network bandwidth overhead by reducing the amount of data transmitted by messages.

In a first aspect, a data transmission method is provided. The method includes: a first node obtains a first message whose sender is a selective forwarding unit (SFU) server, and the first node is the first node that supports data deduplication on the transmission path of the first message. If a target message exists in the historical message sent by the first node to the second node, and the payload part of the target message has repeated content with the payload part of the first message, the first node performs deduplication processing on the payload part of the first message to obtain a second message. The second message does not include repeated content, and the second message carries a deduplication mark and indication information of the repeated content. The deduplication mark is used to indicate that the second message is a deduplication message. The second node is the next hop of the first message on the first node. The first node sends the second message to the second node.

In the present application, after receiving a message whose sender is an SFU server, the first node can determine whether to send a historical message whose payload part has duplicate content with the payload part of the message to the next hop of the message. If the first node sends a historical message whose payload part has duplicate content with the payload part of the message to the next hop of the message, the first node can deduplicate the data of the message and then send the deduplicated message to the subordinate node. Since the data volume of the deduplicated message is smaller than the data volume of the non-deduplicated message, the data volume of the message transmission can be reduced, thereby reducing the network bandwidth overhead.

Optionally, the repeated content includes one or more repeated data blocks, and the indication information of the repeated content includes one or more indications. The one or more indications correspond one-to-one to the one or more repeated data blocks in the repeated content. Each indication is used to indicate the hash value of the corresponding repeated data block.

In one possible implementation, a data set is stored in a first node, and the data set includes the payload part of a historical message sent by the first node to a second node. The first node matches the content of the payload part of the first message with the payload part in the data set. If there is a target payload part in the data set that has a duplicate data block with the payload part of the first message, the first node determines that the target message exists in the historical message. Accordingly, the implementation process of the first node performing deduplication processing on the payload part of the first message includes: for each duplicate data block between the payload part of the first message and the target payload part, the first node calculates the hash value of the duplicate data block. The first node removes the duplicate data block in the payload part of the first message, and adds an indication corresponding to the duplicate data block to the payload part of the first message, where the indication is used to indicate the hash value of the duplicate data block and the position of the duplicate data block in the payload part of the first message.

In the present application, the first node can perform content matching on the payload part of the acquired message with the payload part of the stored historical message. If the payload part of the message and the payload part of the historical message have duplicate data blocks, the first node calculates the hash value of the duplicate data block, removes the duplicate data block in the message to obtain a deduplicated message, and further carries the hash value of the duplicate data block and an indication of the location of the duplicate data block in the deduplicated message to achieve data deduplication of the message.

Optionally, if there is no payload portion having a duplicate data block with the payload portion of the first message in the data set, the first node determines that the target message does not exist in the historical message. The first node adds the payload portion of the first message to the data set. The updated data set can be used by the first node to perform deduplication processing on messages obtained subsequently whose sender is the SFU server.

In another possible implementation, the payload portion includes a protocol portion and a data portion, and the one or more repeated data blocks are located in the data portion of the first message.

Optionally, a sampling tag set is stored in the first node, and the sampling tag set includes a hash value of a historical data block, and the historical data block is a data block obtained by sampling a preset position of a data part of a historical message sent by the first node to the second node. The first node samples a preset position of the data part of the first message to obtain a sampled data block. The first node calculates the hash value of the sampled data block. If the sampling tag set includes the hash value of the sampled data block, the first node determines that the target message exists in the historical message. Accordingly, the implementation process of the first node performing deduplication processing on the payload part of the first message includes: the first node uses the sampled data block whose hash value belongs to the sampling tag set as a duplicate data block, removes the duplicate data block from the data part of the first message, and adds an indication corresponding to the duplicate data block to the payload part of the first message, and the indication is used to indicate the hash value of the duplicate data block.

In the present application, the first node can calculate the hash value of the sampled data block at the preset position of the data part of the acquired message, and compare it with the hash value of the stored historical data block. If the hash value of a sampled data block in the message is the same as the hash value stored by the first node, the first node removes the sampled data block in the message to obtain a deduplicated message, and further carries the hash value of the sampled data block in the deduplicated message to achieve data deduplication of the message.

Optionally, there are multiple preset positions, and the first node obtains multiple sampled data blocks by sampling the preset positions of the data part of the first message. The indication corresponding to the repeated data block is also used to indicate the position of the repeated data block in the data part of the first message.

In the present application, when there are multiple sampling positions of the pre-set data part, the first node samples the data part of the message and obtains multiple sampled data blocks. In this case, it is necessary to indicate the position of the duplicate data blocks removed from the deduplicated message in the original message so that subsequent nodes can restore the data of the deduplicated message.

Optionally, if the sampling tag set does not include the hash value of the sampled data block, the first node determines that the target message does not exist in the historical message. The first node adds the hash value of the sampled data block to the sampling tag set. The updated sampling tag set can be used by the first node to perform deduplication processing on the message whose sender is the SFU server that is subsequently obtained.

Optionally, the sampling tag set also includes a historical data block indicated by a hash value. If the sampling tag set includes the hash value of the sampling data block, the first node determines that the target message exists in the historical message, including: if the sampling tag set includes the hash value of the sampling data block, the first node performs content matching on the sampling data block and the historical data block indicated by the hash value of the sampling data block. When the content of the sampling data block is the same as the content of the historical data block indicated by the hash value of the sampling data block, the first node determines that the target message exists in the historical message.

Since the data contents of two data blocks with the same hash value may be different, by storing the correspondence between historical data blocks and hash values of historical data blocks in a sampling tag set, the first node can further perform content matching on the sampled data block and the historical data block after determining that the hash value of the sampled data block of the message is the same as the hash value of a historical data block, so as to achieve an exact match, thereby improving the accuracy of deduplication of the message.

Optionally, the first node also stores the protocol part of the historical message sent by the first node to the second node. The above-mentioned repeated content also includes the protocol information located in the protocol part of the first message, and the indication information of the repeated content also includes a difference indication, which is used to indicate the difference between the protocol part of the first message and the protocol part of the target message.

In the present application, in addition to deduplicating data blocks at preset positions in the data part of the message, the first node can also deduplicate the protocol part of the message. By carrying a difference indication in the payload part of the message to replace the protocol part, the amount of data transmitted in the message can be further reduced, thereby reducing the network bandwidth overhead.

Optionally, one or more flow grouping sets corresponding to the second node are stored in the first node. Each flow grouping set includes flow identifiers of multiple flows flowing through the second node. After the first node obtains the first message, if there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the flow grouping set corresponding to the second node, the first node determines whether there is a target message in the target historical message sent to the second node, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set. If all flow grouping sets corresponding to the second node do not include the flow identifier of the flow to which the first message belongs, the first node sends the first message to the second node.

In the present application, the first node can determine whether it is necessary to perform deduplication processing on the message sent to the subordinate node based on the flow grouping set corresponding to the subordinate node. If the flow identifier of the flow to which the message belongs is not in the flow grouping set corresponding to the subordinate node, then the first node directly forwards the message to the subordinate node without executing the message deduplication process, which can reduce the processing overhead of the first node. In addition, when there are multiple flow grouping sets corresponding to the subordinate node, under this implementation method, the first node only needs to perform duplicate content judgment on the payload part of the historical message belonging to the multiple flows indicated by a flow grouping set and the payload part of the message, which reduces the number of historical messages that the first node needs to judge, thereby reducing the processing overhead of the first node, while improving the message processing efficiency of the first node, thereby improving the message transmission efficiency.

Optionally, the first node receives grouping information sent by the second node, where the grouping information includes a correspondence between a node identifier of the second node and one or more flow grouping sets corresponding to the second node.

Optionally, the first node is not an SFU server. After the first node receives the third message whose destination port number is the SFU service port number, A first node discovery message is sent to a third node, the third node is the next hop of the third message on the first node, the destination of the third message is the SFU server, the first node discovery message carries the identifier of the SFU server, and the first node discovery message indicates that the first node is a subordinate node of the third node on the transmission path starting from the SFU server. In response to not receiving a first node discovery response message corresponding to the first node discovery message sent by the third node, the first node determines that the first node is the first node that supports data deduplication on the transmission path starting from the SFU server.

Optionally, the first node generates a first node discovery message according to the third message, a message header of the first node discovery message is the same as a message header of the third message, and a payload portion of the first node discovery message carries an indication of a message type of the first node discovery message.

Optionally, the first node receives a second node discovery message sent by the fourth node, the second node discovery message carries an identifier of the SFU server, and the second node discovery message indicates that the fourth node is a subordinate node of the first node on a transmission path starting from the SFU server. The first node determines that the fourth node supports data deduplication based on the second node discovery message, and sends a second node discovery response message corresponding to the second node discovery message to the fourth node, and the second node discovery response message indicates that the first node supports data deduplication.

In the present application, in the scheme where the node discovery process is triggered by a message whose destination is the SFU server, if a node receives a node discovery message sent by a subordinate node and does not receive a node discovery response message sent by an upper node, then the node can determine itself as the first node on the transmission path starting from the SFU server.

Optionally, after the first node receives the fourth message whose source port number is the SFU service port number, the first node sends a third node discovery message to the fifth node, the fifth node is the next hop of the fourth message on the first node, the sender of the fourth message is the SFU server, the third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the superior node of the fifth node on the transmission path starting from the SFU server. In response to receiving a third node discovery response message corresponding to the third node discovery message sent by the fifth node, the first node determines that the fifth node supports data deduplication.

In the present application, in the scheme where the node discovery process is triggered by a message whose sender is an SFU server, if a node receives a node discovery response message sent by a subordinate node and does not receive a node discovery message sent by an upper node, then the node can determine that it is the first node on the transmission path starting from the SFU server.

Optionally, if the target message does not exist in the historical messages sent by the first node to the second node, the first node sends the first message to the second node.

In a second aspect, a data transmission method is provided. The method includes: a first node receives a first message sent by a second node. The sender of the first message is an SFU server. The first message carries a deduplication mark and indication information of repeated content. The deduplication mark is used to indicate that the first message is a deduplication message. The first node is the last node that supports data deduplication on the transmission path of the first message. The first node determines that the first message is a deduplication message based on the deduplication mark in the first message. The first node obtains the repeated content from a data set according to the indication information, and the data set includes at least part of the load part of the historical message received by the first node from the second node. The first node performs deduplication recovery processing on the load part of the first message according to the repeated content to obtain a second message. The load part of the second message includes the repeated content. The first node sends a second message to a third node, and the third node is the next hop of the first message on the first node.

In the present application, the first node can perform data recovery on deduplicated messages whose sender is the SFU server, so that the user can receive the original message carrying complete data content, thereby ensuring the user's business.

Optionally, the repeated content includes one or more repeated data blocks. The indication information of the repeated content includes one or more indications. The one or more indications correspond one-to-one to the one or more repeated data blocks in the repeated content. Each indication is used to indicate the hash value of the corresponding repeated data block.

In a possible implementation, each indication is also used to indicate the position of the corresponding repeated data block in the payload part of the original message corresponding to the first message, and the data set includes the payload part of the historical message received by the first node from the second node. The implementation process of the first node obtaining repeated content from the data set according to the indication information includes: for each indication in the indication information, the first node obtains the data block to be matched at the position of the payload part in the data set according to the position indicated by the indication. The first node calculates the hash value of the data block to be matched. The first node determines the data block to be matched whose hash value is consistent with the hash value indicated by the indication as the repeated data block corresponding to the indication.

In the present application, the first node can calculate the hash values of the data blocks at that position in the payload part of multiple stored payload parts according to the position of the duplicate data block indicated by the indication carried in the deduplication message in the payload part of the original message, so as to obtain a stored data block whose hash value is consistent with the hash value indicated by the indication, and then add the stored data block to the position indicated by the indication to achieve data recovery of the deduplication message.

In another possible implementation, the payload portion includes a protocol portion and a data portion, and one or more repeated data blocks are located in the data portion.

Optionally, the data set includes a corresponding relationship between a hash value of a historical data block and a historical data block, where the historical data block is a corresponding relationship between a hash value of a historical data block and a historical data block received by the first node. The first node obtains the repeated content from the data set according to the indication information, comprising: the first node determines the historical data block in the data set corresponding to the hash value indicated by the indication in the indication information as the repeated data block corresponding to the indication.

Optionally, the indication in the indication information of the repeated content is also used to indicate the position of the corresponding repeated data block in the data part of the original message corresponding to the first message. The implementation method of the first node performing deduplication recovery processing on the payload part of the first message according to the repeated content includes: for each indication in the indication information, the first node adds the repeated data block corresponding to the indication at the position indicated by the indication in the data part of the first message.

In the present application, the first node can search for the hash value carried in the deduplication message among the hash values of the stored historical data blocks, and add the historical data block corresponding to the hit hash value to the data part of the deduplication message to achieve recovery of the data part of the deduplication message.

Optionally, the repeated content also includes protocol information located in the protocol part, and the indication information also includes a difference indication, and the difference indication is used to indicate the difference between the protocol part of the original message corresponding to the first message and the protocol part of the target message, and the target message is a historical message received by the first node from the second node, in which the data part and the data part of the original message have one or more repeated data blocks. The data set also includes the protocol part of the message to which the historical data block belongs. The implementation process of the first node obtaining the repeated content from the data set according to the indication information also includes: the first node obtains the protocol part of the target message to which one or more repeated data blocks belong from the data set. Correspondingly, the implementation process of the first node performing deduplication and recovery processing on the payload part of the first message according to the repeated content also includes: the first node modifies the protocol part of the target message according to the difference indication, and uses the modified protocol part of the target message as the protocol part of the second message.

In the present application, the first node can obtain the protocol part of the historical message to which the historical data block corresponding to the hit hash value belongs, and restore the protocol part of the original message corresponding to the deduplicated message in combination with the difference indication for the protocol part in the deduplicated message, so as to realize the recovery of the protocol part of the deduplicated message.

Optionally, the deduplication mark is located in the payload portion of the first message. One or more flow grouping sets are stored in the first node, and each flow grouping set includes flow identifiers of multiple flows flowing through the first node. Before the first node determines that the first message is a deduplication message based on the deduplication mark, the first node determines that there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the one or more flow grouping sets. The first node parses the payload portion of the first message to obtain the deduplication mark.

Optionally, the implementation method of the first node obtaining repeated content from the data set according to the indication information includes: the first node obtains repeated content from the payload content of the target historical message in the data set according to the indication information, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set.

Under this implementation, when there are multiple flow grouping sets corresponding to the first node, the first node only needs to search the payload part of the historical messages belonging to multiple flows indicated by one flow grouping set to obtain duplicate content, which reduces the number of historical messages that the first node needs to retrieve, thereby reducing the processing overhead of the first node and improving the message processing efficiency of the first node, thereby improving the message transmission efficiency.

Optionally, the first node receives a third message, and the flow identifier of the flow to which the third message belongs does not belong to any flow grouping set. The first node forwards the third message.

In the present application, since the upper-level node will only perform data deduplication on the message in the flow indicated by the flow grouping set corresponding to the lower-level node, the first node can first determine whether the flow identifier of the flow to which the message belongs belongs to a certain flow grouping set corresponding to itself after receiving the message. If the flow identifier of the flow to which the message belongs belongs to a certain flow grouping set corresponding to itself, it means that the message may be a deduplication message that has been deduplication processed by the upper-level node, and the first node needs to further parse the payload part of the message to determine whether the message is a deduplication message. If the flow identifier of the flow to which the message belongs does not belong to any flow grouping set corresponding to itself, it means that the upper-level node will not perform data deduplication on the message, that is, the message cannot be a deduplication message, so the first node can directly forward the message without parsing the payload part of the message to determine whether the message is a deduplication message. This can reduce the processing overhead of the bottom node.

Optionally, the first node receives a fourth message, and the flow identifier of the flow to which the fourth message belongs belongs to the flow grouping set. The first node parses the payload part of the fourth message and determines that the payload part of the fourth message does not carry a deduplication mark. The first node adds at least part of the content of the payload part of the fourth message to the data set and forwards the fourth message. The fourth message here can be regarded as the first packet in a group of deduplicated messages received by the first node. The first node stores at least part of the content of the payload part of the fourth message in the data set so as to perform data recovery on the deduplicated messages received subsequently.

Optionally, the first node adds the flow identifiers of the flows to which the messages with duplicate contents in the payload part belong among the multiple messages received belonging to different flows to the same flow grouping set, and the senders of the different flows are all SFU servers.

Optionally, the first node sends grouping information to the second node, where the grouping information includes a correspondence between a node identifier of the first node and one or more flow grouping sets.

Optionally, after the first node receives the fifth message whose destination port number is the SFU service port number, the first node sends a first node discovery message to the fourth node, the fourth node is the next hop of the fifth message on the first node, the destination of the fifth message is the SFU server, the first node discovery message carries the identifier of the SFU server, and the first node discovery message indicates that the first node is the subordinate node of the fourth node on the transmission path starting from the SFU server. In response to receiving the first node discovery response message corresponding to the first node discovery message sent by the fourth node, the first node determines that the fourth node supports data deduplication.

Optionally, the first node generates a first node discovery message according to the fifth message, a message header of the first node discovery message is the same as a message header of the fifth message, and a payload portion of the first node discovery message carries an indication of a message type of the first node discovery message.

In the present application, in the scheme where the node discovery process is triggered by a message whose destination is the SFU server, if a node receives a node discovery response message sent by an upper-level node and does not receive a node discovery message sent by a lower-level node, then the node can determine that it is the last node on the transmission path starting from the SFU server.

Optionally, the first node receives a second node discovery message sent by the fifth node, the second node discovery message carries an identifier of the SFU server, and the second node discovery message indicates that the fifth node is the superior node of the first node on the transmission path starting from the SFU server. The first node determines that the fifth node supports data deduplication based on the second node discovery message, and sends a second node discovery response message corresponding to the second node discovery message to the fifth node, and the second node discovery response message indicates that the first node supports data deduplication.

Optionally, after the first node receives the sixth message whose source port number is the SFU service port number, the first node sends a third node discovery message to the sixth node, the sixth node is the next hop of the sixth message on the first node, the sender of the sixth message is the SFU server, the third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the superior node of the sixth node on the transmission path starting from the SFU server. In response to not receiving the third node discovery response message corresponding to the third node discovery message sent by the sixth node, the first node determines that the first node is the last node that supports data deduplication on the transmission path starting from the SFU server.

In the present application, in the scheme where the node discovery process is triggered by a message whose sender is an SFU server, if a node receives a node discovery message sent by an upper-level node and does not receive a node discovery response message sent by a lower-level node, then the node can determine that it is the last node on the transmission path starting from the SFU server.

In a third aspect, a data transmission method is provided. The method includes: a first node receives a first message sent by a second node. The sender of the first message is an SFU server. The first node is an intermediate node that supports data deduplication on the transmission path of the first message. If the first message is a non-deduplicated message, and there is a first original message in the historical message sent by the first node to the third node, the payload part of the first original message and the payload part of the first message have a first repeated content, the first node performs deduplication processing on the payload part of the first message to obtain a second message, the second message does not include the first repeated content, and the second message carries a deduplication mark and a first indication information of the first repeated content, the deduplication mark is used to indicate that the second message is a deduplication message, and the third node is the next hop of the first message on the first node. The first node sends a second message to the third node.

In the present application, the first node can perform data deduplication on the non-deduplicated message whose sender is the SFU server, so as to send the deduplicated message to the subordinate node. Since the data volume of the deduplicated message is smaller than that of the non-deduplicated message, the data volume of the message transmission can be reduced, thereby reducing the network bandwidth overhead. When the first node sends a deduplicated message to the subordinate node, it only needs to ensure that the overload part of the historical message carrying the content of the deduplicated message removed relative to the original message is sent to the subordinate node, so as to ensure that there is a node on the subsequent transmission path that can perform data recovery on the deduplicated message, so that the user finally receives the original message carrying the complete data content, thereby ensuring user services.

Optionally, if the first message is a non-deduplicated message and the first original message does not exist in the historical messages sent by the first node to the third node, the first node sends the first message to the third node.

Optionally, the first repetitive content includes one or more repetitive data blocks, and the first indication information includes one or more indications. The one or more indications in the first indication information correspond one-to-one to the one or more repetitive data blocks in the first repetitive content, and each indication is used to indicate a hash value of a corresponding repetitive data block.

In one possible implementation, a first data set is stored in a first node, and the first data set includes a payload portion of a historical message sent by the first node to a third node. The first node performs content matching on the payload portion of the first message and the payload portion in the first data set. If there is a target payload portion in the first data set that has a duplicate data block with the payload portion of the first message, the first node determines that the first original message exists in the historical message. If there is no payload portion in the first data set that has a duplicate data block with the payload portion of the first message, the first node determines that the first original message does not exist in the historical message, and the first node adds the payload portion of the first message to the first data set.

Optionally, the first data set includes a target payload part, and the first node performs deduplication processing on the payload part of the first message, including: for each repeated data block between the payload part of the first message and the target payload part, the first node calculates a hash of the repeated data block; The first node removes the duplicate data block from the payload portion of the first message, and adds an indication corresponding to the duplicate data block to the payload portion of the first message, the indication being used to indicate the hash value of the duplicate data block and the position of the duplicate data block in the payload portion of the first message.

In another possible implementation, the payload portion includes a protocol portion and a data portion. One or more repeated data blocks in the first repeated content are located in the data portion of the first message. A sampling tag set is stored in the first node, and the sampling tag set includes a hash value of a historical data block, where the historical data block is a data block obtained by sampling a preset position of the data portion of a historical message sent by the first node to the third node. The first node samples a preset position of the data portion of the first message to obtain a sampled data block. The first node calculates a hash value of the sampled data block. If the sampling tag set includes the hash value of the sampled data block, the first node determines that the first original message exists in the historical message. If the sampling tag set does not include the hash value of the sampled data block, the first node determines that the first original message does not exist in the historical message, and the first node adds the hash value of the sampled data block to the sampling tag set.

Optionally, the sampling tag set includes a hash value of the sampling data block, and the implementation method of the first node deduplicating the payload part of the first message includes: the first node uses the sampling data block whose hash value belongs to the sampling tag set as a duplicate data block, removes the duplicate data block from the data part of the first message, and adds an indication corresponding to the duplicate data block to the payload part of the first message, where the indication is used to indicate the hash value of the duplicate data block.

Optionally, the first node also stores the protocol part of the historical message sent by the first node to the third node; the first repeated content also includes protocol information located in the protocol part of the first message, and the first indication information also includes a difference indication, which is used to indicate the difference between the protocol part of the first message and the protocol part of the first original message.

Optionally, if the first message is a deduplicated message, the first message carries second indication information of the second duplicate content, and the second original message exists in the historical message sent by the first node to the third node, the payload part of the second original message includes the second duplicate content, and the first node sends the first message to the third node.

Optionally, if the first message is a deduplicated message, the first message carries second indication information for the second repeated content, and the second original message does not exist in the historical message sent by the first node to the third node, the payload part of the second original message includes the second repeated content, and the first node obtains the second repeated content from the second data set according to the second indication information, and the second data set includes at least part of the payload part of the historical message received by the first node from the second node. The first node performs deduplication recovery processing on the payload part of the first message according to the second repeated content to obtain a third message, and the payload part of the third message includes the second repeated content. The first node sends a third message to the third node.

Optionally, the second repetitive content includes one or more repetitive data blocks, and the second indication information includes one or more indications, the one or more indications in the second indication information correspond one-to-one to one or more repetitive data blocks in the second repetitive content, and each indication is used to indicate the hash value of the corresponding repetitive data block.

In a possible implementation, each indication is also used to indicate the position of the corresponding repeated data block in the payload part of the original message corresponding to the first message, and the second data set includes the payload part of the historical message received by the first node from the second node. The implementation method for the first node to obtain the second repeated content from the second data set according to the second indication information includes: for each indication in the second indication information, the first node obtains the data block to be matched at the position of the payload part in the second data set according to the position indicated by the indication. The first node calculates the hash value of the data block to be matched. The first node determines the data block to be matched whose hash value is consistent with the hash value indicated by the indication as the repeated data block corresponding to the indication.

In another possible implementation, the payload portion includes a protocol portion and a data portion, and one or more repeated data blocks in the second repeated content are located in the data portion. The second data set includes a correspondence between a hash value of a historical data block and a historical data block, and the historical data block is a data block obtained by sampling a preset position of a data portion of a historical message received by the first node from the second node. An implementation method in which the first node obtains the second repeated content from the second data set according to the second indication information includes: the first node determines the historical data block in the second data set corresponding to the hash value indicated by the indication in the second indication information as the repeated data block corresponding to the indication.

Optionally, the indication in the second indication information is also used to indicate the position of the corresponding duplicate data block in the data portion of the original message corresponding to the first message. The implementation method of the first node performing deduplication recovery processing on the payload portion of the first message according to the second duplicate content includes: for each indication in the second indication information, the first node adds the duplicate data block corresponding to the indication at the position indicated by the indication in the data portion of the first message.

Optionally, the second repeated content also includes protocol information located in the protocol part, and the second indication information also includes a difference indication, and the difference indication is used to indicate the difference between the protocol part of the original message corresponding to the first message and the protocol part of the target message, and the target message is a historical message received by the first node from the second node, in which the data part and the data part of the original message have one or more repeated data blocks. The second data set also includes the protocol part of the message to which the historical data block belongs. The implementation process of the first node obtaining the second repeated content from the second data set according to the second indication information also includes: the first node obtains the target message to which one or more repeated data blocks belong from the second data set; Protocol part. Accordingly, the implementation process of the first node performing deduplication recovery processing on the payload part of the first message according to the second duplicate content includes: the first node modifies the protocol part of the target message according to the difference indication, and uses the modified protocol part of the target message as the protocol part of the third message.

Optionally, one or more local flow grouping sets are stored in the first node, and each local flow grouping set includes flow identifiers of multiple flows flowing through the first node. After determining that there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in one or more local flow grouping sets, the first node parses the payload part of the first message. If the payload part of the first message carries a deduplication mark, the first node determines that the first message is a deduplication message. If the payload part of the first message does not carry a deduplication mark, the first node determines that the first message is a non-deduplication message.

Optionally, the first node adds the flow identifiers of the flows to which the messages with duplicate contents in the payload part belong among the multiple messages received belonging to different flows to the same local flow grouping set, and the senders of the different flows are all SFU servers.

Optionally, the first node sends first grouping information to the second node, where the first grouping information includes a correspondence between a node identifier of the first node and one or more local flow grouping sets.

Optionally, the first node stores one or more lower-level flow grouping sets corresponding to the third node, and each lower-level flow grouping set includes flow identifiers of multiple flows flowing through the third node. After the first node receives the first message sent by the second node, if there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the lower-level flow grouping set corresponding to the third node, the first node determines whether there is a message in the target historical message sent to the third node whose payload part has repeated content with the payload part of the first message, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set. If all lower-level flow grouping sets corresponding to the third node do not include the flow identifier of the flow to which the first message belongs, the first node sends the first message to the third node.

Optionally, the first node receives second grouping information sent by the third node, where the second grouping information includes a correspondence between a node identifier of the third node and one or more lower-level flow grouping sets.

Optionally, after the first node receives the fourth message whose destination port number is the SFU service port number, the first node sends a first node discovery message to the fourth node, the fourth node is the next hop of the fourth message on the first node, the destination of the fourth message is the SFU server, the first node discovery message carries the identifier of the SFU server, and the first node discovery message indicates that the first node is the subordinate node of the fourth node on the transmission path starting from the SFU server. In response to receiving the first node discovery response message corresponding to the first node discovery message sent by the fourth node, the first node determines that the fourth node supports data deduplication.

Optionally, the first node receives a fourth node discovery message sent by the seventh node, the fourth node discovery message carries an identifier of the SFU server, and the fourth node discovery message indicates that the seventh node is a subordinate node of the first node on a transmission path starting from the SFU server. The first node determines that the seventh node supports data deduplication based on the fourth node discovery message, and sends a fourth node discovery response message corresponding to the fourth node discovery message to the seventh node, and the fourth node discovery response message indicates that the first node supports data deduplication.

In the present application, in the scheme where the node discovery process is triggered by a message whose destination is the SFU server, if a node receives a node discovery response message sent by an upper-level node and receives a node discovery message sent by a lower-level node, then the node can determine itself as an intermediate node on the transmission path starting from the SFU server.

Optionally, after the first node receives the fifth message whose source port number is the SFU service port number, the first node sends a third node discovery message to the sixth node, the sixth node is the next hop of the fifth message on the first node, the sender of the fifth message is the SFU server, the third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the upper node of the sixth node on the transmission path starting from the SFU server. In response to receiving a third node discovery response message corresponding to the third node discovery message sent by the sixth node, the first node determines that the sixth node supports data deduplication.

Optionally, the first node receives a second node discovery message sent by the fifth node, the second node discovery message carries an identifier of the SFU server, and the second node discovery message indicates that the fifth node is the superior node of the first node on the transmission path starting from the SFU server. The first node determines that the fifth node supports data deduplication based on the second node discovery message, and sends a second node discovery response message corresponding to the second node discovery message to the fifth node, and the second node discovery response message indicates that the first node supports data deduplication.

In the present application, in the scheme where the node discovery process is triggered by a message whose sender is an SFU server, if a node receives a node discovery message sent by an upper-level node and a node discovery response message sent by a lower-level node, then the node can determine itself as an intermediate node on the transmission path starting from the SFU server.

In a fourth aspect, a node is provided. The node includes multiple functional modules, and the multiple functional modules interact with each other to implement the method in the first aspect and its embodiments. The multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on specific implementations.

Specifically, the node is a first node, and the first node includes: an acquisition module, which is used to acquire a first message whose sender is an SFU server, and the first node is the first node that supports data deduplication on the transmission path of the first message. A processing module, which is used to deduplicate the payload part of the first message to obtain a second message if there is a target message in the historical message sent by the first node to the second node, and the payload part of the target message has repeated content with the payload part of the first message, wherein the second message does not include the repeated content, and the second message carries a deduplication mark and indication information of the repeated content, wherein the deduplication mark is used to indicate that the second message is a deduplication message, and the second node is the next hop of the first message on the first node. A sending module, which is used to send the second message to the second node.

Optionally, the repeated content includes one or more repeated data blocks, and the indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repeated data blocks, and each indication is used to indicate a hash value of a corresponding repeated data block.

Optionally, a data set is stored in the first node, and the data set includes the payload part of the historical message sent by the first node to the second node, and the processing module is used to: perform content matching on the payload part of the first message and the payload part in the data set; if there is a target payload part in the data set that has a repeated data block with the payload part of the first message, determine that the target message exists in the historical message; for each repeated data block between the payload part of the first message and the target payload part, the first node calculates the hash value of the repeated data block; removes the repeated data block in the payload part of the first message, and adds an indication corresponding to the repeated data block to the payload part of the first message, wherein the indication is used to indicate the hash value of the repeated data block and the position of the repeated data block in the payload part of the first message.

Optionally, the processing module is used to: if there is no payload part with repeated data blocks with the payload part of the first message in the data set, determine that the target message does not exist in the historical message; and add the payload part of the first message to the data set.

Optionally, the payload part includes a protocol part and a data part, and the one or more repeated data blocks are located in the data part of the first message.

Optionally, a sampling tag set is stored in the first node, the sampling tag set includes a hash value of a historical data block, and the historical data block is a data block obtained by sampling a preset position of a data part of a historical message sent by the first node to the second node; the processing module is used to: sample the preset position of the data part of the first message to obtain a sampled data block; calculate the hash value of the sampled data block; if the sampling tag set includes the hash value of the sampled data block, determine that the target message exists in the historical message; use the sampled data block whose hash value belongs to the sampling tag set as a duplicate data block, remove the duplicate data block in the data part of the first message, and add an indication corresponding to the duplicate data block in the payload part of the first message, the indication being used to indicate the hash value of the duplicate data block.

Optionally, there are multiple preset positions, and the first node obtains multiple sampled data blocks by sampling the preset positions of the data part of the first message, and the indication is also used to indicate the position of the repeated data block in the data part of the first message.

Optionally, the processing module is configured to: if the sampling tag set does not include the hash value of the sampling data block, determine that the target message does not exist in the historical message; and add the hash value of the sampling data block to the sampling tag set.

Optionally, the sampling tag set also includes a historical data block indicated by a hash value. If the sampling tag set includes the hash value of the sampling data block, the processing module is used to: if the sampling tag set includes the hash value of the sampling data block, perform content matching on the sampling data block and the historical data block indicated by the hash value of the sampling data block; when the content of the sampling data block is the same as that of the historical data block indicated by the hash value of the sampling data block, determine that the target message exists in the historical message.

Optionally, the first node also stores the protocol part of the historical message sent by the first node to the second node; the repeated content also includes protocol information located in the protocol part of the first message, and the indication information also includes a difference indication, and the difference indication is used to indicate the difference between the protocol part of the first message and the protocol part of the target message.

Optionally, the first node stores one or more flow grouping sets corresponding to the second node, each of the flow grouping sets including flow identifiers of multiple flows flowing through the second node; the processing module is further used to, after the first node obtains the first message, determine whether the target message exists in the target historical message sent to the second node, if there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the flow grouping set corresponding to the second node, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set; the sending module is further used to send the first message to the second node if all flow grouping sets corresponding to the second node do not include the flow identifier of the flow to which the first message belongs.

Optionally, the first node further includes a receiving module; the receiving module is used to receive grouping information sent by the second node, and the grouping information includes a correspondence between a node identifier of the second node and the one or more flow grouping sets.

Optionally, the first node is not the SFU server; the sending module is also used to send a first node discovery message to the third node after receiving a third message whose destination port number is the SFU service port number, the third node is the next hop of the third message on the first node, the destination of the third message is the SFU server, the first node discovery message carries an identifier of the SFU server, and the first node discovery message indicates that the first node is a subordinate node of the third node on the transmission path starting from the SFU server; the processing module is also used to determine that the first node is the first node that supports data deduplication on the transmission path starting from the SFU server in response to not receiving a first node discovery response message corresponding to the first node discovery message sent by the third node.

Optionally, the processing module is also used to: generate the first node discovery message based on the third message, the message header of the first node discovery message is the same as the message header of the third message, and the payload part of the first node discovery message carries an indication of the message type of the first node discovery message.

Optionally, the first node also includes a receiving module; the receiving module is used to receive a second node discovery message sent by a fourth node, the second node discovery message carries the identifier of the SFU server, and the second node discovery message indicates that the fourth node is a subordinate node of the first node on the transmission path starting from the SFU server; the processing module is also used to determine whether the fourth node supports data deduplication based on the second node discovery message; the sending module is also used to send a second node discovery response message corresponding to the second node discovery message to the fourth node, and the second node discovery response message indicates that the first node supports data deduplication.

Optionally, the sending module is also used to send a third node discovery message to the fifth node after receiving a fourth message whose source port number is the SFU service port number, the fifth node is the next hop of the fourth message on the first node, the sender of the fourth message is the SFU server, the third node discovery message carries an identifier of the SFU server, and the third node discovery message indicates that the first node is the superior node of the fifth node on the transmission path starting from the SFU server; the processing module is also used to determine that the fifth node supports data deduplication in response to receiving a third node discovery response message corresponding to the third node discovery message sent by the fifth node.

Optionally, the sending module is further configured to send the first message to the second node if the target message does not exist in the historical messages sent by the first node to the second node.

In a fifth aspect, a node is provided. The node includes multiple functional modules, and the multiple functional modules interact with each other to implement the method in the second aspect and its respective embodiments. The multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on specific implementations.

Specifically, the node is a first node, and the first node includes: a receiving module, used to receive a first message sent by a second node, the sender of the first message is an SFU server, the first message carries a deduplication mark and indication information of repeated content, the deduplication mark is used to indicate that the first message is a deduplication message, and the first node is the last node that supports data deduplication on the transmission path of the first message; a processing module, used to determine that the first message is a deduplication message based on the deduplication mark; obtain the repeated content from a data set according to the indication information, the data set includes at least part of the payload part of the historical message received by the first node from the second node; perform deduplication recovery processing on the payload part of the first message according to the repeated content to obtain a second message, and the payload part of the second message includes the repeated content; a sending module, used to send the second message to a third node, and the third node is the next hop of the first message on the first node.

Optionally, the repeated content includes one or more repeated data blocks, and the indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repeated data blocks, and each indication is used to indicate a hash value of a corresponding repeated data block.

Optionally, each of the indications is also used to indicate the position of a corresponding duplicate data block in the payload part of the original message corresponding to the first message, and the data set includes the payload part of the historical message received by the first node from the second node; the processing module is used to: for each indication in the indication information, obtain the data block to be matched at the position of the payload part in the data set according to the position indicated by the indication; calculate the hash value of the data block to be matched; and determine the data block to be matched whose hash value is consistent with the hash value indicated by the indication as the duplicate data block corresponding to the indication.

Optionally, the payload portion includes a protocol portion and a data portion, and the one or more repeated data blocks are located in the data portion.

Optionally, the data set includes a correspondence between a hash value of a historical data block and the historical data block, wherein the historical data block is a data block obtained by sampling a preset position of a data portion of a historical message received by the first node from the second node; the processing module is used to determine a historical data block in the data set corresponding to the hash value indicated by the indication in the indication information as a duplicate data block corresponding to the indication.

Optionally, the indication is also used to indicate the position of the corresponding repeated data block in the data part of the original message corresponding to the first message; the processing module is used to add the repeated data block corresponding to the indication at the position indicated by the indication in the data part of the first message for each indication in the indication information.

Optionally, the repeated content also includes protocol information located in the protocol part, and the indication information also includes a difference indication, wherein the difference indication is used to indicate the difference between the protocol part of the original message corresponding to the first message and the protocol part of the target message, and the target message is a historical message received by the first node from the second node, in which the data part and the data part of the original message have the one or more repeated data blocks; the data set also includes the protocol part of the message to which the historical data blocks belong; the processing module is also used to: obtain the protocol part of the target message to which the one or more repeated data blocks belong from the data set; modify the protocol part of the target message according to the difference indication, and use the modified protocol part of the target message as the protocol part of the second message.

Optionally, the deduplication mark is located in the payload part of the first message, and one or more flow group sets are stored in the first node, each of the flow group sets including flow identifiers of multiple flows flowing through the first node; the processing module is also used to determine that there is a target flow group set including the flow identifier of the flow to which the first message belongs in the one or more flow group sets before the first node determines that the first message is a deduplication message based on the deduplication mark; and parse the payload part of the first message to obtain the deduplication mark.

Optionally, the processing module is used to: obtain the repeated content from the payload content of the target historical message in the data set according to the indication information, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set.

Optionally, the receiving module is further used to receive a third message, and the flow identifier of the flow to which the third message belongs does not belong to any of the flow grouping sets; the sending module is further used to forward the third message.

Optionally, the receiving module is also used to receive a fourth message, and the flow identifier of the flow to which the fourth message belongs belongs to the flow group set; the processing module is also used to parse the payload part of the fourth message, determine that the payload part of the fourth message does not carry the deduplication mark; add at least part of the content of the payload part of the fourth message to the data set, and forward the fourth message.

Optionally, the processing module is further used to: add the flow identifiers of the flows to which the messages with duplicate content in the payload part belong among the multiple messages received belonging to different flows to the same flow grouping set, and the senders of the different flows are all the SFU servers.

Optionally, the sending module is further used to send grouping information to the second node, where the grouping information includes a correspondence between a node identifier of the first node and the one or more flow grouping sets.

Optionally, the sending module is also used to send a first node discovery message to the fourth node after receiving a fifth message whose destination port number is the SFU service port number, the fourth node is the next hop of the fifth message on the first node, the destination of the fifth message is the SFU server, the first node discovery message carries an identifier of the SFU server, and the first node discovery message indicates that the first node is a subordinate node of the fourth node on the transmission path starting from the SFU server; the processing module is also used to determine that the fourth node supports data deduplication in response to receiving a first node discovery response message corresponding to the first node discovery message sent by the fourth node.

Optionally, the processing module is also used to: generate the first node discovery message based on the fifth message, the message header of the first node discovery message is the same as the message header of the fifth message, and the payload part of the first node discovery message carries an indication of the message type of the first node discovery message.

Optionally, the receiving module is also used to receive a second node discovery message sent by the fifth node, the second node discovery message carries the identifier of the SFU server, and the second node discovery message indicates that the fifth node is the superior node of the first node on the transmission path starting from the SFU server; the processing module is also used to determine whether the fifth node supports data deduplication based on the second node discovery message; the sending module is also used to send a second node discovery response message corresponding to the second node discovery message to the fifth node, and the second node discovery response message indicates that the first node supports data deduplication.

Optionally, the sending module is also used to send a third node discovery message to the sixth node after receiving the sixth message whose source port number is the SFU service port number, the sixth node is the next hop of the sixth message on the first node, the sender of the sixth message is the SFU server, the third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the superior node of the sixth node on the transmission path starting from the SFU server; the processing module is also used to determine that the first node is the last node supporting data deduplication on the transmission path starting from the SFU server in response to not receiving the third node discovery response message corresponding to the third node discovery message sent by the sixth node.

In a sixth aspect, a node is provided. The node includes multiple functional modules, and the multiple functional modules interact with each other to implement the above-mentioned The plurality of functional modules can be implemented based on software, hardware or a combination of software and hardware, and the plurality of functional modules can be arbitrarily combined or divided based on specific implementation.

Specifically, the node is a first node, and the first node includes: a receiving module, used to receive a first message sent by a second node, the sender of the first message is an SFU server, and the first node is an intermediate node that supports data deduplication on the transmission path of the first message; a processing module, used to, if the first message is a non-deduplicated message, and there is a first original message in the historical message sent by the first node to the third node, and the payload part of the first original message and the payload part of the first message have a first repeated content, deduplicate the payload part of the first message to obtain a second message, the second message does not include the first repeated content, and the second message carries a deduplication mark and a first indication information of the first repeated content, the deduplication mark is used to indicate that the second message is a deduplicated message, and the third node is the next hop of the first message on the first node; a sending module, used to send the second message to the third node.

Optionally, the sending module is further used to send the first message to the third node if the first message is a non-deduplicated message and the first original message does not exist in the historical messages sent by the first node to the third node.

Optionally, the first repeated content includes one or more repeated data blocks, and the first indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repeated data blocks, and each indication is used to indicate a hash value of a corresponding repeated data block.

Optionally, a first data set is stored in the first node, and the first data set includes the payload part of the historical message sent by the first node to the third node. The processing module is also used to: perform content matching on the payload part of the first message and the payload part in the first data set; if there is a target payload part in the first data set that has a repeated data block with the payload part of the first message, determine that the first original message exists in the historical message; if there is no payload part in the first data set that has a repeated data block with the payload part of the first message, determine that the first original message does not exist in the historical message, and the first node adds the payload part of the first message to the first data set.

Optionally, the first data set includes the target payload part, and the processing module is used to: calculate the hash value of the repeated data block for each repeated data block between the payload part and the target payload part of the first message; remove the repeated data block in the payload part of the first message, and add an indication corresponding to the repeated data block to the payload part of the first message, wherein the indication is used to indicate the hash value of the repeated data block and the position of the repeated data block in the payload part of the first message.

Optionally, the payload part includes a protocol part and a data part, and the one or more repeated data blocks are located in the data part of the first message; the first node stores a sampling tag set, and the sampling tag set includes a hash value of a historical data block, and the historical data block is a data block obtained by sampling a preset position of the data part of the historical message sent by the first node to the third node; the processing module is further used to: sample the preset position of the data part of the first message to obtain a sampled data block; calculate the hash value of the sampled data block; if the sampling tag set includes the hash value of the sampled data block, determine that the first original message exists in the historical message; if the sampling tag set does not include the hash value of the sampled data block, determine that the first original message does not exist in the historical message, and the first node adds the hash value of the sampled data block to the sampling tag set.

Optionally, the sampling tag set includes a hash value of the sampling data block, and the processing module is used to: take the sampling data block whose hash value belongs to the sampling tag set as a repeated data block, remove the repeated data block from the data part of the first message, and add an indication corresponding to the repeated data block to the payload part of the first message, wherein the indication is used to indicate the hash value of the repeated data block.

Optionally, the first node also stores the protocol part of the historical message sent by the first node to the third node; the first repeated content also includes protocol information located in the protocol part of the first message, and the first indication information also includes a difference indication, and the difference indication is used to indicate the difference between the protocol part of the first message and the protocol part of the first original message.

Optionally, the sending module is also used to send the first message to the third node if the first message is a deduplicated message, the first message carries second indication information for second repeated content, and there is a second original message in the historical message sent by the first node to the third node, and the payload part of the second original message includes the second repeated content.

Optionally, the processing module is also used to, if the first message is a deduplicated message, the first message carries second indication information for second repeated content, and the second original message does not exist in the historical messages sent by the first node to the third node, and the payload part of the second original message includes the second repeated content, obtain the second repeated content from a second data set according to the second indication information, the second data set including at least part of the payload part of the historical message received by the first node from the second node; perform deduplication and recovery processing on the payload part of the first message according to the second repeated content to obtain a third message, the payload part of the third message including the second repeated content; the sending module is also used to send the third message to the third node.

Optionally, the second repeated content includes one or more repeated data blocks, and the second indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repeated data blocks, and each indication is used to indicate a hash value of a corresponding repeated data block.

Optionally, each of the indications is also used to indicate the position of the corresponding repeated data block in the payload part of the original message corresponding to the first message, and the second data set includes the payload part of the historical message received by the first node from the second node; the processing module is used to: for each indication in the second indication information, obtain the data block to be matched at the position of the payload part in the second data set according to the position indicated by the indication; calculate the hash value of the data block to be matched; and determine the data block to be matched whose hash value is consistent with the hash value indicated by the indication as the repeated data block corresponding to the indication.

Optionally, the payload part includes a protocol part and a data part, and the one or more repeated data blocks are located in the data part; the second data set includes the correspondence between the hash value of the historical data block and the historical data block, and the historical data block is a data block obtained by sampling the preset position of the data part of the historical message received by the first node from the second node; the processing module is used to determine the historical data block in the second data set corresponding to the hash value indicated by the indication in the second indication information as the repeated data block corresponding to the indication.

Optionally, the indication is also used to indicate the position of the corresponding repeated data block in the data part of the original message corresponding to the first message; the processing module is used to add the repeated data block corresponding to the indication at the position indicated by the indication in the data part of the first message for each indication in the second indication information.

Optionally, the second repeated content also includes protocol information located in the protocol part, and the second indication information also includes a difference indication, wherein the difference indication is used to indicate the difference between the protocol part of the original message corresponding to the first message and the protocol part of the target message, and the target message is a historical message received by the first node from the second node, in which the data part and the data part of the original message have the one or more repeated data blocks; the second data set also includes the protocol part of the message to which the historical data blocks belong; the processing module is also used to: obtain the protocol part of the target message to which the one or more repeated data blocks belong from the second data set; modify the protocol part of the target message according to the difference indication, and use the modified protocol part of the target message as the protocol part of the third message.

Optionally, one or more local flow group sets are stored in the first node, and each of the local flow group sets includes flow identifiers of multiple flows flowing through the first node; the processing module is also used to: after determining that there is a target flow group set including the flow identifier of the flow to which the first message belongs in the one or more local flow group sets, parse the payload part of the first message; if the payload part of the first message carries a deduplication mark, determine that the first message is a deduplication message; if the payload part of the first message does not carry a deduplication mark, determine that the first message is a non-deduplication message.

Optionally, the processing module is further used to: add the flow identifiers of the flows to which the messages with duplicate content in the payload part among the multiple messages received belonging to different flows belong to, to the same local flow grouping set, and the senders of the different flows are all the SFU server.

Optionally, the sending module is further used to send first grouping information to the second node, where the first grouping information includes a correspondence between a node identifier of the first node and the one or more local flow grouping sets.

Optionally, the first node stores one or more lower-level flow group sets corresponding to the third node, each of the lower-level flow group sets including flow identifiers of multiple flows flowing through the third node. The processing module is further used to, after the first node receives the first message sent by the second node, determine whether there is a message in the target historical message sent to the third node whose payload part has repeated content with the payload part of the first message, if there is a target flow group set including the flow identifier of the flow to which the first message belongs in the lower-level flow group set corresponding to the third node, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow group set; the sending module is further used to send the first message to the third node if all lower-level flow group sets corresponding to the third node do not include the flow identifier of the flow to which the first message belongs.

Optionally, the receiving module is further used to receive second grouping information sent by the third node, where the second grouping information includes a correspondence between a node identifier of the third node and the one or more lower-level flow grouping sets.

Optionally, the sending module is also used to send a first node discovery message to a fourth node after receiving a fourth message whose destination port number is the SFU service port number, wherein the fourth node is the next hop of the fourth message on the first node, and the destination of the fourth message is the SFU server. The first node discovery message carries an identifier of the SFU server, and the first node discovery message indicates that the first node is a subordinate node of the fourth node on a transmission path starting from the SFU server; the processing module is also used to determine that the fourth node supports data deduplication in response to receiving a first node discovery response message corresponding to the first node discovery message sent by the fourth node.

Optionally, the receiving module is also used to receive a second node discovery message sent by the fifth node, the second node discovery message carries the identifier of the SFU server, and the second node discovery message indicates that the fifth node is the superior node of the first node on the transmission path starting from the SFU server; the processing module is also used to determine whether the fifth node supports data deduplication based on the second node discovery message; the sending module is also used to send a second node discovery response message corresponding to the second node discovery message to the fifth node, and the second node discovery response message indicates that the first node supports data deduplication.

Optionally, the sending module is also used to send a third node discovery message to the sixth node after receiving a fifth message whose source port number is the SFU service port number, the sixth node is the next hop of the fifth message on the first node, the sender of the fifth message is the SFU server, the third node discovery message carries an identifier of the SFU server, and the third node discovery message indicates that the first node is the superior node of the sixth node on the transmission path starting from the SFU server; the processing module is also used to determine that the sixth node supports data deduplication in response to receiving a third node discovery response message corresponding to the third node discovery message sent by the sixth node.

Optionally, the receiving module is also used to receive a fourth node discovery message sent by the seventh node, the fourth node discovery message carries the identifier of the SFU server, and the fourth node discovery message indicates that the seventh node is a subordinate node of the first node on the transmission path starting from the SFU server; the processing module is also used to determine whether the seventh node supports data deduplication based on the fourth node discovery message; the sending module is also used to send a fourth node discovery response message corresponding to the fourth node discovery message to the seventh node, and the fourth node discovery response message indicates that the first node supports data deduplication.

In a seventh aspect, a data transmission system is provided, comprising: an SFU server and a plurality of nodes in a communication network, the plurality of nodes comprising a first node and a second node, the first node being located between the SFU server and the second node. The SFU server is used to send a message to the first node, the first node is used to execute the method in the first aspect and its various embodiments, and the second node is used to execute the method in the second aspect and its various embodiments.

Optionally, the multiple nodes further include a third node, the third node is located between the first node and the second node, and the third node is used to execute the method in the above third aspect and its various embodiments.

In an eighth aspect, another data transmission system is provided, comprising: an SFU server and a first node in a communication network. The SFU server is used to execute the method in the first aspect and its respective embodiments, and the first node is used to execute the method in the second aspect and its respective embodiments.

Optionally, the communication network further includes a second node, the second node is located between the SFU server and the first node, and the second node is used to execute the method in the above third aspect and its various embodiments.

In a ninth aspect, a communication node is provided, comprising: a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to call the computer program to implement the method in the first aspect and its various embodiments, or to implement the method in the second aspect and its various embodiments, or to implement the method in the third aspect and its various embodiments.

In the tenth aspect, a computer-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the method of the above-mentioned first aspect and its various embodiments is implemented, or the method of the above-mentioned second aspect and its various embodiments is implemented, or the method of the above-mentioned third aspect and its various embodiments is implemented.

In the eleventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method of the above-mentioned first aspect and its various embodiments, or implements the method of the above-mentioned second aspect and its various embodiments, or implements the method of the above-mentioned third aspect and its various embodiments.

In the twelfth aspect, a chip is provided, which includes a programmable logic circuit and/or program instructions. When the chip is running, it implements the method in the above-mentioned first aspect and its various embodiments, or implements the method in the above-mentioned second aspect and its various embodiments, or implements the method in the above-mentioned third aspect and its various embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an SFU communication architecture provided in an embodiment of the present application;

FIG2 is a schematic diagram of the structure of a data transmission system provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of another data transmission system provided in an embodiment of the present application;

FIG4 is a schematic diagram of the structure of another data transmission system provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of another data transmission system provided in an embodiment of the present application;

FIG6 is a schematic diagram of a flow chart of a data transmission method provided in an embodiment of the present application;

FIG7 is a schematic diagram of a message deduplication process provided in an embodiment of the present application;

FIG8 is a schematic diagram of another message deduplication process provided in an embodiment of the present application;

FIG9 is a schematic diagram of another message deduplication process provided in an embodiment of the present application;

FIG10 is a schematic flow chart of another data transmission method provided in an embodiment of the present application;

FIG11 is a flow chart of another data transmission method provided in an embodiment of the present application;

FIG12 is a schematic diagram of the structure of a communication node provided in an embodiment of the present application;

13 is a schematic diagram of the structure of another communication node provided in an embodiment of the present application;

FIG14 is a schematic diagram of the structure of another communication node provided in an embodiment of the present application;

FIG. 15 is a schematic diagram of the hardware structure of a communication device provided in an embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

In communication scenarios such as video conferencing or live broadcasting, the audio and video data of the same user usually needs to be distributed to multiple recipients. For example, in a video conferencing scenario, the audio and video data of the conference host needs to be distributed to multiple conference members. For example, in a live broadcasting scenario, the audio and video data of the anchor user needs to be distributed to multiple audience users. The current mainstream communication scenarios such as video conferencing and live broadcasting usually adopt the SFU communication architecture. Under the SFU communication architecture, the source user sends the audio and video data to the SFU server, and the SFU server is responsible for completing the replication and distribution of the audio and video data. Among them, in the video conferencing scenario, the SFU server can be a video conferencing server. In the live broadcasting scenario, the SFU server can be a live broadcasting server.

For example, FIG1 is a schematic diagram of an SFU communication architecture provided by an embodiment of the present application. As shown in FIG1 , the SFU communication architecture includes a source user, an SFU server, and multiple receivers, such as multiple receivers including receiver a, receiver b, and receiver c. The implementation process of the source user sending audio and video data to multiple receivers includes: the source user sends audio and video data to the SFU server; the SFU server encapsulates the audio and video data in the payload part of multiple messages respectively, and the destinations of the multiple messages are the multiple receivers respectively; the SFU server sends corresponding messages to the multiple receivers respectively through the communication network, such as the SFU server sends message a to receiver a, sends message b to receiver b, and sends message c to receiver c through the communication network. Since the nodes in the communication network need to forward the messages sent by the SFU server to different receivers respectively, and the payload part of each message carries the complete audio and video data from the source user, that is, the data volume of each message is large, this will cause the nodes in the communication network to forward multiple messages. The transmission data volume is large, which in turn leads to a large network bandwidth overhead.

Based on this, the present application proposes a technical solution. For the original message whose sender is the SFU server, after the first node or intermediate node that supports data deduplication on the transmission path of the original message obtains the original message, it determines whether it has sent a historical message whose payload part has duplicate content with the payload part of the original message to the next hop of the original message. If it has sent a historical message whose payload part has duplicate content with the payload part of the original message to the next hop of the message, the node performs deduplication processing on the payload part of the original message to obtain a deduplication message, which does not include the duplicate content, and the deduplication message carries a deduplication mark indicating that it is a deduplication message, and then the node sends the deduplication message to the next hop of the original message. Before the deduplication message arrives at the destination, the intermediate node or the last node that supports data deduplication on the transmission path of the original message performs deduplication recovery processing on the payload part of the deduplication message, restores the original message, and then continues to send the original message to the destination. The original messages in this application all refer to non-deduplication messages. The present application can transmit deduplicated messages in the communication network to replace the original messages. Without affecting user services, the amount of message transmission data can be reduced through deduplication and recovery processing, thereby reducing network bandwidth overhead, improving transmission efficiency and quality, and reducing user costs. For example, in video conferencing scenarios or live broadcast scenarios that use the SFU communication architecture, when multiple recipients in the same video conference or live broadcast receive audio and video data from the same user, the received data is highly overlapping. By applying the technical solution of the present application, data deduplication can be performed on concurrent traffic sent by the same user to multiple recipients, and duplicate traffic sent by the same user to different recipients can be reduced, reducing the amount of message transmission data, thereby reducing network bandwidth opening and closing. pin.

The technical solution of the present application is explained below from multiple perspectives such as system, method flow, virtual device, hardware device, etc.

The following is an example of the system involved in this application.

The data transmission system provided in the embodiment of the present application adopts the SFU communication architecture and can be applied to a variety of communication scenarios, such as video conferencing scenarios or live broadcast scenarios. Optionally, in the video conferencing scenario, the SFU server can be a video conferencing server. In the live broadcast scenario, the SFU server can be a live broadcast server.

A typical application scenario of the present application is a cloud-to-branch scenario. In the cloud-to-branch scenario, the SFU server is deployed on the cloud, that is, the cloud is where the SFU server is located. The branch is usually deployed in a wide area network, and the branch is where the user is located. The cloud and the branch can communicate through a local area network. By deploying multiple nodes that support data deduplication between the cloud and the branch. When the SFU server sends traffic to the user, during the traffic forwarding process, the nodes close to the cloud perform data deduplication on the traffic, and the nodes close to the branch perform data recovery on the traffic. In the present application, nodes with data deduplication and/or data recovery functions are collectively referred to as nodes that support data deduplication. Among them, the nodes that support data deduplication can be independent hardware devices, or they can also be deployed in the form of software on a server or network device. Of course, the application scenario of the present application is not limited to the cloud-to-branch scenario, and can also be deployed according to actual needs. For example, if multiple nodes that support data deduplication are deployed in a local area network, data deduplication and data recovery can be performed on the traffic in the local area network. For example, if multiple nodes supporting data deduplication are deployed in a certain section of a wide area network, data deduplication and data recovery can be performed on the traffic in this section of the network. The present application can be used to perform data deduplication and data recovery on various traffic that conforms to the SFU communication architecture.

The present application is not limited to deploying only two layers of nodes that support data deduplication, but can also support the deployment of multiple layers of nodes that support data deduplication. Among them, the node close to the SFU server can be called the top node, the node close to the user can be called the bottom node, and the node between the top node and the bottom node can be called the intermediate node. The data transmission system provided in the embodiment of the present application includes at least a top node and a bottom node. The top node has a data deduplication function and is responsible for deduplicating data for the traffic sent by the SFU server to the user. The bottom node has a data recovery function and is responsible for recovering data for the traffic sent by the SFU server to the user. Optionally, the data transmission system provided in the embodiment of the present application may also include an intermediate node. The intermediate node has a data deduplication function and a data recovery function, and is responsible for deduplicating data or recovering data for the traffic sent by the SFU server to the user.

For example, Figure 2 is a structural diagram of a data transmission system provided in an embodiment of the present application. As shown in Figure 2, the data transmission system 200 adopts a one-to-one branch two-layer node deployment model. The data transmission system 200 includes an SFU server 201 and nodes 202A and 202B in a communication network. Among them, node 202A is deployed close to the SFU server 201, that is, node 202A is the top node, that is, node 202A is the first node that supports data deduplication on the transmission path starting from the SFU server 201. Node 202B is deployed close to branch 21, that is, node 202B is the bottom node, that is, node 202B is the last node that supports data deduplication on the transmission path from the SFU server 201 to the branch 21.

For another example, FIG. 3 is a schematic diagram of the structure of another data transmission system provided by an embodiment of the present application. As shown in FIG. 3, a data transmission system 300 adopts a one-to-many branch two-layer node deployment model. The data transmission system 300 includes an SFU server 301 and nodes 302A, 302B, 302C, and 302D in a communication network. Among them, node 302A is deployed close to the SFU server 301, that is, node 302A is a top node, that is, node 302A is the first node that supports data deduplication on the transmission path starting from the SFU server 301. Node 302B is deployed close to branch 31, that is, node 302B is the bottom node corresponding to branch 31, that is, node 302B is the last node that supports data deduplication on the transmission path from the SFU server 301 to branch 31. Node 302C is deployed close to branch 32, that is, node 302C is the bottom node corresponding to branch 32, that is, node 302C is the last node that supports data deduplication on the transmission path from the SFU server 301 to branch 32. Node 302D is deployed close to branch 33 , that is, node 302D is the bottom node corresponding to branch 33 , that is, node 302D is the last node supporting data deduplication on the transmission path from SFU server 301 to branch 33 .

For another example, FIG. 4 is a structural diagram of another data transmission system provided by an embodiment of the present application. As shown in FIG. 4, a data transmission system 400 adopts a multi-layer node multi-branch deployment model. The data transmission system 400 includes an SFU server 401 and nodes 402A, 402B, 402C, 402D, 402E, 402F, and 402G in a communication network. Among them, node 402A is deployed close to the SFU server 401, that is, node 402A is a top node, that is, node 402A is the first node that supports data deduplication on the transmission path starting from the SFU server 401. Node 402B is deployed close to branch 41, that is, node 402B is the bottom node corresponding to branch 41, that is, node 402B is the last node that supports data deduplication on the transmission path from the SFU server 401 to the branch 41. Node 402D is deployed close to branch 45, that is, node 402D is the bottom node corresponding to branch 45, that is, node 402D is the last node that supports data deduplication on the transmission path from SFU server 401 to branch 45. Node 402E is deployed close to branch 42, that is, node 402E is the bottom node corresponding to branch 45. 42, that is, node 402E is the last node supporting data deduplication on the transmission path from SFU server 401 to branch 42. Node 402F is deployed close to branch 43, that is, node 402F is the bottom node corresponding to branch 43, that is, node 402F is the last node supporting data deduplication on the transmission path from SFU server 401 to branch 43. Node 402G is deployed close to branch 44, that is, node 402G is the bottom node corresponding to branch 44, that is, node 402G is the last node supporting data deduplication on the transmission path from SFU server 401 to branch 44. Node 402C is deployed between node 402A and node 402E, node 402F and node 402G, that is, node 402C is the intermediate node corresponding to branch 42, branch 43 and branch 44 respectively, that is to say, node 402C is an intermediate node that supports data deduplication on the transmission path from SFU server 401 to branch 42, and node 402C is an intermediate node that supports data deduplication on the transmission path from SFU server 401 to branch 43, and node 402C is an intermediate node that supports data deduplication on the transmission path from SFU server 401 to branch 44.

Optionally, the SFU server may also support data deduplication, and the SFU server is the first node that supports data deduplication on the transmission path starting from the SFU server. For example, FIG. 5 is a structural diagram of another data transmission system provided in an embodiment of the present application. As shown in FIG. 5, a data transmission system 500 includes an SFU server 501 and a node 502 in a communication network. Among them, the SFU server 501 is a top node, that is, the SFU server 501 is the first node that supports data deduplication on the transmission path starting from the SFU server 501. Node 502 is deployed close to branch 51, that is, node 502 is the bottom node corresponding to branch 51, that is, node 502 is the last node that supports data deduplication on the transmission path from the SFU server 501 to branch 51. In the data transmission system 500 shown in FIG. 5, an intermediate node that supports data deduplication may also be deployed between the SFU server 501 and the node 502. For details, refer to the data transmission system 400 shown in FIG. 4, and the embodiments of the present application will not be repeated here.

Optionally, in the data transmission system shown in Figures 2 to 5, the node supporting data deduplication in the communication network may be an independent hardware device, or may be deployed in the form of software to a server or network device in an existing network. Network devices include but are not limited to routers or switches. The embodiment of the present application does not limit the type of communication network. For example, the communication network may be a dedicated line network, or it may be a non-dedicated line network. For another example, the communication network may include a tunnel, or it may not include a tunnel. For another example, the communication network may be a wide area network or a local area network, etc.

In one possible implementation, the communication scenario involved in the embodiments of the present application adopts scalable video coding (SVC). SVC is a type of video coding, which is implemented by layered encoding and selective transmission of video data. The specific implementation method of using SVC in the communication scenario is that the source host sends a multi-layer stream to the SFU server, and the multi-layer stream includes a basic layer stream and an enhanced layer stream, and then the SFU server selects and distributes the multi-layer stream according to the capabilities of the receiving host. For example, for a receiving host with strong decoding capabilities, the SFU sends a basic layer stream and an enhanced layer stream to the receiving host. For a receiving host with weak decoding capabilities, the SFU only sends a basic layer stream to the receiving host.

In one possible implementation, the communication scenario involved in the embodiments of the present application supports end-to-end encryption (E2EE). In this implementation, the source host uses the advanced encryption standard (AES) key to encrypt the business data, and then sends the encrypted ciphertext to the SFU server. The SFU server carries the ciphertext in the message and distributes it to the receiving host. The receiving host uses the AES key to decrypt the ciphertext to obtain the business data. In the case where the communication scenario includes multiple receiving hosts, the multiple receiving hosts can use the same AES key. For example, in a video conferencing scenario, the conference host can distribute the AES key to multiple conference members uniformly through signaling.

For example, the communication scenario involved in the embodiment of the present application adopts the SFU communication architecture, adopts SVC, and supports E2EE. The data transmission in this communication scenario has the following characteristics: signaling data is transmitted based on the transmission control protocol (TCP) and the transport layer security (TLS) protocol, and business data (such as audio and video data) is based on the user datagram protocol (UDP) and AES encryption transmission. For weak network or weak terminal scenarios, business data can also be transmitted using TCP and AES encryption.

The following is an example of the method flow involved in this application.

In the data transmission system provided in the embodiment of the present application, when the top node, the bottom node and the middle node transmit the message whose sender is the SFU server, the processes of deduplication and recovery processing of the message are different. The present application respectively describes the implementation methods of the three types of node transmission messages through the following three embodiments.

The first embodiment is applied to the top node, and the top node is used to perform deduplication processing on the message. For example, FIG6 is a flow chart of a data transmission method 600 provided in an embodiment of the present application. In order to facilitate the distinction between the contents of each embodiment, in the method 600, the first node is referred to as node 11, the second node is referred to as node 12, the first message is referred to as message 11, and the second message is referred to as message 12. The method 600 can be applied to For the data transmission system 200 shown in FIG. 2 , the node 11 in the method 600 is the node 202A, and the node 12 is the node 202B. Alternatively, the method 600 can be applied to the data transmission system 300 shown in FIG. 3 , and the node 11 in the method 600 is the node 302A, and the node 12 is the node 302B, the node 302C, or the node 302D. Alternatively, the method 600 can be applied to the data transmission system 400 shown in FIG. 4 , and the node 11 in the method 600 is the node 402A, and the node 12 is the node 402B, the node 402C, or the node 402D. Alternatively, the method 600 can be applied to the data transmission system 500 shown in FIG. 5 , and the node 11 in the method 600 is the SFU server 501, and the node 12 is the node 502. As shown in FIG. 6 , the method 600 includes but is not limited to the following steps 601 to 603.

Step 601: Node 11 obtains message 11 whose sender is the SFU server. Node 11 is the first node on the transmission path of message 11 that supports data deduplication.

Among them, message 11 is the original message, that is, the message is not deduplicated. The sender of message 11 is the SFU server, and node 11 is the first node on the transmission path of message 11 that supports data deduplication, that is, node 11 is the first node on the transmission path starting from the SFU server and ending at the destination of message 11 that supports data deduplication. Optionally, the source Internet Protocol (IP) address of message 11 can be the IP address of the SFU server. For example, if the SFU server is deployed in the public network, the source IP address of message 11 can be represented by the public network address of the SFU server. Alternatively, the source IP address of message 11 can be the address obtained after the IP address of the SFU server is converted to network address translation (NAT). For example, if the SFU server is deployed in a private network, the source IP address of message 11 can be represented by the public network address corresponding to the private network address of the SFU server.

Optionally, the node 11 is an SFU server, and the node 11 obtains the message 11 , which may be the node 11 generating the message 11 .

Alternatively, if node 11 is not an SFU server, for example, node 11 is a network device connected to an SFU server, node 11 obtains message 11, which may be node 11 receiving message 11 sent by the SFU server. Optionally, after receiving a message, node 11 may determine whether the sender of the message is an SFU server according to the source port number carried by the message.

Step 602: If the target message exists in the historical message sent by node 11 to node 12, and the payload part of the target message has repeated content with the payload part of message 11, node 11 deduplicates the payload part of message 11 to obtain message 12. Message 12 does not include the repeated content, and message 12 carries a deduplication mark and indication information of the repeated content. The deduplication mark is used to indicate that message 12 is a deduplication message, and node 12 is the next hop of message 11 on node 11.

Optionally, the payload part is a UDP payload or a TCP payload. The payload part includes a protocol part and a data part. The protocol part can be used to carry application layer protocols, such as real-time transport protocol (RTP), file transfer protocol (FTP) or hypertext transfer protocol (HTTP). The data part can be used to carry application layer data, such as signaling data or service data.

Optionally, the repeated content between the payload portion of the target message and the payload portion of message 11 includes one or more repeated data blocks, and the indication information of the repeated content includes one or more indications, and the one or more indications correspond one-to-one to one or more repeated data blocks in the repeated content. Each indication is used to indicate the hash value of the corresponding repeated data block, and specifically, each indication includes the hash value of the corresponding repeated data block. Optionally, the hash value of the repeated data block can be calculated based on the data content and length of the repeated data block, or part of the data content of the repeated data block can be used as the hash value of the repeated data block. The embodiment of the present application does not limit the calculation method of the hash value of the repeated data block.

Optionally, after obtaining message 11 whose sender is the SFU server, node 11 needs to execute a judgment process, which is used to determine whether there is a target message in the historical message sent by itself to node 12 whose payload part has repeated content with the payload part of message 11. The historical message here refers to the original message sent by node 11 to node 12 before executing the judgment process. The following embodiments of the present application provide two possible implementation methods for node 11 to execute the judgment process, including possible implementation method A1 and possible implementation method A2.

Possible implementation method A1, one or more repeated data blocks in the above-mentioned repeated content are located in the data part and/or the protocol part of the payload part of the message 11. Node 11 stores a data set, which includes the payload part of the historical message sent by node 11 to node 12. Node 11 executes a judgment process, including: node 11 matches the payload part of message 11 with the payload part in the data set. If there is a target payload part with repeated data blocks in the payload part of message 11 in the data set, node 11 determines that there is a target message in the historical message sent to node 12. If there is no payload part with repeated data blocks in the payload part of message 11 in the data set, node 11 determines that there is no target message in the historical message sent to node 12. Optionally, after node 11 determines that there is no target message in the historical message sent to node 12, node 11 can add the payload part of message 11 to the data set to obtain an updated data set. The updated data set can be used by node 11 to perform deduplication processing on the message whose sender is the SFU server obtained subsequently.

Optionally, node 11 performs content matching on the payload part of message 11 and the payload part in the data set, which may be sampling one or more positions of the payload part of message 11 and sampling one or more positions of the payload part in the data set. The sampling content of position 1 of the payload part of message 11 is the same as the sampling content of position 2 of a payload part in the data set, and node 11 continues to accurately match the content before and after position 1 of the payload part of message 11 with the content before and after position 2 of the payload part in the data set to determine the repeated data blocks of the payload part of message 11 and the payload part in the data set. The specific implementation method of content matching is not limited in the embodiment of the present application.

For example, the content of the payload part of message 11 is "44336655", and the content of a payload part in the data set is "11332255". Node 11 matches the content of the payload part of message 11 with the payload part in the data set, and determines that the payload part of message 11 and the payload part in the data set have repeated data blocks "33" and "55". Then, node 11 can determine the payload part in the data set as the target payload part, and determine that the target message exists in the historical messages sent by node 11 to node 12. The target message is the historical message including the target payload part.

In combination with the above possible implementation A1, the implementation process of node 11 performing deduplication processing on the payload part of message 11 includes: for each duplicate data block between the payload part of message 11 and the target payload part, node 11 calculates the hash value of the duplicate data block. Node 11 removes the duplicate data block in the payload part of message 11, and adds an indication corresponding to the duplicate data block in the payload part of message 11, the indication is used to indicate the hash value of the duplicate data block and the position of the duplicate data block in the payload part of message 11. Optionally, the position of the duplicate data block in the payload part of message 11 can be represented by the starting position of the duplicate data block in the payload part of message 11 and the length of the duplicate data block, or can be represented by the ending position of the duplicate data block in the payload part of message 11 and the length of the duplicate data block, or can be represented by the starting position of the duplicate data block in the payload part of message 11 and the ending position of the duplicate data block in the payload part of message 11.

For example, referring to the above example, the hash value of the repeated data block "33" is a, and the hash value of the repeated data block "55" is b. After deduplication of the payload part of message 11, the content of the payload part of message 12 can be expressed as "4466; Indication 1: <a, position 3-4>; Indication 2: <b, position 7-8>". Among them, "4466" is the remaining content after removing the duplicate content in the payload part of message 11 and the target payload part, Indication 1 "<a, position 3-4>" is used to indicate that the data block corresponding to hash value a is located at the 3rd byte and 4th byte of the payload part of message 12, and Indication 2 "<b, position 7-8>" is used to indicate that the data block corresponding to hash value b is located at the 7th byte and 8th byte of the payload part of message 12.

Optionally, if node 11 has multiple subordinate nodes, multiple node-level data sets may be stored in node 11, each node-level data set is used to store the payload portion of the historical message sent by node 11 to a subordinate node, that is, node 11 stores a data set for each subordinate node. Alternatively, a global data set may be stored in node 11, and the global data set includes a correspondence between the payload portion and the node identifier, and the node identifier is used to indicate that the historical message to which the corresponding payload portion belongs is sent to the subordinate node indicated by the node identifier, that is, node 11 stores a common data set for all subordinate nodes. The subordinate node of node 11 refers to a node located after node 11 on the transmission path starting from the SFU server. For example, in the data transmission system 300 shown in FIG. 3, node 11 may be node 302A, and node 302B, node 302C, and node 302D are all subordinate nodes of node 11.

In the possible implementation A1, the top node stores the complete content of the payload part of the historical message sent to the lower node. When the top node determines whether the original message obtained has duplicate content with the historical message, it matches the entire content of the payload part of the message without distinguishing whether the duplicate content is in the data part or the protocol part.

In the embodiment of the present application, the top node can perform content matching on the payload part of the acquired original message and the payload part of the stored historical message. If there are duplicate data blocks in the payload part of the original message and the payload part of the historical message, the top node calculates the hash value of the duplicate data block, removes the duplicate data block in the original message to obtain a deduplicated message, and further carries the hash value of the duplicate data block and the position of the duplicate data block in the deduplicated message to achieve data deduplication of the original message.

For example, FIG7 is a schematic diagram of a message deduplication process provided by an embodiment of the present application. As shown in FIG7, the original message includes an Ethernet header, an IP header, a TCP/UDP header and a payload part, and the payload part includes a protocol part and a data part. The payload part of the original message is deduplicated to obtain a deduplicated message. The deduplicated message includes an Ethernet header, an IP header, a TCP/UDP header and a payload part, and the payload part includes a deduplication mark, an indication corresponding to a duplicate data block, and other contents in the payload part of the original message except the duplicate data block. Among them, the indication corresponding to the duplicate data block is used to indicate the hash value of the duplicate data block and the position of the duplicate data block in the payload part of the original message. Among them, in the payload part of the deduplication message, the deduplication mark and the indication corresponding to the duplicate data block may be before the protocol part and the data part, or may also be after the protocol part and the data part, and the embodiment of the present application does not limit this.

Possible implementation method A2, one or more repeated data blocks in the above repeated content are located in the data part of the payload part of message 11. Node 11 stores a sampling label set, which includes the hash value of the historical data block. The historical data block is a data block obtained by sampling the preset position of the data part of the historical message sent by node 11 to node 12. Node 11 executes a judgment process, including: node 11 samples the preset position of the data part of message 11 to obtain a sampled data block. Node 11 calculates the hash value of the sampled data block. If the sample The tag set includes the hash value of the sampled data block, and the node 11 determines that the target message exists in the historical messages sent to the node 12. If the sampling tag set does not include the hash value of the sampled data block, the node 11 determines that the target message does not exist in the historical messages sent to the node 12. Optionally, after determining that the target message does not exist in the historical messages sent to the node 12, the node 11 can add the hash value of the sampled data block to the sampling tag set to obtain an updated sampling tag set. The updated sampling tag set can be used by the node 11 to perform deduplication processing on the messages whose sender is the SFU server that are subsequently obtained.

The preset position of the data part is the sampling position of the data part set in advance. Optionally, the preset position of the data part can be the entire data part, and the sampled data block obtained by node 11 sampling the preset position of the data part of message 11 is the complete content of the data part of message 11. Alternatively, the preset position of the data part can also be a local field of the data part. In this case, there can be one or more preset positions, and node 11 samples each preset position of the data part of message 11 to obtain one or more sampled data blocks.

In combination with the possible implementation method A2, the implementation process of node 11 deduplicating the payload part of message 11 includes: node 11 takes the sampling data block whose hash value belongs to the sampling tag set as a duplicate data block, removes the duplicate data block from the data part of message 11, and adds an indication corresponding to the duplicate data block to the payload part of message 11, where the indication is used to indicate the hash value of the duplicate data block.

Optionally, there are multiple preset positions, and node 11 obtains multiple sampled data blocks by sampling the preset positions of the data part of message 11. Correspondingly, there may be one or more duplicate data blocks. In this case, the indication corresponding to the duplicate data block is also used to indicate the position of the duplicate data block in the data part of message 11. For example, the indication corresponding to a duplicate data block can be expressed as <hash value, position>. In an embodiment of the present application, when there are multiple sampling positions of the pre-set data part, the top node samples the data part of the original message and obtains multiple sampled data blocks. In this case, it is necessary to indicate the position of the duplicate data block removed from the deduplicated message in the original message so that subsequent nodes can perform data recovery on the deduplicated message.

Optionally, if node 11 has multiple subordinate nodes, node 11 may store multiple node-level sampling label sets, each node-level sampling label set corresponds to a subordinate node, and each node-level sampling label set is used to store the hash value of the data block sampled at the preset position of the data part of the historical message sent by node 11 to the corresponding subordinate node. That is, node 11 stores a sampling label set for each subordinate node. Alternatively, node 11 may store a global sampling label set, which includes the correspondence between hash values and node identifiers, and the node identifier is used to indicate that the corresponding hash value comes from the historical message sent to the subordinate node indicated by the node identifier, that is, node 11 stores a common sampling label set for all subordinate nodes.

In the possible implementation A2, the top node stores the hash value of the data block at the preset position of the data part of the historical message sent to the lower node. When the top node determines whether the acquired original message and the historical message have duplicate content, it calculates the hash value of the sampled data block sampled at the preset position of the data part of the original message, and compares it with the stored hash value to achieve overall deduplication of the data block at the preset position of the data part of the original message.

In an embodiment of the present application, the top node can calculate the hash value of the sampled data block at a preset position of the data portion of the acquired original message, and compare it with the hash value of the stored historical data block. If the hash value of a sampled data block in the original message is the same as the hash value stored by the top node, the top node removes the sampled data block in the original message to obtain a deduplicated message, and further carries the hash value of the sampled data block in the deduplicated message to achieve data deduplication of the original message. Compared with the possible implementation method A1 described above, the top node does not need to perform content matching on the payload part of the original message and the historical message, thereby improving the efficiency of message deduplication.

For example, Figure 8 is a schematic diagram of another message deduplication process provided by an embodiment of the present application. As shown in Figure 8, the original message includes an Ethernet header, an IP header, a TCP/UDP header and a payload part, and the payload part includes a protocol part and a data part. The payload part of the original message is deduplicated to obtain a deduplicated message. The deduplicated message includes an Ethernet header, an IP header, a TCP/UDP header and a payload part. Assuming that the sampling position of the pre-set data part is the entire data part, the payload part of the deduplicated message includes a deduplication mark, a hash value of the data part of the original message, and the protocol part of the original message. Among them, in the payload part of the deduplication message, the deduplication mark and the hash value of the data part can be before the protocol part, or can also be after the protocol part, and the embodiment of the present application does not limit this.

Optionally, the sampling label set stored in the node 11 also includes the historical data block indicated by the hash value, that is, the sampling label set stored in the node 11 may include the correspondence between the historical data block and the hash value of the historical data block. Then, if the sampling label set includes the hash value of the sampling data block, the node 11 determines that the target message exists in the historical message in an implementation manner, including: if the sampling label set includes the hash value of the sampling data block, the node 11 performs content matching on the sampling data block and the historical data block indicated by the hash value of the sampling data block. If the sampling data block has the same content as the historical data block indicated by the hash value of the sampling data block, the node 11 determines that the target message exists in the historical message.

Since the data contents of two data blocks with the same hash value may be different, by storing the historical data blocks and historical The corresponding relationship between the hash values of the data blocks enables the node 11 to further perform content matching on the sampled data block and the historical data block after determining that the hash value of the sampled data block is the same as the hash value of a historical data block, so as to achieve an accurate match, thereby improving the accuracy of deduplication of the message.

In combination with the above possible implementation A2, the node 11 may also store the protocol part of the historical message sent by the node 11 to the node 12. The repeated content of the payload part of the message 11 and the payload part of the target message may also include the protocol information located in the protocol part of the message 11. Accordingly, the indication information of the repeated content may also include a difference indication, and the difference indication is used to indicate the difference between the protocol part of the message 11 and the protocol part of the target message. The difference between the protocol part of the message 11 and the protocol part of the target message specifically includes the difference information between the protocol part of the message 11 and the protocol part of the target message and the position of the difference information in the protocol part of the message 11.

Optionally, after determining that the target message exists in the historical message, node 11 performs content matching on the protocol part of message 11 and the protocol part of the target message to determine the difference information between the protocol part of message 11 and the protocol part of the target message and the position of the difference information in the protocol part of message 11.

For example, Figure 9 is another schematic diagram of a message deduplication process provided by an embodiment of the present application. As shown in Figure 9, the original message includes an Ethernet header, an IP header, a TCP/UDP header and a payload part, and the payload part includes a protocol part and a data part. The payload part of the original message is deduplicated to obtain a deduplicated message. The deduplicated message includes an Ethernet header, an IP header, a TCP/UDP header and a payload part. Assuming that the sampling position of the pre-set data part is the entire data part, the payload part of the deduplicated message includes a deduplication mark, a hash value of the data part of the original message, and a difference indication. The difference indication is used to indicate the difference information between the protocol part of the original message and the protocol part of the target message and the position of the difference information in the protocol part of the original message.

In an embodiment of the present application, in addition to deduplicating data blocks at preset positions in the data part of the message, the top node can also deduplicate the protocol part of the message. By carrying a difference indication in the payload part of the message instead of the protocol part, the amount of data transmitted in the message can be further reduced, thereby reducing the network bandwidth overhead.

Step 603 : Node 11 sends message 12 to node 12 .

Alternatively, after node 11 executes the above-mentioned judgment process for determining whether there is a target message with a payload part having repeated content with the payload part of message 11 in the historical messages sent by itself to node 12, if the target message does not exist in the historical messages sent by node 11 to node 12, node 11 sends message 11 to node 12, that is, node 11 directly forwards message 11 without performing deduplication processing on message 11, that is, the above-mentioned steps 602 and 603 are not executed.

In an embodiment of the present application, after receiving a message whose sender is an SFU server, the top node can determine whether to send a historical message whose overload part has duplicate content with the load part of the message to the next hop of the message. If the top node sends a historical message whose overload part has duplicate content with the load part of the message to the next hop of the message, the top node can perform data deduplication on the message, and then send a deduplication message to the subordinate node. Since the data volume of the deduplication message is smaller than the data volume of the non-deduplication message, the data volume of the message transmission can be reduced, thereby reducing the network bandwidth overhead. When the top node sends a deduplication message to the subordinate node, it only needs to ensure that the overload part sent to the subordinate node carries the historical message with the content removed from the deduplication message relative to the original message, so as to ensure that there is a node on the subsequent transmission path that can perform data recovery on the deduplication message, so that the user finally receives the original message carrying the complete data content, thereby ensuring user services.

The second embodiment is applied to the bottom node, and the bottom node is used to perform deduplication recovery processing on the message. For example, Figure 10 is a flow chart of another data transmission method 1000 provided in an embodiment of the present application. In order to facilitate the distinction between the contents of each embodiment, in method 1000, the first node is referred to as node 21, the second node is referred to as node 22, the third node is referred to as node 23, the first message is referred to as message 21, and the second message is referred to as message 22. The method 1000 can be applied to the data transmission system 200 shown in Figure 2, then the node 21 in the method 1000 is node 202B, and the node 22 is node 202A. Alternatively, the method 1000 can be applied to the data transmission system 300 shown in Figure 3, then the node 21 in the method 1000 is node 302B, node 302C or node 302D, and the node 22 is node 302A. Alternatively, the method 1000 may be applied to the data transmission system 400 shown in FIG4 , then the node 21 in the method 1000 is the node 402B or the node 402D, and the node 22 is the node 402A, or the node 21 is the node 402E, the node 402F or the node 402G, and the node 22 is the node 402C. Alternatively, the method 1000 may be applied to the data transmission system 500 shown in FIG5 , then the node 21 in the method 1000 is the node 502, and the node 22 is the SFU server 501. As shown in FIG10 , the method 1000 includes but is not limited to the following steps 1001 to 1005.

Step 1001, node 21 receives message 21 sent by node 22. The sender of message 21 is the SFU server. Message 21 carries a deduplication mark and indication information of repeated content. The deduplication mark is used to indicate that message 21 is a deduplication message. Node 21 is the last node on the transmission path of message 21 that supports data deduplication.

The sender of message 21 is the SFU server, and node 21 is the last node that supports data deduplication on the transmission path of message 21, that is, node 21 is the last node that supports data deduplication on the transmission path starting from the SFU server and ending at the destination of message 21. Optionally, the source IP address of message 21 may be the IP address of the SFU server, or may be the address obtained after the IP address of the SFU server is NATed.

Optionally, the repeated content includes one or more repeated data blocks, and the indication information of the repeated content includes one or more indications, and the one or more indications correspond to one or more repeated data blocks in the repeated content. Each indication is used to indicate the hash value of the corresponding repeated data block, and specifically, each indication includes the hash value of the corresponding repeated data block. Among them, the specific explanation of the repeated content and the indication information can refer to the relevant content in the above step 602, and the embodiment of the present application will not be repeated here.

Step 1002: Node 21 determines that message 21 is a deduplicated message based on the deduplication flag.

Optionally, the deduplication mark is located in the payload of the message 21. After receiving the message 21, the node 21 parses the payload of the message 21 to obtain the deduplication mark, and further determines that the node 21 is a deduplication message based on the deduplication mark.

Step 1003 : Node 21 obtains the repeated content from a data set according to the indication information of the repeated content. The data set includes at least part of the payload part of the historical message received by node 21 from node 22 .

If the node in the data transmission system uses the possible implementation method A1 in the above step 602 to perform deduplication processing on the message, the data set stored in the node 21 includes the payload part of the historical message received by the node 21 from the node 22. If the node in the data transmission system uses the possible implementation method A2 in the above step 602 to perform deduplication processing on the message, the data set stored in the node 21 includes the corresponding relationship between the hash value of the historical data block and the historical data block, and the historical data block is a data block obtained by sampling the preset position of the data part of the historical message received by the node 21 from the node 22. If the node in the data transmission system uses the possible implementation method A2 in the above step 602 to perform deduplication processing on the message, the data set stored in the node 21 may also include the protocol part of the message to which the historical data block belongs.

Possible implementation B1: Message 21 is obtained by a node located before node 21 on the transmission path by performing deduplication processing on the original message using possible implementation A1 in step 602. Each indication in the indication information of repeated content is used to indicate the hash value of the corresponding repeated data block and the position of the corresponding repeated data block in the payload part of the original message corresponding to message 21.

In combination with the above possible implementation mode B1, the implementation process of node 21 obtaining the repeated content from the data set according to the indication information of the repeated content includes: for each indication in the indication information, node 21 obtains the to-be-matched data block at the position indicated by the indication in the load part of the data set. Node 21 calculates the hash value of the to-be-matched data block. Node 21 determines the to-be-matched data block whose hash value is consistent with the hash value indicated by the indication as the repeated data block corresponding to the indication.

For example, the content of the payload part of message 21 includes "4466; Indication 1: <a, position 3-4>; Indication 2: <b, position 7-8>". Among them, "4466" includes the content of the protocol part and/or the content of the data part, Indication 1 "<a, position 3-4>" is used to indicate that the data block corresponding to the hash value a is located at the 3rd byte and the 4th byte of the payload part of the original message corresponding to message 21, and Indication 2 "<b, position 7-8>" is used to indicate that the data block corresponding to the hash value b is located at the 7th byte and the 8th byte of the payload part of the original message corresponding to message 21. Assuming that the content of a payload part in the data set is "11332255", node 21 obtains the to-be-matched data block "33" in the payload part according to the position indicated by Indication 1, and obtains the to-be-matched data block "55" in the payload part according to the position indicated by Indication 2. If the hash value of the to-be-matched data block "33" is a, node 21 uses the to-be-matched data block "33" as the duplicate data block corresponding to Indication 1. If the hash value of the to-be-matched data block "55" is b, node 21 uses the to-be-matched data block "55" as the duplicate data block corresponding to indication 2.

Possible implementation B2: Message 21 is obtained by a node located before node 21 on the transmission path by performing deduplication processing on the original message using possible implementation A2 in step 602. One or more repeated data blocks in the repeated content are located in the data part of the payload part of message 11.

In combination with the above-mentioned possible implementation method B2, the implementation process of node 21 obtaining the repeated content from the data set according to the indication information of the repeated content includes: node 21 determines the historical data block in the data set corresponding to the hash value indicated by the indication in the indication information as the repeated data block corresponding to the indication.

Optionally, if there is only one sampling position of the data portion pre-set in possible implementation manner A2 in step 602, the indication in the indication information of repeated content does not need to indicate the position of the corresponding repeated data block in the original message corresponding to message 21, and node 21 assumes that the repeated data block is located at the sampling position of the data portion of the original message corresponding to message 21. If there are multiple sampling positions of the data portion pre-set in possible implementation manner A2 in step 602, the indication in the indication information of repeated content is also used to indicate the position of the corresponding repeated data block in the data portion of the original message corresponding to message 21.

In combination with the possible implementation B2, the data set stored in the node 21 may include the protocol part of the message to which the historical data block belongs. The repeated content may also include protocol information located in the protocol part. Accordingly, the indication information of the repeated content also includes a difference indication. The difference indication is used to indicate the difference between the protocol part of the original message corresponding to the message 21 and the protocol part of the target message, and the target message is a historical message received by the node 21 from the node 22, in which the data part and the data part of the original message corresponding to the message 21 have the above-mentioned one or more repeated data blocks. In this case, the implementation process of the node 21 obtaining the repeated content from the data set according to the indication information of the repeated content also includes: the node 21 obtains the protocol part of the target message to which the one or more repeated data blocks belong from the data set.

Step 1004: Node 21 performs deduplication recovery processing on the payload of message 21 according to the repeated content to obtain message 22, wherein the payload of message 22 includes the repeated content.

Among them, message 22 is the original message corresponding to the above message 21.

In combination with the possible implementation method B1 in the above step 1003, the implementation method in which node 21 performs deduplication recovery processing on the payload part of message 21 based on the repeated content is as follows: for each indication in the indication information, node 21 adds the repeated data block corresponding to the indication at the position indicated by the indication in the payload part of message 21.

For example, referring to the example in the first possible implementation method in the above step 1003, the content of the payload part of message 21 includes "4466; Indication 1: <a, position 3-4>; Indication 2: <b, position 7-8>", assuming that the repeated data block corresponding to hash value a is "33", and the repeated data block corresponding to hash value b is "55", then node 21 inserts data block "33" between "44" and "66" to obtain "443366", so that data block "33" is located at the 3rd byte and 4th byte of the payload part of the restored message, and adds data block "55" after "443366" to obtain "44336655", so that data block "55" is located at the 7th byte and 8th byte of the payload part of the restored message.

In an embodiment of the present application, the bottom node can calculate the hash values of the data blocks at that position in the payload part of the multiple stored payload parts according to the position of the duplicate data block indicated by the indication carried in the deduplication message in the payload part of the original message, so as to obtain a storage data block whose hash value is consistent with the hash value indicated by the indication, and then add the storage data block to the position indicated by the indication to achieve data recovery of the deduplication message.

In combination with the possible implementation method B2 in the above step 1003, if the indication in the indication information of the repeated content is used to indicate the position of the corresponding repeated data block in the data part of the original message corresponding to the message 21, then the implementation method of node 21 performing deduplication recovery processing on the payload part of the message 21 according to the repeated content is: for each indication in the indication information, node 21 adds the repeated data block corresponding to the indication at the position indicated by the indication in the data part of the message 21. If the indication in the indication information of the repeated content does not indicate the position of the repeated data block in the data part of the original message corresponding to the message 21, then the implementation method of node 21 performing deduplication recovery processing on the payload part of the message 21 according to the repeated content is: node 21 adds the obtained repeated data block to the default deduplication position of the data part of the message 21, and the default deduplication position is the above-mentioned pre-set sampling position of the data part.

Optionally, if the indication information for the duplicate content includes a difference indication for the protocol part, the implementation process of node 21 performing deduplication recovery processing on the payload part of message 21 according to the duplicate content further includes: node 21 modifies the protocol part of the target message according to the difference indication, and uses the modified protocol part of the target message as the protocol part of message 22. For a specific explanation of the difference indication, reference may be made to the relevant content in the above step 602, which will not be repeated in detail in the embodiment of the present application.

In an embodiment of the present application, the bottom node can search for the hash value carried in the deduplication message in the hash value of the stored historical data block, and add the historical data block corresponding to the hit hash value to the data part of the deduplication message to achieve the recovery of the data part of the deduplication message. In addition, the bottom node can also obtain the protocol part of the historical message to which the historical data block corresponding to the hit hash value belongs, and restore the protocol part of the original message corresponding to the deduplication message in combination with the difference indication for the protocol part in the deduplication message to achieve the recovery of the protocol part of the deduplication message.

In the above step 1004 , when node 21 performs data recovery on message 21 , it may also delete the deduplication mark and the indication information of the duplicate content in message 21 to restore the original message corresponding to message 21 .

Step 1005 , node 21 sends message 22 to node 23 , and node 23 is the next hop of message 21 on node 21 .

Optionally, the node 23 may be the destination of the message 22 , or may be a network device on the transmission path of the message 22 that does not support data deduplication.

Due to network instability factors, messages may arrive at the bottom node out of order. For example, the bottom node may first receive a deduplicated message, and then receive a non-deduplicated message with duplicate content corresponding to the original message of the deduplicated message, and the non-deduplicated message carries the duplicate content indicated in the deduplicated message. In this case, the bottom node cannot immediately perform data recovery on the deduplicated message after receiving the deduplicated message, and needs to wait for the arrival of the non-deduplicated message carrying the duplicate content indicated in the deduplicated message, and then perform data recovery on the deduplicated message based on the payload of the non-deduplicated message.

Optionally, if the node 21 receives a non-deduplicated message, the node 21 directly forwards the non-deduplicated message. In the embodiment of the present application, the bottom node does not need to perform the message deduplication process.

In an embodiment of the present application, the bottom node can perform data recovery on deduplicated messages whose sender is the SFU server, so that the user can receive the original message carrying complete data content, thereby ensuring user services.

The third embodiment is applied to an intermediate node, and the intermediate node is used to perform deduplication processing or deduplication recovery processing on the message. For example, Figure 11 is a flow chart of another data transmission method 1100 provided in an embodiment of the present application. In order to facilitate the distinction between the contents of each embodiment, in method 1100, the first node is referred to as node 31, the second node is referred to as node 32, the third node is referred to as node 33, the first message is referred to as message 31, the second message is referred to as message 32, and the third message is referred to as message 33. The method 1100 can be applied to the data transmission system 400 shown in Figure 4, then the node 31 in the method 1100 is 402C, the node 32 is 402A, and the node 33 is 402E, node 402F or node 402G. As shown in Figure 11, the method 1100 includes but is not limited to the following steps 1101 to 1105.

Step 1101 , node 31 receives message 31 sent by node 32 , the sender of message 31 is the SFU server, and node 31 is an intermediate node supporting data deduplication on the transmission path of message 31 .

Optionally, message 31 can be a deduplicated message or a non-deduplicated message. The sender of message 31 is the SFU server, and node 31 is an intermediate node that supports data deduplication on the transmission path of message 31, that is, node 31 is an intermediate node that supports data deduplication on the transmission path with the SFU server as the starting point and the destination of message 31 as the end point. Optionally, the source IP address of message 31 can be the IP address of the SFU server, or it can be the address obtained after the IP address of the SFU server is NATed.

Step 1102: Node 31 determines whether message 31 is a deduplicated message or a non-deduplicated message.

The deduplication message in the embodiment of the present application carries a deduplication mark. Optionally, the deduplication mark is located in the payload part of the deduplication message. After receiving the message 31, the node 31 parses the payload part of the message 31. If the payload part of the message 31 carries the deduplication mark, the node 31 determines that the message 31 is a deduplication message. If the payload part of the message 31 does not carry the deduplication mark, the node 31 determines that the message 31 is a non-deduplication message.

Step 1103, if message 31 is a non-deduplicated message, node 31 determines whether there is a first original message in the historical messages sent to node 33, the payload part of the first original message and the payload part of message 31 have first repeated content, and node 33 is the next hop of message 31 on node 31.

Optionally, the first repetitive content includes one or more repetitive data blocks.

Possible implementation mode C1, a first data set is stored in node 31, and the first data set includes the payload part of the historical message sent by node 31 to node 33. The implementation process of node 31 judging whether there is a first original message in the historical message sent to node 33 includes: node 31 matches the payload part of message 31 with the payload part in the first data set. If there is a target payload part with a repeated data block with the payload part of message 31 in the first data set, node 31 determines that there is a first original message in the historical message sent to node 33. If there is no payload part with a repeated data block with the payload part of message 31 in the first data set, node 31 determines that there is no first original message in the historical message sent to node 33. Optionally, after node 31 determines that there is no first original message in the historical message sent to node 33, node 31 can add the payload part of message 31 to the first data set to obtain an updated first data set. The updated first data set can be used by node 31 to perform deduplication processing on the non-deduplication messages obtained subsequently and sent by the SFU server. For this implementation, reference may be made to possible implementation A1 in step 602 above.

Optionally, if the node 31 has multiple subordinate nodes, multiple node-level data sets may be stored in the node 31, each node-level data set is used to store the payload part of the historical message sent by the node 31 to a subordinate node, that is, the node 31 stores a data set for each subordinate node. Alternatively, the node 31 may store a global data set, which includes the corresponding relationship between the payload part and the node identifier, and the node identifier is used to indicate that the corresponding payload part is sent to the subordinate node indicated by the node identifier, that is, the node 31 stores a common data set for all subordinate nodes.

Possible implementation C2: a sampling label set is stored in the node 31, and the sampling label set includes a hash value of a historical data block. The historical data block is a data block obtained by sampling a preset position of a data part of a historical message sent by the node 31 to the node 33. The implementation process of the node 31 judging whether there is a first original message in the historical message sent to the node 33 includes: the node 31 samples a preset position of the data part of the message 31 to obtain a sampling data block. The node 31 calculates the hash value of the sampling data block. If the sampling label set includes the hash value of the sampling data block, the node 31 determines that there is a first original message in the historical message sent to the node 33. If the sampling label set does not include the hash value of the sampling data block, the node 31 determines that there is no first original message in the historical message sent to the node 33. Optionally, after determining that there is no first original message in the historical message sent to the node 33, the node 31 can add the hash value of the sampling data block to the sampling label set to obtain an updated sampling label set. The updated sampling tag set can be used by the node 31 to perform deduplication processing on the subsequently acquired non-deduplication messages whose sender is the SFU server. This implementation method can refer to the possible implementation method A2 in the above step 602.

Optionally, if the node 31 has multiple subordinate nodes, multiple node-level sampling label sets may be stored in the node 31, each node-level sampling label set corresponds to a subordinate node, and each node-level sampling label set is used to store the hash value of the data block sampled at the preset position of the data part of the historical message sent by the node 31 to the corresponding subordinate node. That is, the node 31 stores a sampling label set for each subordinate node. Alternatively, a global sampling label set may be stored in the node 31, and the global sampling label set includes the correspondence between the hash value and the node identifier, and the node identifier is used to indicate that the corresponding hash value comes from the historical message sent to the subordinate node indicated by the node identifier, that is, the node 31 stores a common sampling label set for all subordinate nodes.

Optionally, the sampling tag set stored in the node 31 also includes the historical data block indicated by the hash value, that is, the sampling tag set stored in the node 31 may include the correspondence between the historical data block and the hash value of the historical data block. Then the implementation method of the node 31 determining that the first original message exists in the historical message if the sampling tag set includes the hash value of the sampling data block includes: if the sampling tag set includes the hash value of the sampling data block, the node 31 performs content matching on the sampling data block and the historical data block indicated by the hash value of the sampling data block. If the sampling data block has the same content as the historical data block indicated by the hash value of the sampling data block, the node 31 determines that the first original message exists in the historical message.

Since the data contents of two data blocks with the same hash value may be different, by storing the correspondence between historical data blocks and hash values of historical data blocks in the sampling label set, node 31 can further perform content matching on the sampled data block and the historical data block after determining that the hash value of the sampled data block is the same as the hash value of a historical data block, so as to achieve an accurate match, thereby improving the accuracy of deduplication of the message.

Step 1104: If the first original message exists in the historical message sent by node 31 to node 33, node 31 deduplicates the payload part of message 31 to obtain message 32. Message 32 does not include the first repeated content, and message 32 carries a deduplication mark and first indication information for the first repeated content. The deduplication mark is used to indicate that message 32 is a deduplication message.

Optionally, the first repetitive content includes one or more repetitive data blocks, and the first indication information for the first repetitive content includes one or more indications, which correspond one to one with the one or more repetitive data blocks in the first repetitive content, and each indication is used to indicate a hash value of a corresponding repetitive data block.

In combination with the possible implementation C1 in the above step 1103, one or more repeated data blocks in the first repeated content are located in the data part and/or the protocol part of the payload part of the message 31. The first original message exists in the historical message sent by the node 31 to the node 33, that is, the target payload part having repeated data blocks with the payload part of the message 31 exists in the first data set. The implementation process of node 31 performing deduplication processing on the payload part of the message 31 includes: for each repeated data block between the payload part of the message 31 and the target payload part, the node 31 calculates the hash value of the repeated data block. The node 31 removes the repeated data block in the payload part of the message 31, and adds an indication corresponding to the repeated data block to the payload part of the message 31, the indication is used to indicate the hash value of the repeated data block and the position of the repeated data block in the payload part of the message 31. Here, the implementation process of node 31 performing deduplication processing on the payload part of the message 31 can refer to the implementation process of node 11 performing deduplication processing on the payload part of the message 11 under the possible implementation A1 in the above step 602.

In an embodiment of the present application, the intermediate node can perform content matching on the payload portion of the acquired non-deduplicated message and the payload portion of the stored historical message. If the payload portion of the non-deduplicated message and the payload portion of the historical message have duplicate data blocks, the intermediate node calculates the hash value of the duplicate data block, removes the duplicate data block from the non-deduplicated message to obtain a deduplicated message, and further carries the hash value of the duplicate data block and an indication of the location of the duplicate data block in the deduplicated message to achieve data deduplication of the non-deduplicated message.

In combination with the possible implementation C2 in the above step 1103, one or more repeated data blocks in the first repeated content are located in the data part of the payload part of the message 31. The implementation process of node 31 performing deduplication processing on the payload part of the message 31 includes: node 31 uses the sampling data block whose hash value belongs to the sampling tag set as a repeated data block, removes the repeated data block from the data part of the message 31, and adds an indication corresponding to the repeated data block to the payload part of the message 31, and the indication is used to indicate the hash value of the repeated data block. Here, the implementation process of node 31 performing deduplication processing on the payload part of the message 31 can refer to the implementation process of node 11 performing deduplication processing on the payload part of the message 11 under the possible implementation A2 in the above step 602.

In an embodiment of the present application, the intermediate node can calculate the hash value of the sampled data block at the preset position of the data portion of the acquired non-deduplicated message, and compare it with the hash value of the stored historical data block. If the hash value of a sampled data block in the non-deduplicated message is the same as the hash value stored by the intermediate node, the intermediate node removes the sampled data block in the non-deduplicated message to obtain a deduplicated message, and further carries the hash value of the sampled data block in the deduplicated message to achieve data deduplication of the non-deduplicated message. Compared with the possible implementation method C1 described above, the intermediate node does not need to perform content matching on the payload part of the non-deduplicated message and the historical message, thereby improving the efficiency of message deduplication.

In combination with the possible implementation C2 in step 1103, the node 31 may also store the protocol part of the historical message sent by the node 31 to the node 33. The first repetition of the payload part of the message 31 and the payload part of the first original message also includes the protocol part located in the message 31. The first indication information also includes a difference indication, which is used to indicate the difference between the protocol part of the message 31 and the protocol part of the first original message. Here, the implementation method of node 31 performing deduplication processing on the protocol part of the message 31 can refer to the implementation method of node 11 performing deduplication processing on the protocol part of the message 11 under the possible implementation method A2 in the above step 602.

In an embodiment of the present application, in addition to deduplicating data blocks at preset positions in the data part of the message, the intermediate node can also deduplicate the protocol part of the message. By carrying a difference indication in the payload part of the message to replace the protocol part, the amount of data transmitted in the message can be further reduced, thereby reducing the network bandwidth overhead.

Step 1105 , node 31 sends message 32 to node 33 .

In an embodiment of the present application, after receiving a non-deduplicated message whose sender is an SFU server, the intermediate node can determine whether to send a historical message whose overload part has duplicate content with the load part of the non-deduplicated message to the next hop of the non-deduplicated message. If the intermediate node sends a historical message whose overload part has duplicate content with the load part of the non-deduplicated message to the next hop of the non-deduplicated message, the intermediate node can perform data deduplication on the non-deduplicated message, and then send a deduplication message to the subordinate node. Since the data volume of the deduplication message is smaller than the data volume of the non-deduplication message, the data volume of the message transmission can be reduced, thereby reducing the network bandwidth overhead. When the intermediate node sends a deduplication message to the subordinate node, it only needs to ensure that the historical message with the content of the deduplication message removed relative to the original message is sent to the subordinate node, so as to ensure that there is a node on the subsequent transmission path that can perform data recovery on the deduplication message, so that the user finally receives the original message carrying the complete data content, and guarantees the user service.

Step 1106 : If the first original message does not exist in the historical messages sent by node 31 to node 33 , node 31 sends message 31 to node 33 .

Step 1107: If message 31 is a deduplicated message, message 31 carries second indication information for the second duplicate content, node 31 determines whether there is a second original message in the historical message sent to node 33, the payload part of the second original message includes the second duplicate content, and node 33 is the next hop of message 31 on node 31.

Optionally, the second repetitive content includes one or more repetitive data blocks, and the second indication information for the second repetitive content includes one or more indications, which correspond one to one with the one or more repetitive data blocks in the second repetitive content, and each indication is used to indicate a hash value of a corresponding repetitive data block.

Possible implementation method D1, message 31 is obtained by the node located before node 31 on the transmission path by performing deduplication processing on the original message using possible implementation method A1 in the above step 602. Each indication in the second indication information of the second repeated content is used to indicate the hash value of the corresponding repeated data block and the position of the corresponding repeated data block in the payload part of the original message corresponding to message 31. Node 31 stores a first data set, and the first data set includes the payload part of the historical message sent by node 31 to node 33. The implementation process of node 31 judging whether there is a second original message in the historical message sent to node 33 includes: for any indication in the second indication information, node 31 obtains the data block to be matched at the position of the payload part in the first data set according to the position indicated by the indication. Node 31 calculates the hash value of the data block to be matched. Node 31 determines the data block to be matched whose hash value is consistent with the hash value indicated by the indication as the repeated data block corresponding to the indication. If the first data set includes a payload portion including repeated data blocks corresponding to the respective indications in the second indication information, the node 31 determines that the second original message exists in the historical message sent to the node 33. The second original message is a historical message including a payload portion including repeated data blocks corresponding to the respective indications in the second indication information. If the first data set does not include a payload portion including repeated data blocks corresponding to the respective indications in the second indication information, the node 31 determines that the second original message does not exist in the historical message sent to the node 33.

Possible implementation D2, message 31 is obtained by the node located before node 31 on the transmission path by performing deduplication processing on the original message using possible implementation A2 in the above step 602. Each indication in the second indication information of the second repeated content is used to indicate the hash value of the corresponding repeated data block. Node 31 stores a sampling label set, which includes the hash value of the historical data block. The historical data block is a data block obtained by sampling the preset position of the data part of the historical message sent by node 31 to node 33. The implementation process of node 31 judging whether there is a second original message in the historical message sent to node 33 includes: if the sampling label set includes the hash value indicated by each indication in the second indication information, node 31 determines that there is a second original message in the historical message sent to node 33. The second original message, that is, the payload part includes the historical message of all historical data blocks corresponding to the hash value indicated by each indication in the second indication information. If the sampling label set does not include the hash value indicated by any indication in the second indication information, node 31 determines that there is no second original message in the historical message sent to node 33.

Step 1108 : If the second original message exists in the historical messages sent by node 31 to node 33 , node 31 sends message 31 to node 33 .

In the embodiment of the present application, after receiving the deduplication message sent by the SFU server, the intermediate node can determine whether to send a historical message with a payload portion carrying the content of the deduplication message removed from the original message to the next hop of the deduplication message. The next hop of the deduplication message sends a historical message with the content of the deduplication message removed relative to the original message over the payload part, and the intermediate node can directly forward the deduplication message to the lower-level node.

Step 1109: If the second original message does not exist in the historical message sent by node 31 to node 33, node 31 obtains the second repeated content from the second data set according to the second indication information, and the second data set includes at least part of the payload part of the historical message received by node 31 from node 32.

In combination with the possible implementation D1 in the above step 1107, each indication in the second indication information is used to indicate the hash value of the corresponding repeated data block and the position of the corresponding repeated data block in the payload part of the original message corresponding to the message 31. The second data set includes the payload part of the historical message received by the node 31 from the node 32. The implementation process of the node 31 obtaining the second repeated content from the second data set according to the second indication information includes: for each indication in the second indication information, the node 31 obtains the to-be-matched data block at the position of the payload part in the second data set according to the position indicated by the indication. The node 31 calculates the hash value of the to-be-matched data block. The node 31 determines the to-be-matched data block whose hash value is consistent with the hash value indicated by the indication as the repeated data block corresponding to the indication. Here, the implementation process of the node 31 obtaining the second repeated content from the second data set according to the second indication information can refer to the implementation process of the node 21 obtaining the repeated content from the data set according to the indication information of the repeated content under the possible implementation B1 in the above step 1003.

In combination with the possible implementation D2 in the above step 1107, one or more repeated data blocks in the second repeated content are located in the data part. The second data set includes the correspondence between the hash value of the historical data block and the historical data block, and the historical data block is a data block obtained by sampling the preset position of the data part of the historical message received by the node 31 from the node 32. The implementation process of node 31 obtaining the second repeated content from the second data set according to the second indication information includes: node 31 determines the historical data block in the second data set corresponding to the hash value indicated by the indication in the second indication information as the repeated data block corresponding to the indication. Here, the implementation process of node 31 obtaining the second repeated content from the second data set according to the second indication information can refer to the implementation process of node 21 obtaining the repeated content from the data set according to the indication information of the repeated content under the possible implementation B2 in the above step 1003.

In combination with the possible implementation D2 in the above step 1107, the second data set may also include the protocol part of the message to which the historical data block belongs. The second repeated content may also include protocol information located in the protocol part. Accordingly, the second indication information also includes a difference indication, which is used to indicate the difference between the protocol part of the original message corresponding to the message 31 and the protocol part of the target message. The target message is a historical message in which the data part of the historical message received by the node 31 from the node 32 and the data part of the original message corresponding to the message 31 have the above one or more repeated data blocks. In this case, the implementation process of the node 31 obtaining the second repeated content from the second data set according to the second indication information also includes: the node 31 obtains the protocol part of the target message to which the one or more repeated data blocks belong from the second data set.

Step 1110: Node 31 performs deduplication recovery processing on the payload of message 31 according to the second repeated content to obtain message 33, where the payload of message 33 includes the second repeated content.

Here, message 31 is a deduplicated message, and message 33 is the original message corresponding to message 31 .

In combination with the possible implementation D1 in the above step 1109, the implementation of node 31 performing deduplication recovery processing on the payload part of message 31 according to the second repeated content is as follows: for each indication in the second indication information, node 31 adds the repeated data block corresponding to the indication at the position indicated by the indication in the payload part of message 31. This implementation can refer to the implementation of node 21 performing deduplication recovery processing on the payload part of message 21 according to the repeated content under the possible implementation B1 in the above step 1004.

In an embodiment of the present application, the intermediate node can calculate the hash values of the data blocks at that position in the payload part of multiple stored payload parts according to the position of the duplicate data block indicated by the indication carried in the deduplication message in the payload part of the original message, so as to obtain a storage data block whose hash value is consistent with the hash value indicated by the indication, and then add the storage data block to the position indicated by the indication to achieve data recovery of the deduplication message.

In combination with the possible implementation method D2 in the above step 1109, if the indication in the second indication information is used to indicate the position of the corresponding duplicate data block in the data part of the original message corresponding to the message 31, then the implementation method of node 31 performing deduplication recovery processing on the payload part of the message 31 according to the second duplicate content is: for each indication in the second indication information, node 31 adds the duplicate data block corresponding to the indication at the position indicated by the indication in the data part of the message 31. If the indication in the second indication information does not indicate the position of the duplicate data block in the data part of the original message corresponding to the message 31, then node 31 adds the obtained duplicate data block to the default deduplication position of the data part of the message 31, and the default deduplication position is the sampling position of the data part pre-set above. This implementation method can refer to the possible implementation method B2 in the above step 1004, in which node 21 performs deduplication recovery processing on the payload part of the message 21 according to the duplicate content.

Optionally, if the second indication information includes a difference indication for the protocol part, the implementation process of node 31 performing deduplication recovery processing on the payload part of message 31 according to the second repeated content also includes: node 31 modifies the protocol part of the target message according to the difference indication, and uses the modified protocol part of the target message as the protocol part of message 33.

In an embodiment of the present application, the intermediate node can search for the hash value carried in the deduplication message among the hash values of the stored historical data blocks, and add the historical data block corresponding to the hit hash value to the data portion of the deduplication message to achieve recovery of the data portion of the deduplication message. In addition, the intermediate node can also obtain the protocol portion of the historical message to which the historical data block corresponding to the hit hash value belongs, and restore the protocol portion of the original message corresponding to the deduplication message in combination with the difference indication for the protocol portion in the deduplication message to achieve recovery of the protocol portion of the deduplication message.

In the above step 1110 , when node 31 performs data recovery on message 31 , it may also delete the deduplication mark and the second indication information in message 31 to restore the original message corresponding to message 31 .

Step 1111 , node 31 sends message 33 to node 33 .

Optionally, the node 33 is a network device on the transmission path of the message 33 .

Due to network instability factors, messages may arrive at the intermediate node out of order. For example, the intermediate node may first receive a deduplication message, and then receive a non-deduplication message with duplicate content corresponding to the original message of the deduplication message. The non-deduplication message carries the duplicate content indicated in the deduplication message. In this case, the intermediate node needs to perform order-preserving processing on the out-of-order messages. After receiving the deduplication message, the intermediate node first caches the deduplication message. After the non-deduplication message carrying the duplicate content indicated in the deduplication message arrives, the intermediate node further determines whether to directly forward the deduplication message or perform data recovery on the deduplication message based on the payload part of the non-deduplication message.

In an embodiment of the present application, the intermediate node can perform data deduplication on the non-deduplicated message whose sender is the SFU server, so as to send the deduplication message to the subordinate node. Since the data volume of the deduplication message is smaller than the data volume of the non-deduplicated message, the message transmission data volume can be reduced, thereby reducing the network bandwidth overhead. In addition, the intermediate node can also perform data recovery on the deduplication message whose sender is the SFU server, or directly forward the deduplication message to the subordinate node. When the intermediate node sends a deduplication message to the subordinate node, it only needs to ensure that the historical message carrying the content of the deduplication message removed relative to the original message is sent to the subordinate node in the overload part, so as to ensure that there is a node on the subsequent transmission path that can perform data recovery on the deduplication message, so that the user finally receives the original message carrying the complete data content, thereby ensuring user services.

Optionally, an embodiment of the present application also proposes a traffic grouping scheme for an SFU communication architecture. Traffic grouping is used by nodes to determine which flows can be used as a group of flows, and then to perform data deduplication and/or data recovery on the messages in the same group of flows to improve message processing efficiency. In the traffic grouping scheme for the SFU communication architecture, group identification can be performed by the bottom node and the middle node, that is, the deduplication-capable flows are actively identified, and the deduplication-capable flows are grouped. After the grouping is completed, the bottom node and the middle node can send the grouping results to the upper node respectively, and the upper node can perform data deduplication or data recovery on the messages according to the grouping results. The top node does not need to perform group identification. The specific implementation methods of applying the traffic grouping scheme to the top node, the bottom node, and the middle node during data transmission are described below.

For the top node, for example, node 11 in method 600 shown in FIG. 6 . One or more flow grouping sets corresponding to node 12 may be stored in node 11. Each flow grouping set includes flow identifiers of multiple flows flowing through node 12. A flow grouping set indicates a packet, and the flows indicated by the flow identifiers in a flow grouping set belong to the same packet. The flow grouping set stored in node 11 is the flow grouping set corresponding to the lower-level node. In an embodiment of the present application, the flow grouping set corresponding to the stored lower-level node may be referred to as a lower-level flow grouping set. The flow identifier may be represented by one or more of the five-tuple information of the flow. The five-tuple information includes the source IP address, the destination IP address, the source port, the destination port, and the transport layer protocol.

In the case where one or more flow grouping sets corresponding to node 12 are stored in node 11, after node 11 obtains message 11, node 11 may first determine whether the flow grouping set corresponding to node 12 includes the flow identifier of the flow to which message 11 belongs. If there is a target flow grouping set including the flow identifier of the flow to which message 11 belongs in the flow grouping set corresponding to node 12, node 11 executes the above steps 602 and 603. If all flow grouping sets corresponding to node 12 do not include the flow identifier of the flow to which message 11 belongs, node 11 sends message 11 to node 12.

Among them, node 11 determines whether the flow grouping set corresponding to node 12 includes the flow identifier of the flow to which message 11 belongs, that is, node 11 determines whether the flow to which message 11 belongs has been added to a certain group corresponding to node 12. Optionally, node 11 can search the flow grouping set corresponding to node 12 according to the five-tuple information of message 11 to determine whether the flow to which message 11 belongs has been added to a certain group corresponding to node 12. For messages in the flow that has been added to the group, node 11 further determines whether to perform data deduplication processing on the message, and for messages in the flow that has not been added to the group, node 11 directly forwards the message.

Optionally, node 11 receives the grouping information sent by node 12, and the grouping information includes the correspondence between the node identification of node 12 and one or more stream grouping sets corresponding to node 12. In the case where node 11 has multiple subordinate nodes, node 11 stores the correspondence between the node identification of the subordinate node and the subordinate stream grouping set. In the case where node 11 has only one subordinate node, node 11 may only store the subordinate stream grouping set. The node identification may be information such as an IP address, a Media Access Control (MAC) address or a hardware address of the node that can uniquely identify the node in a communication network.

Optionally, the grouping information sent by node 12 to node 11 may also include one or more grouping identifiers corresponding to one or more stream grouping sets in the grouping information. Each grouping identifier is used to uniquely identify a stream grouping set corresponding to node 12. For example, the grouping identifier may be a number that node 12 assigns to a stream grouping set within the node and that can uniquely identify the stream grouping set. Optionally, in a video conference scenario, the grouping information may also include a conference identifier such as a conference number. In a live broadcast scenario, the grouping information may also include a live broadcast identifier such as a live broadcast room number.

Optionally, the grouping information is carried in a join message. That is, node 12 can send the grouping information to node 11 via a join message. After receiving the join message, node 11 stores the grouping information carried in the join message in a grouping table. Afterwards, node 11 can also send a join response message to node 12 so that node 12 confirms that the grouping is successful. Further, after receiving the join response message, node 12 can also send a join response confirmation message to node 11.

It is worth noting that the grouping information in the grouping table stored in the top node all comes from the lower-level nodes, and the grouping table can be called a lower-level grouping table. For example, the grouping table stored in the top node can be as shown in Table 1.

Table 1

Referring to Table 1, the grouping table uses the flow identifier as the key, the node identifier and the group identifier as the value. Among them, the node identifier A is used to identify a subordinate node A of the top node, and the node identifier B is used to identify another subordinate node B of the top node. Group identifier A1 is used to identify a flow grouping set of the subordinate node A, and group identifier A2 is used to identify another flow grouping set of the subordinate node A. Group identifier B1 is used to identify the flow grouping set of the subordinate node B. Based on Table 1, it can be seen that flow identifier 1 and flow identifier 2 belong to a flow grouping set corresponding to the subordinate node A, flow identifier 3 and flow identifier 4 belong to another flow grouping set corresponding to the subordinate node A, and flow identifier 5 and flow identifier 6 belong to the flow grouping set corresponding to the subordinate node B. After receiving the message, the top node can search the key column according to the flow identifier of the flow to which the message belongs. If the key column does not have the flow identifier of the flow to which the message belongs, the top node directly forwards the message. If the key column has the flow identifier of the flow to which the message belongs, the top node further determines whether to perform data deduplication on the message.

In the embodiment of the present application, the top node can determine whether it is necessary to perform deduplication processing on the message sent to the lower node according to the flow grouping set corresponding to the lower node. If the flow identifier of the flow to which the message belongs is not in the flow grouping set corresponding to the lower node, the top node directly forwards the message to the lower node without executing the message deduplication process, which can reduce the processing overhead of the top node.

Optionally, when there is a target flow grouping set including the flow identifier of the flow to which message 11 belongs in the flow grouping set corresponding to node 12, when node 11 determines whether there is a target message whose payload part has duplicate content with the payload part of message 11 in the historical messages sent to node 12, it can only determine whether there is a target message in the target historical messages sent to node 12, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set. In this way, when there are multiple flow grouping sets corresponding to node 12, node 11 only needs to determine the duplicate content of the payload part of the historical messages belonging to multiple flows indicated by one flow grouping set and the payload part of message 11, which reduces the number of historical messages that node 11 needs to determine, thereby reducing the processing overhead of node 11, while improving the message processing efficiency of node 11, thereby improving the message transmission efficiency.

Optionally, node 11 uses the flow identifier of the flow to which message 11 belongs as a key, performs key value matching in the grouping table shown in Table 1, and determines that the flow identifier of the flow to which message 11 belongs belongs to the target flow grouping set corresponding to node 12.

In conjunction with the possible implementation A1 in the above step 602, a data set is stored in the node 11, and the data set includes a payload portion of a historical message sent by the node 11 to a subordinate node.

In the first implementation, node 11 stores a common global data set for all lower-level nodes, and the global data set includes the payload part of the historical message sent by node 11 to all lower-level nodes. The global data set may include the correspondence between the payload part, the node identifier and the group identifier. Here, the node identifier can be used to indicate that the historical message to which the corresponding payload part belongs is sent to the lower-level node indicated by the node identifier, and the group identifier can be used to indicate that the flow identifier of the flow to which the historical message to which the corresponding payload part belongs belongs is in the group identifier. The stream grouping set is shown.

For example, the global data set stored in the top node may be as shown in Table 2.

Table 2

Referring to Table 2, the global data set uses the node identifier and the group identifier as the key and the payload part as the value. Among them, the node identifier A is used to identify a subordinate node A of the top node, and the node identifier B is used to identify another subordinate node B of the top node. The group identifier A1 is used to identify a flow packet set of the subordinate node A, and the group identifier A2 is used to identify another flow packet set of the subordinate node A. The group identifier B1 is used to identify the flow packet set of the subordinate node B. Based on Table 2, it can be seen that payload part 1, payload part 2 and payload part 3 come from historical messages sent by the top node to the subordinate node A and belong to a group. Payload part 4 and payload part 5 come from historical messages sent by the top node to the subordinate node A and belong to another group. Payload part 6 and payload part 7 come from historical messages sent by the top node to the subordinate node B and belong to the same group. Under this implementation method, after receiving the message, the top node can determine the node identifier and group identifier corresponding to the message based on Table 1, and then use the corresponding node identifier and group identifier as the key to perform key value matching in Table 2 to obtain the corresponding payload part, and further determine whether the payload part of the message and the obtained payload part have duplicate content.

For example, if the flow identifier is five-tuple information, node 11 uses the five-tuple information of message 11 as a key, performs key-value matching in the grouping table shown in Table 1, and determines that message 11 corresponds to node identifier A and group identifier A2. Then node 11 uses node identifier A and group identifier A2 as keys, performs key-value matching in the global data set shown in Table 2, and obtains payload part 4 and payload part 5. Further, node 11 performs content matching on the payload part of message 11 with payload part 4 and payload part 5, respectively. If the payload part of message 11 has duplicate content with payload part 4 or payload part 5, node 11 determines that the target message exists in the historical message sent to node 12, and the target message includes the historical message of payload part 4 or payload part 5 that has duplicate content with the payload part of message 11.

In a second implementation, the node 11 stores a node-level data set for each subordinate node, each node-level data set includes the payload part of the historical message sent by the node 11 to a subordinate node. The node-level data set may include the correspondence between the payload part and the grouping identifier.

For example, referring to the example shown in Table 2, the top node may store a node-level data set corresponding to the lower-level node A and a node-level data set corresponding to the lower-level node B. The node-level data set corresponding to the lower-level node A may be as shown in Table 3, and the node-level data set corresponding to the lower-level node B may be as shown in Table 4.

table 3

Table 4

Under this implementation method, after receiving the message, the top node can determine the node identifier and group identifier corresponding to the message based on Table 1, and then find the node-level data set corresponding to the node identifier, and then use the group identifier as the key to perform key-value matching in the node-level data set shown in, for example, Table 3 or Table 4 to obtain the corresponding payload part, and further determine whether the payload part of the message and the obtained payload part have duplicate content.

In a third implementation, the node 11 stores a packet-level data set for each flow packet set, each packet-level data set including the payload portion of historical messages belonging to multiple flows indicated by a flow packet set sent by the node 11 to a lower-level node. Each packet-level data set includes the payload portion corresponding to a flow packet set.

Under this implementation method, after receiving the message, the top node can determine the node identifier and group identifier corresponding to the message based on Table 1, and then find the group-level data set corresponding to the node identifier and the group identifier, and then obtain the payload part in the group-level data set, and further determine whether the payload part of the message and the obtained payload part have duplicate content.

In combination with possible implementation A2 in step 602 , a sampling tag set including a hash value of a historical data block is stored in node 11 . The historical data block is a data block obtained by sampling a preset position of the data part of a historical message sent by node 11 to node 12 .

In a first implementation, the node 11 stores a common global sampling label set for all lower-level nodes, and the global sampling label set includes a hash value of a data block sampled at a preset position of a data portion of a historical message sent by the node 11 to all lower-level nodes. The global sampling set may include a correspondence between a hash value, a node identifier, and a group identifier. Here, the node identifier may be used to indicate that the corresponding hash value is obtained based on a historical message sent to a lower-level node indicated by the node identifier, and the group identifier may be used to indicate that the flow identifier of the flow to which the historical message to which the corresponding hash value originates belongs is in the flow group set indicated by the group identifier.

For example, the global sampling label set stored in the top node may be as shown in Table 5.

table 5

Referring to Table 5, the global sampling label set uses the node identifier and the group identifier as the key and the hash value as the value. Among them, the node identifier A is used to identify a subordinate node A of the top node, and the node identifier B is used to identify another subordinate node B of the top node. The group identifier A1 is used to identify a flow grouping set of the subordinate node A, and the group identifier A2 is used to identify another flow grouping set of the subordinate node A. The group identifier B1 is used to identify the flow grouping set of the subordinate node B. In this implementation mode, after receiving the message, the top node can determine the node identifier and group identifier corresponding to the message based on Table 1, and then use the corresponding node identifier and group identifier as the key to perform key value matching in Table 5 to obtain the corresponding hash value, and further determine whether the obtained hash value includes the hash value of the sampled data block sampled from the preset position of the data part of the message 11.

For example, if the flow identifier is five-tuple information, node 11 uses the five-tuple information of message 11 as a key, performs key value matching in the grouping table shown in Table 1, and determines that message 11 corresponds to node identifier A and group identifier A2. Then node 11 uses node identifier A and group identifier A2 as keys, performs key value matching in the global sampling label set shown in Table 5, and obtains hash value 4 and hash value 5. Node 11 calculates the hash value of the sampled data block sampled from the preset position of the data part of message 11. If the hash value of the sampled data block is hash value 4 or hash value 5, node 11 determines that the target message exists in the historical messages sent to node 12.

In a second implementation, the node 11 stores a node-level sampling label set for each subordinate node, each node-level sampling label set including a hash value of a data block sampled at a preset position of a data portion of a historical message sent by the node 11 to a subordinate node. The node-level sampling label set may include a correspondence between a hash value and a grouping identifier.

For example, referring to the example shown in Table 5, the top node may store a node-level sampling label set corresponding to the lower node A and a node-level sampling label set corresponding to the lower node B. The node-level sampling label set corresponding to the lower node A may be as shown in Table 6, and the node-level sampling label set corresponding to the lower node B may be as shown in Table 7.

Table 6

Table 7

In this implementation mode, after receiving the message, the top node can determine the node identifier and group identifier corresponding to the message based on Table 1, and then find the node-level sampling label set corresponding to the node identifier, and then use the group identifier as the key to perform key value matching in the node-level sampling label set shown in, for example, Table 6 or Table 7 to obtain the corresponding hash value, and further determine whether the obtained hash value includes the hash value of the sampling data block sampled from the preset position of the data part of the message 11.

In a third implementation, the node 11 stores a group-level sampling label set for each flow group set, each group-level sampling label set including a hash value of a data block sampled at a preset position of a data portion of a historical message belonging to multiple flows indicated by a flow group set and sent by the node 11 to a lower-level node. Each group-level sampling set includes a hash value corresponding to a flow group set.

In this implementation mode, after receiving the message, the top node can determine the node identifier and group identifier corresponding to the message based on Table 1, and then find the group-level sampling label set corresponding to the node identifier and the group identifier, and then obtain the hash value in the group-level sampling label set, and further determine whether the obtained hash value includes the hash value of the sampling data block sampled from the preset position of the data part of the message 11.

For the bottom node, for example, the node 21 in the method 1000 shown in FIG. 10 . One or more flow grouping sets may be stored in the node 21. Each flow grouping set includes flow identifiers of multiple flows flowing through the node 21. The flow grouping set stored in the node 21 is the flow grouping set corresponding to the node 21 itself. In the embodiment of the present application, the flow grouping set corresponding to the node itself may be referred to as a local flow grouping set.

Optionally, before node 21 determines that message 21 is a deduplicated message based on the deduplication mark, that is, before the above step 1002 is executed, node 21 determines that there is a target flow group set including the flow identifier of the flow to which message 21 belongs in the one or more flow group sets. That is, after receiving message 21, node 21 first determines whether the flow group set corresponding to node 21 includes the flow identifier of the flow to which message 21 belongs. Node 21 will parse the payload part of message 21 only after determining that the flow identifier of the flow to which message 21 belongs belongs to a certain flow group set corresponding to node 21. The implementation method of node 21 determining whether the flow group set corresponding to node 21 includes the flow identifier of the flow to which message 21 belongs can refer to the implementation method of node 11 determining whether the flow group set corresponding to node 12 includes the flow identifier of the flow to which message 11 belongs.

Optionally, the node 21 receives the message 23 , and the flow identifier of the flow to which the message 23 belongs does not belong to any flow grouping set corresponding to the node 21 . The node 21 forwards the message 23 .

In the embodiment of the present application, since the upper-level node will only perform data deduplication on the messages in the flow indicated by the flow grouping set corresponding to the lower-level node, after receiving the message, the bottom node can first determine whether the flow identifier of the flow to which the message belongs belongs to a certain flow grouping set corresponding to itself. If the flow identifier of the flow to which the message belongs belongs to a certain flow grouping set corresponding to itself, it means that the message may be a deduplicated message that has been deduplicated by the upper-level node, and the bottom node needs to further parse the payload part of the message to determine whether the message is a deduplicated message. If the flow identifier of the flow to which the message belongs does not belong to any flow grouping set corresponding to itself, it means that the upper-level node will not perform data deduplication on the message. That is, the message cannot be a deduplicated message, so the bottom node can directly forward the message without parsing the payload part of the message to determine whether the message is a deduplicated message. This can reduce the processing overhead of the bottom node.

Optionally, node 21 receives message 24, and the flow identifier of the flow to which message 24 belongs belongs to the flow group set corresponding to node 21. Node 21 parses the payload part of message 24 and determines that the payload part of message 24 does not carry a deduplication mark, that is, message 24 is a non-deduplicated message. Node 21 adds at least part of the content of the payload part of message 24 to the data set and forwards message 24. Message 24 here can be regarded as the first packet in a group of deduplicated messages received by node 21. Node 21 stores at least part of the content of the payload part of message 24 in the data set so as to perform data recovery on the deduplicated messages received subsequently.

Optionally, node 21 adds the flow identifiers of the flows to which the messages with duplicate contents in the payload part of the received multiple messages belonging to different flows belong to the same flow grouping set, and the sender of these different flows is the same SFU server. For example, after receiving message 23, node 21 can store at least the contents of the payload part of message 23, and then after receiving a message whose sender is the same SFU server as the sender of message 23 and belongs to a different flow from message 23, determine whether there are duplicate contents in the payload part of the message and the payload part of message 23, and if so, determine that the flow to which the message belongs and the flow to which message 23 belongs can form a new group, and then generate a new flow grouping set, which includes the flow identifier of the flow to which the message belongs and the flow identifier of the flow to which message 23 belongs.

Further, node 21 may send grouping information to node 22, and the grouping information includes a correspondence between the node identifier of node 21 and one or more stream grouping sets corresponding to node 21. When a stream grouping set corresponding to node 21 is updated, for example, a new stream identifier is added to a stream grouping set, node 21 may send the new stream grouping set to node 22. Alternatively, node 21 may also send the grouping identifier corresponding to the updated stream grouping set and the new stream identifier to node 22, so that node 22 adds the new stream identifier to the stream grouping set indicated by the grouping identifier, thereby realizing synchronous update of the stream grouping set.

Optionally, the grouping information is carried in a join message. That is, node 21 can send the grouping information to node 22 via a join message. After receiving the join message, node 22 stores the grouping information carried in the join message in a grouping table. Afterwards, node 22 can also send a join response message to node 21 so that node 21 confirms that the grouping is successful. Further, after receiving the join response message, node 21 can also send a join response confirmation message to node 22.

In combination with the possible implementation manner B1 in the above step 1003 , the data set stored in the node 21 includes the payload part of the non-deduplicated message received by the node 21 .

In this implementation, for a flow whose sender is an SFU server and whose flow identifier does not belong to any flow grouping set corresponding to the bottom node, after the bottom node receives the message in the flow, it stores the payload part of the message in the flow. Afterwards, if the bottom node receives a message in another flow whose sender is the SFU server and whose flow identifier does not belong to any flow grouping set corresponding to the bottom node, the bottom node performs content matching on the payload parts of the messages of the two flows. If there is duplicate content in the payload parts of the messages of the two flows, it means that the two flows are matched successfully, and a flow grouping set including the flow identifiers of the two flows is further generated.

In combination with possible implementation B2 in step 1003, the data set stored in node 21 includes a correspondence between a hash value of a data block sampled at a preset position of a data portion of a non-deduplicated message received by node 21 and the data block.

In this implementation, for a flow whose sender is an SFU server and whose flow identifier does not belong to any flow grouping set corresponding to the bottom node, after the bottom node receives the message in the flow, the preset position of the data part of the message is sampled to obtain a sampled data block, and the hash value of the sampled data block is calculated and stored. Afterwards, if the bottom node receives a message in another flow whose sender is the SFU server and whose flow identifier does not belong to any flow grouping set corresponding to the bottom node, the bottom node also samples the preset position of the data part of the message to obtain a sampled data block, and calculates the hash value of the sampled data block. If the hash values of the sampled data blocks obtained from the messages of the two flows are the same, it means that the two flows are matched successfully, and a flow grouping set including the flow identifiers of the two flows is further generated. In order to improve the accuracy of the flow grouping, in this implementation, the bottom node can also store the content of the sampled data block. When the hash values of the sampled data blocks of two messages belonging to different flows are the same, the bottom node can further accurately match the data content of the sampled data blocks of the two messages to determine whether the data content of the sampled data blocks of the two messages is the same.

In the case where one or more flow grouping sets are stored in the node 21, the implementation of the above step 1003 may be: the node 21 obtains the repeated content from the payload content of the target historical message in the data set according to the indication information of the repeated content, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set. In this way, when there are multiple flow grouping sets corresponding to the node 21, the node 21 only needs to search for the payload part of the historical messages belonging to multiple flows indicated by one flow grouping set to obtain the repeated content, which reduces the number of historical messages that the node 21 needs to retrieve, thereby reducing the processing overhead of the node 21, and at the same time improving the message processing efficiency of the node 21, thereby improving the message transmission efficiency.

For an intermediate node, for example, the node 31 in the method 1100 shown in FIG. 11 , one or more local flow grouping sets may be stored in the node 31, each of which includes flow identifiers of multiple flows flowing through the node 31. And/or, the node 31 may store one or more subordinate flow grouping sets corresponding to the node 33, each of which includes flow identifiers of multiple flows flowing through the node 33.

In the case where one or more local flow grouping sets are stored in node 31, before node 31 determines whether message 31 is a deduplicated message or a non-deduplicated message, that is, before the above step 1102 is executed, node 31 determines that there is a target flow grouping set including the flow identifier of the flow to which message 31 belongs in the one or more local flow grouping sets. That is, after receiving message 31, node 31 first determines whether the local flow grouping set includes the flow identifier of the flow to which message 31 belongs. After determining that the flow identifier of the flow to which message 31 belongs belongs to a certain local flow grouping set, node 31 parses the payload part of message 31. The implementation method of node 31 determining whether the local flow grouping set includes the flow identifier of the flow to which message 31 belongs can refer to the implementation method of node 11 determining whether the flow grouping set corresponding to node 12 includes the flow identifier of the flow to which message 11 belongs.

In the embodiment of the present application, since the upper node will only perform data deduplication on the message in the flow indicated by the flow grouping set corresponding to the lower node, the intermediate node can first determine whether the flow identifier of the flow to which the message belongs belongs to a certain flow grouping set corresponding to itself after receiving the message. If the flow identifier of the flow to which the message belongs belongs to a certain flow grouping set corresponding to itself, it means that the message may be a deduplication message that has been deduplication processed by the upper node, and the intermediate node needs to further parse the payload part of the message to determine whether the message is a deduplication message. If the flow identifier of the flow to which the message belongs does not belong to any flow grouping set corresponding to itself, it means that the upper node will not perform data deduplication on the message, that is, the message cannot be a deduplication message, so the intermediate node can directly forward the message without parsing the payload part of the message to determine whether the message is a deduplication message. This can reduce the processing overhead of the intermediate node.

Optionally, node 31 adds the flow identifier of the flow to which the message with repeated content in the payload part belongs among the multiple messages received belonging to different flows to the same local flow grouping set, and the sender of these different flows is the same SFU server. Further, node 31 can send first grouping information to node 32, and the first grouping information includes the correspondence between the node identifier of node 31 and one or more local flow grouping sets. The generation method, sending method and function of the local flow grouping set stored in node 31 can refer to the generation method, sending method and function of the flow grouping set corresponding to node 21 stored in the above-mentioned node 21, respectively. The content in the second data set stored in node 31 can also refer to the content in the data set stored in node 21, and the embodiments of the present application will not be repeated here.

In the case where one or more lower-level flow grouping sets corresponding to node 33 are stored in node 31, after node 31 obtains message 31, if message 31 is a non-deduplicated message, node 31 may first determine whether the lower-level flow grouping set corresponding to node 33 includes the flow identifier of the flow to which message 31 belongs. If there is a target flow grouping set including the flow identifier of the flow to which message 31 belongs in the lower-level flow grouping set corresponding to node 33, node 31 executes the above step 1103, that is, node 31 determines whether there is a message in the historical message sent to node 33 whose payload part has duplicate content with the payload part of message 31. If all flow grouping sets corresponding to node 33 do not include the flow identifier of the flow to which message 31 belongs, node 31 sends message 31 to node 33.

In the embodiment of the present application, the intermediate node can determine whether it is necessary to perform deduplication processing on the received non-deduplication message that needs to be sent to the subordinate node based on the subordinate flow group set corresponding to the subordinate node. If the flow identifier of the flow to which the non-deduplication message belongs is not in the subordinate flow group set corresponding to the subordinate node, then the intermediate node directly forwards the non-deduplication message to the subordinate node without executing the message deduplication process, which can reduce the processing overhead of the intermediate node.

Optionally, in the case where there is a target flow grouping set including the flow identifier of the flow to which message 31 belongs in the lower-level flow grouping set corresponding to node 33, when node 31 determines whether there is a first original message whose payload part has first repetitive content with the payload part of message 31 in the historical message sent to node 33, it can only determine whether there is a first original message in the target historical message sent to node 33, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set. In this way, when there are multiple lower-level flow grouping sets corresponding to node 33, node 31 only needs to perform repetitive content determination on the payload part of the historical messages belonging to multiple flows indicated by one lower-level flow grouping set and the payload part of message 31, which reduces the number of historical messages that node 31 needs to determine, thereby reducing the processing overhead of node 31, while improving the message processing efficiency of node 31, thereby improving the message transmission efficiency.

Optionally, node 31 receives second grouping information sent by node 33, and the second grouping information includes a correspondence between a node identifier of node 33 and one or more lower-level flow grouping sets corresponding to node 33. The acquisition method and function of the lower-level flow grouping set corresponding to node 33 stored in node 31 can refer to the acquisition method and function of the flow grouping set corresponding to node 12 stored in the above-mentioned node 11, and the content in the first data set or sampling label set stored in node 31 can also correspond to the content in the data set or sampling label set stored in the reference node 11, which will not be described in detail in the embodiments of the present application.

The top node, middle node and bottom node defined in the embodiment of the present application are used to distinguish the positions of different nodes relative to the SFU server, and the functions of these nodes may be the same. Optionally, the positions of multiple nodes in the data transmission system relative to the SFU server may be manually configured. Alternatively, multiple nodes in the data transmission system may respectively execute the node discovery process to determine their own positions relative to the SFU server and the connection relationships with other nodes to achieve automated deployment.

In an embodiment of the present application, a message whose sender is an SFU server or whose destination is an SFU server can trigger a node in a data transmission system to execute a node discovery process. The source port number of the message whose sender is an SFU server is the SFU service port number, and the node can judge whether the sender of the message is an SFU server based on the source port number of the message. The destination port number of the message whose destination is an SFU server is the SFU service port number, and the node can judge whether the destination of the message is an SFU server based on the destination port number of the message. The specific implementation methods of executing the node discovery process for the top node, the bottom node, and the intermediate node are described below.

For the top node, for example, the node 11 in the method 600 shown in FIG6 . If the node 11 is an SFU server, and the SFU server is configured with a function that supports data deduplication, the node 11 directly determines itself as the top node. If the node 11 is not an SFU server, the node 11 can determine its position relative to the SFU server through the following two possible implementations.

In one possible implementation, a message destined for an SFU server triggers the node discovery process.

In this implementation, after node 11 receives message 13 whose destination port number is the SFU service port number, node 11 can send node discovery message 11 to node 13. Node 13 is the next hop of message 13 on node 11. The destination of message 13 is the SFU server, and node discovery message 11 carries the identifier of the SFU server, and node discovery message 11 indicates that node 11 is the subordinate node of node 13 on the transmission path starting from the SFU server. In response to not receiving the node discovery response message 11 corresponding to the node discovery message 11 sent by node 13, node 11 determines that node 11 is the first node that supports data deduplication on the transmission path starting from the SFU server. Among them, the identifier of the SFU server can be the IP address of the SFU server, or the address obtained after the IP address of the SFU server is NATed.

Optionally, there are multiple ways for the node discovery message 11 to indicate that the node 11 is a subordinate node of the node 13 on the transmission path starting from the SFU server. For example, if the destination port number of the node discovery message 11 is the SFU service port number, and the node discovery message 11 carries the identifier of the SFU server, then it means that the node 11 that sends the node discovery message 11 is a subordinate node of the node 13 that receives the node discovery message 11 on the transmission path starting from the SFU server. For another example, if the node discovery message 11 carries a position indication for indicating that it is a subordinate node, then the position indication carried by the node discovery message 11 and the identifier of the SFU server jointly indicate that the node 11 that sends the node discovery message 11 is a subordinate node of the node 13 that receives the node discovery message 11 on the transmission path starting from the SFU server.

Optionally, node 11 generates node discovery message 11 according to message 13, the message header of node discovery message 11 is the same as the message header of message 13, and the payload of node discovery message 11 carries an indication of the message type of node discovery message 11. For example, the implementation method of node 11 generating node discovery message 11 according to message 13 may be that node 11 copies message 13, then only retains the message header of the copied message, and adds an indication to the payload of the copied message that indicates that its message type is a node discovery message, thereby obtaining node discovery message 11. Among them, the message header includes an Ethernet header, an IP header, and a transport layer protocol header (UDP header or TCP header). Alternatively, node discovery message 11 may also be a new message constructed by node 11, and the embodiment of the present application does not limit the message structure of the node discovery message.

In this implementation, node 11 can also receive node discovery message 12 sent by node 14, node discovery message 12 carries the identifier of the SFU server, and node discovery message 12 indicates that node 14 is a subordinate node of node 11 on the transmission path starting from the SFU server. Node 11 determines that node 14 supports data deduplication based on node discovery message 12, and sends node discovery response message 12 corresponding to node discovery message 12 to node 14, node discovery response message 12 indicates that node 11 supports data deduplication, and node 11 can also determine that it is not the last node on the transmission path starting from the SFU server. Node 14 can be, for example, the above-mentioned node 12.

In an embodiment of the present application, in a scheme where the node discovery process is triggered by a message whose destination is an SFU server, if a node receives a node discovery message sent by a subordinate node and does not receive a node discovery response message sent by an upper node, then the node can determine that it is the first node on the transmission path starting from the SFU server.

Another possible implementation is that the node discovery process is triggered by a message sent by the SFU server.

In this implementation, after node 11 receives message 14 whose source port number is the SFU service port number, it sends node discovery message 13 to node 15. Node 15 is the next hop of message 14 on node 11. The sender of message 14 is the SFU server. Node discovery message 13 carries the identifier of the SFU server, and node discovery message 13 indicates that node 11 is the superior node of node 15 on the transmission path starting from the SFU server. In response to receiving the node discovery response message 13 corresponding to the node discovery message 13 sent by node 15, node 11 determines that node 15 supports data deduplication, and node 11 can also determine that it is not the last node on the transmission path starting from the SFU server. Node 15 can be, for example, the above-mentioned node 12. Further, node 11 can also send a node discovery response confirmation message to node 15.

Optionally, there are multiple ways for the node discovery message 13 to indicate that the node 11 is the superior node of the node 15 on the transmission path starting from the SFU server. For example, if the source port number of the node discovery message 13 is the SFU service port number, and the node discovery message 11 carries the SFU The identifier of the server indicates that the node 11 sending the node discovery message 13 is the superior node of the node 15 receiving the node discovery message 11 on the transmission path starting from the SFU server. For another example, if the node discovery message 13 carries a position indication for indicating that it is the superior node, the position indication carried by the node discovery message 13 and the identifier of the SFU server together indicate that the node 11 sending the node discovery message 13 is the superior node of the node 15 receiving the node discovery message 11 on the transmission path starting from the SFU server.

Optionally, the node 11 generates a node discovery message 13 according to the message 14 , the message header of the node discovery message 13 is the same as the message header of the message 14 , and the payload part of the node discovery message 13 carries an indication of the message type of the node discovery message 13 .

In an embodiment of the present application, in a scheme where the node discovery process is triggered by a message whose sender is an SFU server, if a node receives a node discovery response message sent by a subordinate node and does not receive a node discovery message sent by an upper node, then the node can determine that it is the first node on the transmission path starting from the SFU server.

For the bottom node, for example, the node 21 in the method 1000 shown in Fig. 10, the node 21 may determine its position relative to the SFU server through the following two possible implementations.

In one possible implementation, a message destined for an SFU server triggers the node discovery process.

In this implementation, after node 21 receives message 25 whose destination port number is the SFU service port number, it sends node discovery message 21 to node 24. Node 24 is the next hop of message 25 on node 21. The destination of message 25 is the SFU server, and node discovery message 21 carries the identifier of the SFU server, and node discovery message 21 indicates that node 21 is the subordinate node of node 24 on the transmission path starting from the SFU server. In response to receiving the node discovery response message 21 corresponding to the node discovery message 21 sent by node 24, node 21 determines that node 24 supports data deduplication, and node 21 can also determine that it is not the first node on the transmission path starting from the SFU server. Node 24 can be, for example, the above-mentioned node 22. Further, node 21 can also send a node discovery response confirmation message to node 24.

Optionally, node 21 generates node discovery message 21 according to message 25, the message header of node discovery message 21 is the same as the message header of message 25, and the payload of node discovery message 21 carries an indication of the message type of node discovery message 21. The manner in which node 21 generates a node discovery message may refer to the manner in which node 11 generates a node discovery message.

In an embodiment of the present application, in a scheme where the node discovery process is triggered by a message whose destination is an SFU server, if a node receives a node discovery response message sent by an upper-level node and does not receive a node discovery message sent by a lower-level node, then the node can determine that it is the last node on the transmission path starting from the SFU server.

Another possible implementation is that the node discovery process is triggered by a message sent by the SFU server.

In this implementation, node 21 can receive node discovery message 22 sent by node 25, node discovery message 22 carries the identifier of the SFU server, and node discovery message 22 indicates that node 25 is the superior node of node 21 on the transmission path starting from the SFU server. Node 21 determines that node 25 supports data deduplication based on node discovery message 22, and sends node discovery response message 22 corresponding to node discovery message 22 to node 25, node discovery response message 22 indicates that node 21 supports data deduplication, and node 21 can also determine that it is not the first node on the transmission path starting from the SFU server. Node 25 can be, for example, the above-mentioned node 22.

In this implementation, after node 21 receives message 26 whose source port number is the SFU service port number, it can also send a node discovery message 23 to node 26, and node 26 is the next hop of message 26 on node 21. The sender of message 26 is the SFU server, and node discovery message 23 carries the identifier of the SFU server, and node discovery message 23 indicates that node 21 is the superior node of node 26 on the transmission path starting from the SFU server. In response to not receiving a node discovery response message 23 corresponding to the node discovery message 23 sent by node 26, node 21 determines that node 21 is the last node that supports data deduplication on the transmission path starting from the SFU server.

Optionally, the node 21 generates a node discovery message 23 according to the message 26 , the message header of the node discovery message 23 is the same as the message header of the message 26 , and the payload part of the node discovery message 23 carries an indication of the message type of the node discovery message 23 .

In an embodiment of the present application, in a scheme where the node discovery process is triggered by a message whose sender is an SFU server, if a node receives a node discovery message sent by an upper-level node and does not receive a node discovery response message sent by a lower-level node, then the node can determine that it is the last node on the transmission path starting from the SFU server.

For an intermediate node, such as the node 31 in the method 1100 shown in Fig. 11, the node 31 may determine its position relative to the SFU server through the following two possible implementations.

In one possible implementation, a message destined for an SFU server triggers the node discovery process.

In this implementation, after node 31 receives message 34 whose destination port number is the SFU service port number, it sends node 34 a message. Discovery message 31. Node 34 is the next hop of message 34 on node 31. The destination of message 34 is the SFU server, the node discovery message 31 carries the identifier of the SFU server, and the node discovery message 31 indicates that node 31 is the subordinate node of node 34 on the transmission path starting from the SFU server. In response to receiving the node discovery response message 31 corresponding to the node discovery message 31 sent by node 34, node 31 determines that node 34 supports data deduplication, and node 31 can also determine that it is not the first node on the transmission path starting from the SFU server. Node 34 can be, for example, the above-mentioned node 32. Further, node 31 can also send a node discovery response confirmation message to node 34.

Optionally, the node 31 generates a node discovery message 31 according to the message 34, the message header of the node discovery message 31 is the same as the message header of the message 34, and the payload of the node discovery message 31 carries an indication of the message type of the node discovery message 31. The manner in which the node 31 generates the node discovery message may refer to the manner in which the node 11 generates the node discovery message.

In this implementation, node 31 can also receive node discovery message 34 sent by node 37, node discovery message 34 carries the identifier of the SFU server, and node discovery message 34 indicates that node 37 is a subordinate node of node 31 on the transmission path starting from the SFU server. Node 31 determines that node 37 supports data deduplication based on node discovery message 34, and sends node discovery response message 34 corresponding to node discovery message 34 to node 37, node discovery response message 34 indicates that node 31 supports data deduplication, and node 31 can also determine that it is not the last node on the transmission path starting from the SFU server. Node 37 can be, for example, the above-mentioned node 33.

In an embodiment of the present application, in a scheme where the node discovery process is triggered by a message whose destination is an SFU server, if a node receives a node discovery response message sent by an upper-level node and receives a node discovery message sent by a lower-level node, then the node can determine itself as an intermediate node on the transmission path starting from the SFU server.

Another possible implementation is that the node discovery process is triggered by a message sent by the SFU server.

In this implementation, after node 31 receives message 35 whose source port number is the SFU service port number, it can send a node discovery message 33 to node 36. Node 36 is the next hop of message 35 on node 31. The sender of message 35 is the SFU server, and the node discovery message 33 carries the identifier of the SFU server, and the node discovery message 33 indicates that node 31 is the superior node of node 36 on the transmission path starting from the SFU server. In response to receiving the node discovery response message 33 corresponding to the node discovery message 33 sent by node 36, node 31 determines that node 36 supports data deduplication, and node 31 can also determine that it is not the last node on the transmission path starting from the SFU server. Node 36 can be, for example, the above-mentioned node 33. Further, node 31 can also send a node discovery response confirmation message to node 36.

Optionally, the node 31 generates a node discovery message 33 according to the message 35 , the message header of the node discovery message 33 is the same as the message header of the message 35 , and the payload part of the node discovery message 33 carries an indication of the message type of the node discovery message 33 .

In this implementation, node 31 can also receive node discovery message 32 sent by node 35, node discovery message 32 carries the identifier of the SFU server, and node discovery message 32 indicates that node 35 is the superior node of node 31 on the transmission path starting from the SFU server. Node 31 determines that node 35 supports data deduplication based on node discovery message 32, and sends node discovery response message 32 corresponding to node discovery message 32 to node 35, node discovery response message 32 indicates that node 31 supports data deduplication, and node 31 can also determine that it is not the first node on the transmission path starting from the SFU server. Node 35 can be, for example, the above-mentioned node 32.

In an embodiment of the present application, in a scheme where the node discovery process is triggered by a message whose sender is an SFU server, if a node receives a node discovery message sent by an upper-level node and a node discovery response message sent by a lower-level node, then the node can determine that it is an intermediate node on the transmission path starting from the SFU server.

Optionally, each node in the data transmission system provided by the embodiment of the present application may also store a correspondence between the identifier of the SFU server and the node position, and the node position is used to indicate the position of the node itself relative to the corresponding SFU server. Since there may be multiple SFU servers in the data transmission system, the same node may be in different positions relative to different SFU servers. For example, a certain node is a top node relative to SFU server 1 and an intermediate node relative to SFU server 2. By storing the correspondence between the identifier of the SFU server and the node position in the node, after receiving a message from a certain SFU server as the sender, the node can determine the node at which position it should perform deduplication or recovery processing on the message, which is suitable for application scenarios with complex networking.

The order of the steps of the data transmission method provided in the embodiment of the present application can be adjusted appropriately, and the steps can also be increased or decreased accordingly according to the situation. Any technician familiar with the technical field can easily think of the method of change within the technical scope disclosed in this application, which should be covered within the protection scope of this application. For example, in the embodiment of the present application, the content of the data set and/or sampling label set stored in each node can be updated regularly, such as only storing data within a certain period of time, and automatically deleting expired data to reduce the memory resource consumption of the node. For another example, the message with the destination as the SFU server or the sender as the SFU server can trigger the reception of the message as described in the above embodiment. In addition to sending a node discovery message in the transmission direction of the message, the node receiving the message can also trigger the node receiving the message to send a node discovery message in the reverse transmission direction of the message. The principle of the node sending a node discovery message in the reverse transmission direction of the message to determine its own position relative to the SFU server can refer to the principle of the node sending a node discovery message in the transmission direction of the message to determine its own position relative to the SFU server in the above-mentioned embodiment, and the embodiments of the present application will not be described one by one here.

The following is an illustration of the virtual device in the embodiment of the present application.

For example, FIG12 is a schematic diagram of the structure of a communication node provided in an embodiment of the present application. The communication node is a first node, for example, it can be the node 11 in the method 600 shown in FIG6. As shown in FIG12, the communication node 1200 includes an acquisition module 1201, a processing module 1202 and a sending module 1203. Optionally, the communication node 1200 also includes a receiving module 1204.

The acquisition module 1201 is used to acquire the first message whose sender is the SFU server, and the first node is the first node that supports data deduplication on the transmission path of the first message. The processing module 1202 is used to deduplicate the payload part of the first message if there is a target message in the historical message sent by the first node to the second node, and the payload part of the target message has repeated content with the payload part of the first message, to obtain a second message, the second message does not include repeated content, and the second message carries a deduplication mark and indication information of the repeated content, the deduplication mark is used to indicate that the second message is a deduplication message, and the second node is the next hop of the first message on the first node. The sending module 1203 is used to send the second message to the second node. Here, the first node can be the above-mentioned node 11, the second node can be the above-mentioned node 12, the first message can be the above-mentioned message 11, and the second message can be the above-mentioned message 12.

Optionally, the repeated content includes one or more repeated data blocks, the indication information includes one or more indications, the one or more indications correspond to the one or more repeated data blocks one-to-one, and each indication is used to indicate a hash value of a corresponding repeated data block.

Optionally, a data set is stored in the first node, and the data set includes the payload part of the historical message sent by the first node to the second node. The processing module 1202 is used to: match the content of the payload part of the first message with the payload part in the data set. If there is a target payload part in the data set that has a duplicate data block with the payload part of the first message, determine that the target message exists in the historical message. For each duplicate data block between the payload part of the first message and the target payload part, the first node calculates the hash value of the duplicate data block. Remove the duplicate data blocks in the payload part of the first message, and add an indication corresponding to the duplicate data block to the payload part of the first message, the indication is used to indicate the hash value of the duplicate data block and the position of the duplicate data block in the payload part of the first message.

Optionally, the processing module 1202 is configured to: if there is no payload part having a duplicate data block with the payload part of the first message in the data set, determine that the target message does not exist in the historical message, and add the payload part of the first message to the data set.

Optionally, the payload part includes a protocol part and a data part, and one or more repeated data blocks are located in the data part of the first message.

Optionally, a sampling tag set is stored in the first node, and the sampling tag set includes a hash value of a historical data block, and the historical data block is a data block obtained by sampling a preset position of the data part of a historical message sent by the first node to the second node. Processing module 1202 is used to: sample a preset position of the data part of the first message to obtain a sampled data block. Calculate the hash value of the sampled data block. If the sampling tag set includes the hash value of the sampled data block, determine that the target message exists in the historical message. The sampled data block whose hash value belongs to the sampling tag set is used as a duplicate data block, remove the duplicate data block of the data part of the first message, and add an indication corresponding to the duplicate data block in the payload part of the first message, the indication is used to indicate the hash value of the duplicate data block.

Optionally, there are multiple preset positions, and the first node obtains multiple sampled data blocks by sampling the preset positions of the data part of the first message. The indication is also used to indicate the position of the repeated data block in the data part of the first message.

Optionally, the processing module 1202 is configured to: if the sampling tag set does not include the hash value of the sampling data block, determine that the target message does not exist in the historical message, and add the hash value of the sampling data block to the sampling tag set.

Optionally, the sampling tag set also includes a historical data block indicated by a hash value. If the sampling tag set includes the hash value of the sampling data block, the processing module 1202 is used to: if the sampling tag set includes the hash value of the sampling data block, perform content matching on the sampling data block and the historical data block indicated by the hash value of the sampling data block. When the content of the sampling data block is the same as the content of the historical data block indicated by the hash value of the sampling data block, it is determined that the target message exists in the historical message.

Optionally, the first node also stores the protocol part of the historical message sent by the first node to the second node; the repeated content also includes protocol information located in the protocol part of the first message, and the indication information also includes a difference indication, which is used to indicate the difference between the protocol part of the first message and the protocol part of the target message.

Optionally, the first node stores one or more flow grouping sets corresponding to the second node, each flow grouping set including flow identifiers of multiple flows flowing through the second node. The processing module 1202 is further configured to, after the first node obtains the first message, determine whether the target history message sent to the second node contains the flow identifier of the flow to which the first message belongs if there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the flow grouping set corresponding to the second node. The target message exists, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set. The sending module 1203 is further configured to send the first message to the second node if all flow grouping sets corresponding to the second node do not include the flow identifier of the flow to which the first message belongs.

Optionally, the receiving module 1204 is configured to receive grouping information sent by the second node, where the grouping information includes a correspondence between a node identifier of the second node and one or more flow grouping sets.

Optionally, the first node is not an SFU server; the sending module 1203 is further used to send a first node discovery message to the third node after receiving a third message whose destination port number is the SFU service port number, the third node is the next hop of the third message on the first node, the destination of the third message is the SFU server, the first node discovery message carries the identifier of the SFU server, and the first node discovery message indicates that the first node is a subordinate node of the third node on the transmission path starting from the SFU server. The processing module 1202 is also used to respond to the first node discovery response message corresponding to the first node discovery message sent by the third node not being received, and determine that the first node is the first node that supports data deduplication on the transmission path starting from the SFU server. Here, the third node can be the above-mentioned node 13, the third message can be the above-mentioned message 13, the first node discovery message can be the above-mentioned node discovery message 11, and the first node discovery response message can be the above-mentioned node discovery response message 11.

Optionally, the processing module 1202 is also used to: generate a first node discovery message based on the third message, the message header of the first node discovery message is the same as the message header of the third message, and the payload part of the first node discovery message carries an indication of the message type of the first node discovery message.

Optionally, the receiving module 1204 is used to receive a second node discovery message sent by the fourth node, the second node discovery message carries the identifier of the SFU server, and the second node discovery message indicates that the fourth node is a subordinate node of the first node on the transmission path starting from the SFU server. The processing module 1202 is also used to determine that the fourth node supports data deduplication based on the second node discovery message. The sending module 1203 is also used to send a second node discovery response message corresponding to the second node discovery message to the fourth node, and the second node discovery response message indicates that the first node supports data deduplication. Here, the fourth node can be the above-mentioned node 14, the second node discovery message can be the above-mentioned node discovery message 12, and the second node discovery response message can be the above-mentioned node discovery response message 12.

Optionally, the sending module 1203 is also used to send a third node discovery message to the fifth node after receiving the fourth message whose source port number is the SFU service port number. The fifth node is the next hop of the fourth message on the first node. The sender of the fourth message is the SFU server. The third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the upper node of the fifth node on the transmission path starting from the SFU server. The processing module 1202 is also used to respond to the third node discovery response message corresponding to the third node discovery message sent by the fifth node, and determine that the fifth node supports data deduplication. Here, the fifth node can be the above-mentioned node 15, the fourth message can be the above-mentioned message 14, the third node discovery message can be the above-mentioned node discovery message 13, and the third node discovery response message can be the above-mentioned node discovery response message 13.

Optionally, the sending module 1203 is further configured to send the first message to the second node if the target message does not exist in the historical messages sent by the first node to the second node.

For another example, Figure 13 is a schematic diagram of the structure of another communication node provided in an embodiment of the present application. The communication node is a first node, for example, it can be the node 21 in the method 1000 shown in Figure 10. As shown in Figure 13, the communication node 1300 includes a receiving module 1301, a processing module 1302 and a sending module 1303.

The receiving module 1301 is used to receive a first message sent by a second node. The sender of the first message is an SFU server. The first message carries a deduplication mark and indication information of duplicate content. The deduplication mark is used to indicate that the first message is a deduplication message. The first node is the last node that supports data deduplication on the transmission path of the first message. The processing module 1302 is used to determine that the first message is a deduplication message based on the deduplication mark; obtain duplicate content from a data set according to the indication information, and the data set includes at least part of the content of the load part of the historical message received by the first node from the second node; perform deduplication recovery processing on the load part of the first message according to the duplicate content to obtain a second message, and the load part of the second message includes duplicate content. The sending module 1303 is used to send a second message to a third node, and the third node is the next hop of the first message on the first node. Here, the first node can be the above-mentioned node 21, the second node can be the above-mentioned node 22, the third node can be the above-mentioned node 23, the first message can be the above-mentioned message 21, and the second message can be the above-mentioned message 22.

Optionally, the repeated content includes one or more repeated data blocks, the indication information includes one or more indications, the one or more indications correspond to the one or more repeated data blocks one-to-one, and each indication is used to indicate a hash value of a corresponding repeated data block.

Optionally, each indication is also used to indicate the position of the corresponding repeated data block in the payload part of the original message corresponding to the first message, and the data set includes the payload part of the historical message received by the first node from the second node. Processing module 1302 is used to: for each indication in the indication information, obtain the data block to be matched at the position of the payload part in the data set according to the position indicated by the indication. Calculate the hash value of the data block to be matched. The data block to be matched whose hash value is consistent with the hash value indicated by the indication is determined as the repeated data block corresponding to the indication.

Optionally, the payload portion includes a protocol portion and a data portion, and one or more repeated data blocks are located in the data portion.

Optionally, the data set includes a correspondence between a hash value of a historical data block and a historical data block, where the historical data block is a data block obtained by sampling a preset position of a data portion of a historical message received by the first node from the second node. The processing module 1302 is configured to determine a historical data block in the data set corresponding to the hash value indicated by the indication in the indication information as a duplicate data block corresponding to the indication.

Optionally, the indication is also used to indicate the position of the corresponding repeated data block in the data portion of the original message corresponding to the first message. The processing module 1302 is used to indicate the indicated position in the data portion of the first message for each indication in the indication information, and add the repeated data block corresponding to the indication.

Optionally, the repeated content also includes protocol information located in the protocol part, and the indication information also includes a difference indication, the difference indication is used to indicate the difference between the protocol part of the original message corresponding to the first message and the protocol part of the target message, the target message is a historical message received by the first node from the second node, in which the data part and the data part of the original message have one or more repeated data blocks; the data set also includes the protocol part of the message to which the historical data block belongs. The processing module 1302 is also used to: obtain the protocol part of the target message to which one or more repeated data blocks belong from the data set. Modify the protocol part of the target message according to the difference indication, and use the modified protocol part of the target message as the protocol part of the second message.

Optionally, the deduplication mark is located in the payload part of the first message, and one or more flow grouping sets are stored in the first node, each flow grouping set including flow identifiers of multiple flows flowing through the first node. The processing module 1302 is also used to determine that there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the one or more flow grouping sets before the first node determines that the first message is a deduplication message based on the deduplication mark; parse the payload part of the first message to obtain the deduplication mark.

Optionally, the processing module 1302 is configured to obtain the repeated content from the payload content of the target historical message in the data set according to the indication information, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set.

Optionally, the receiving module 1301 is further configured to receive a third message, wherein the flow identifier of the flow to which the third message belongs does not belong to any flow grouping set. The sending module 1303 is further configured to forward the third message. Here, the third message may be the message 23 described above.

Optionally, the receiving module 1301 is further used to receive a fourth message, and the flow identifier of the flow to which the fourth message belongs belongs to the flow grouping set. The processing module 1302 is further used to parse the payload part of the fourth message, determine that the payload part of the fourth message does not carry a deduplication mark; add at least part of the content of the payload part of the fourth message to the data set, and forward the fourth message. Here, the fourth message can be the above-mentioned message 24.

Optionally, the processing module 1302 is further configured to add the flow identifiers of the flows to which the messages with duplicate contents in the payload part belong among the received multiple messages belonging to different flows to the same flow grouping set, and the senders of the different flows are all SFU servers.

Optionally, the sending module 1303 is further configured to send grouping information to the second node, where the grouping information includes a correspondence between a node identifier of the first node and one or more flow grouping sets.

Optionally, the sending module 1303 is also used to send a first node discovery message to the fourth node after receiving the fifth message whose destination port number is the SFU service port number. The fourth node is the next hop of the fifth message on the first node, and the destination of the fifth message is the SFU server. The first node discovery message carries the identifier of the SFU server, and the first node discovery message indicates that the first node is the lower node of the fourth node on the transmission path starting from the SFU server. The processing module 1302 is also used to determine that the fourth node supports data deduplication in response to receiving a first node discovery response message corresponding to the first node discovery message sent by the fourth node. Here, the fourth node can be the above-mentioned node 24, the fifth message can be the above-mentioned message 25, the first node discovery message can be the above-mentioned node discovery message 21, and the first node discovery response message can be the above-mentioned node discovery response message 21.

Optionally, the processing module 1302 is also used to generate a first node discovery message based on the fifth message, the message header of the first node discovery message is the same as the message header of the fifth message, and the payload part of the first node discovery message carries an indication of the message type of the first node discovery message.

Optionally, the receiving module 1301 is also used to receive a second node discovery message sent by the fifth node, the second node discovery message carries the identifier of the SFU server, and the second node discovery message indicates that the fifth node is the upper node of the first node on the transmission path starting from the SFU server. The processing module 1302 is also used to determine that the fifth node supports data deduplication based on the second node discovery message. The sending module 1303 is also used to send a second node discovery response message corresponding to the second node discovery message to the fifth node, and the second node discovery response message indicates that the first node supports data deduplication. Here, the fifth node can be the above-mentioned node 25, the second node discovery message can be the above-mentioned node discovery message 22, and the second node discovery response message can be the above-mentioned node discovery response message 22.

Optionally, the sending module 1303 is further used to send a third node discovery message to the sixth node after receiving the sixth message whose source port number is the SFU service port number, the sixth node is the next hop of the sixth message on the first node, the sender of the sixth message is the SFU server, the third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the upper node of the sixth node on the transmission path starting from the SFU server. The processing module 1302 is also used to respond to the failure to receive the third node discovery message sent by the sixth node. The third node discovery response message corresponding to the message determines that the first node is the last node that supports data deduplication on the transmission path starting from the SFU server. Here, the sixth node can be the above-mentioned node 26, the sixth message can be the above-mentioned message 26, the third node discovery message can be the above-mentioned node discovery message 23, and the third node discovery response message can be the above-mentioned node discovery response message 23.

For another example, Figure 14 is a schematic diagram of the structure of another communication node provided in an embodiment of the present application. The communication node is a first node, for example, it can be the node 31 in the method 1100 shown in Figure 11. As shown in Figure 14, the communication node 1400 includes a receiving module 1401, a processing module 1402 and a sending module 1403.

The receiving module 1401 is used to receive a first message sent by the second node, the sender of the first message is an SFU server, and the first node is an intermediate node that supports data deduplication on the transmission path of the first message. The processing module 1402 is used to, if the first message is a non-deduplicated message, and there is a first original message in the historical message sent by the first node to the third node, and the payload part of the first original message and the payload part of the first message have a first repeated content, deduplication processing is performed on the payload part of the first message to obtain a second message, the second message does not include the first repeated content, and the second message carries a deduplication mark and a first indication information of the first repeated content, the deduplication mark is used to indicate that the second message is a deduplication message, and the third node is the next hop of the first message on the first node. The sending module 1403 is used to send a second message to the third node. Here, the first node can be the above-mentioned node 31, the second node can be the above-mentioned node 32, the third node can be the above-mentioned node 33, the first message can be the above-mentioned message 31, and the second message can be the above-mentioned message 32.

Optionally, the sending module 1403 is further configured to send the first message to the third node if the first message is a non-deduplicated message and the first original message does not exist in the historical messages sent by the first node to the third node.

Optionally, the first repetitive content includes one or more repetitive data blocks, the first indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repetitive data blocks, and each indication is used to indicate a hash value of a corresponding repetitive data block.

Optionally, a first data set is stored in the first node, and the first data set includes the payload part of the historical message sent by the first node to the third node. The processing module 1402 is also used to: content match the payload part of the first message with the payload part in the first data set. If there is a target payload part in the first data set that has a repeated data block with the payload part of the first message, it is determined that the first original message exists in the historical message. If there is no payload part in the first data set that has a repeated data block with the payload part of the first message, it is determined that the first original message does not exist in the historical message, and the first node adds the payload part of the first message to the first data set.

Optionally, the first data set includes a target payload part, and the processing module 1402 is used to: for each repeated data block between the payload part and the target payload part of the first message, calculate a hash value of the repeated data block, remove the repeated data block in the payload part of the first message, and add an indication corresponding to the repeated data block to the payload part of the first message, where the indication is used to indicate the hash value of the repeated data block and the position of the repeated data block in the payload part of the first message.

Optionally, the payload portion includes a protocol portion and a data portion, and one or more repeated data blocks are located in the data portion of the first message; a sampling tag set is stored in the first node, and the sampling tag set includes a hash value of a historical data block, and the historical data block is a data block obtained by sampling a preset position of the data portion of the historical message sent by the first node to the third node. The processing module 1402 is also used to: sample a preset position of the data portion of the first message to obtain a sampled data block. Calculate the hash value of the sampled data block. If the sampling tag set includes the hash value of the sampled data block, it is determined that the first original message exists in the historical message. If the sampling tag set does not include the hash value of the sampled data block, it is determined that the first original message does not exist in the historical message, and the first node adds the hash value of the sampled data block to the sampling tag set.

Optionally, the sampling tag set includes a hash value of the sampling data block, and the processing module 1402 is used to take the sampling data block whose hash value belongs to the sampling tag set as a repeated data block, remove the repeated data block in the data part of the first message, and add an indication corresponding to the repeated data block in the payload part of the first message, where the indication is used to indicate the hash value of the repeated data block.

Optionally, the first node also stores the protocol part of the historical message sent by the first node to the third node; the first repeated content also includes protocol information located in the protocol part of the first message, and the first indication information also includes a difference indication, which is used to indicate the difference between the protocol part of the first message and the protocol part of the first original message.

Optionally, the sending module 1403 is also used to send the first message to the third node if the first message is a deduplicated message, the first message carries second indication information for the second duplicate content, and the second original message exists in the historical message sent by the first node to the third node, and the payload part of the second original message includes the second duplicate content.

Optionally, the processing module 1402 is further used to obtain the second duplicate content from a second data set according to the second indication information if the first message is a deduplicated message, the first message carries second indication information for the second duplicate content, and the second original message does not exist in the historical message sent by the first node to the third node, and the payload part of the second original message includes the second duplicate content, and the second duplicate content is obtained from the second data set according to the second indication information, and the second data set includes at least part of the payload part of the historical message received by the first node from the second node; deduplicated the payload part of the first message according to the second duplicate content The third message is obtained by re-recovery processing, and the payload part of the third message includes the second repeated content. The sending module 1403 is further used to send the third message to the third node. Here, the third message can be the message 33 mentioned above.

Optionally, the second repetitive content includes one or more repetitive data blocks, and the second indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repetitive data blocks, and each indication is used to indicate a hash value of a corresponding repetitive data block.

Optionally, each indication is also used to indicate the position of the corresponding repeated data block in the payload part of the original message corresponding to the first message, and the second data set includes the payload part of the historical message received by the first node from the second node. Processing module 1402 is used to: for each indication in the second indication information, obtain the data block to be matched at the position of the payload part in the second data set according to the position indicated by the indication. Calculate the hash value of the data block to be matched. The data block to be matched whose hash value is consistent with the hash value indicated by the indication is determined as the repeated data block corresponding to the indication.

Optionally, the payload portion includes a protocol portion and a data portion, and one or more repeated data blocks are located in the data portion; the second data set includes a correspondence between a hash value of a historical data block and a historical data block, and the historical data block is a data block obtained by sampling a preset position of a data portion of a historical message received by the first node from the second node. The processing module 1402 is configured to determine a historical data block in the second data set corresponding to the hash value indicated by the indication in the second indication information as a repeated data block corresponding to the indication.

Optionally, the indication is also used to indicate the position of the corresponding repeated data block in the data portion of the original message corresponding to the first message. The processing module 1402 is used to indicate the indicated position in the data portion of the first message for each indication in the second indication information, and add the repeated data block corresponding to the indication.

Optionally, the second repeated content also includes protocol information located in the protocol part, and the second indication information also includes a difference indication, the difference indication is used to indicate the difference between the protocol part of the original message corresponding to the first message and the protocol part of the target message, the target message is a historical message received by the first node from the second node, in which the data part and the data part of the original message have one or more repeated data blocks; the second data set also includes the protocol part of the message to which the historical data block belongs. The processing module 1402 is also used to: obtain the protocol part of the target message to which one or more repeated data blocks belong from the second data set. Modify the protocol part of the target message according to the difference indication, and use the modified protocol part of the target message as the protocol part of the third message.

Optionally, one or more local flow grouping sets are stored in the first node, each local flow grouping set including flow identifiers of multiple flows flowing through the first node; the processing module 1402 is further used to: after determining that there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the one or more local flow grouping sets, parse the payload part of the first message. If the payload part of the first message carries a deduplication mark, determine that the first message is a deduplication message. If the payload part of the first message does not carry a deduplication mark, determine that the first message is a non-deduplication message.

Optionally, the processing module 1402 is further used to add the flow identifiers of the flows to which the messages with duplicate contents in the payload part belong among the received multiple messages belonging to different flows to the same local flow grouping set, and the senders of the different flows are all SFU servers.

Optionally, the sending module 1403 is further configured to send first grouping information to the second node, where the first grouping information includes a correspondence between a node identifier of the first node and one or more local flow grouping sets.

Optionally, one or more lower-level flow grouping sets corresponding to the third node are stored in the first node, and each lower-level flow grouping set includes flow identifiers of multiple flows flowing through the third node. Processing module 1402 is also used to determine whether there is a message with a payload part that has repeated content with the payload part of the first message in the target historical message sent to the third node after the first node receives the first message sent by the second node, if there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the lower-level flow grouping set corresponding to the third node, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set. Sending module 1403 is also used to send the first message to the third node if all lower-level flow grouping sets corresponding to the third node do not include the flow identifier of the flow to which the first message belongs.

Optionally, the receiving module 1401 is further configured to receive second grouping information sent by a third node, where the second grouping information includes a correspondence between a node identifier of the third node and one or more lower-level flow grouping sets.

Optionally, the sending module 1403 is also used to send a first node discovery message to the fourth node after receiving a fourth message whose destination port number is the SFU service port number. The fourth node is the next hop of the fourth message on the first node. The destination of the fourth message is the SFU server. The first node discovery message carries the identifier of the SFU server, and the first node discovery message indicates that the first node is the lower node of the fourth node on the transmission path starting from the SFU server. The processing module 1402 is also used to determine that the fourth node supports data deduplication in response to receiving a first node discovery response message corresponding to the first node discovery message sent by the fourth node. Here, the fourth node can be the above-mentioned node 34, the fourth message can be the above-mentioned message 34, the first node discovery message can be the above-mentioned node discovery message 31, and the first node discovery response message can be the above-mentioned node discovery response message 31.

Optionally, the receiving module 1401 is further used to receive a second node discovery message sent by the fifth node, the second node discovery message carries an identifier of the SFU server, and the second node discovery message indicates that the fifth node is the first node on the transmission path starting from the SFU server. The processing module 1402 is also used to determine that the fifth node supports data deduplication according to the second node discovery message. The sending module 1403 is also used to send a second node discovery response message corresponding to the second node discovery message to the fifth node, and the second node discovery response message indicates that the first node supports data deduplication. Here, the fifth node can be the above-mentioned node 35, the second node discovery message can be the above-mentioned node discovery message 32, and the second node discovery response message can be the above-mentioned node discovery response message 32.

Optionally, the sending module 1403 is also used to send a third node discovery message to the sixth node after receiving the fifth message whose source port number is the SFU service port number. The sixth node is the next hop of the fifth message on the first node. The sender of the fifth message is the SFU server. The third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the upper node of the sixth node on the transmission path starting from the SFU server. The processing module 1402 is also used to respond to the third node discovery response message corresponding to the third node discovery message sent by the sixth node, and determine that the sixth node supports data deduplication. Here, the sixth node can be the above-mentioned node 36, the fifth message can be the above-mentioned message 35, the third node discovery message can be the above-mentioned node discovery message 33, and the third node discovery response message can be the above-mentioned node discovery response message 33.

Optionally, the receiving module 1401 is also used to receive a fourth node discovery message sent by the seventh node, the fourth node discovery message carries the identifier of the SFU server, and the fourth node discovery message indicates that the seventh node is a subordinate node of the first node on the transmission path starting from the SFU server. The processing module 1402 is also used to determine that the seventh node supports data deduplication based on the fourth node discovery message. The sending module 1403 is also used to send a fourth node discovery response message corresponding to the fourth node discovery message to the seventh node, and the fourth node discovery response message indicates that the first node supports data deduplication. Here, the seventh node can be the above-mentioned node 37, the fourth node discovery message can be the above-mentioned node discovery message 34, and the fourth node discovery response message can be the above-mentioned node discovery response message 34.

Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

The following is an illustration of the basic hardware structure involved in the embodiments of the present application.

For example, FIG15 is a schematic diagram of the hardware structure of a communication device provided in an embodiment of the present application. As shown in FIG15 , a communication device 1500 includes a processor 1501 and a memory 1502, and the memory 1501 is connected to the memory 1502 via a bus 1503. FIG15 illustrates that the processor 1501 and the memory 1502 are independent of each other. Optionally, the processor 1501 and the memory 1502 are integrated together. The communication device 1500 may be, for example, a network device or a server.

The memory 1502 is used to store computer programs, including operating systems and program codes. The memory 1502 is a storage medium of various types, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), flash memory, optical storage, register, optical disk storage, optical disk storage, magnetic disk or other magnetic storage devices.

The processor 1501 is a general-purpose processor or a special-purpose processor. The processor 1501 may be a single-core processor or a multi-core processor. The processor 1501 includes at least one circuit to execute the above-mentioned data transmission method provided in the embodiment of the present application, such as the steps performed by the node 11, the node 21 or the node 31 in the above-mentioned method embodiment.

Optionally, the communication device 1500 further includes a network interface 1504, and the network interface 1504 is connected to the processor 1501 and the memory 1502 via the bus 1503. The network interface 1504 enables the communication device 1500 to communicate with other devices.

Optionally, the communication device 1500 further includes an input/output (I/O) interface 1505, which is connected to the processor 1501 and the memory 1502 via the bus 1503. The processor 1501 can receive input commands or data, etc. through the I/O interface 1505. The I/O interface 1505 is used for the communication device 1500 to connect input devices, such as keyboards, mice, etc. Optionally, in some possible scenarios, the above-mentioned network interface 1504 and I/O interface 1505 are collectively referred to as a communication interface.

Optionally, the communication device 1500 further includes a display 1506, which is connected to the processor 1501 and the memory 1502 via the bus 1503. The display 1506 can be used to display the intermediate results and/or final results generated by the processor 1501 executing the above method. In a possible implementation, the display 1506 is a touch display screen to provide a human-computer interaction interface.

The bus 1503 is any type of communication bus for interconnecting the internal devices of the communication device 1500. For example, a system bus. The embodiment of the present application takes the interconnection of the above-mentioned devices inside the communication device 1500 through the bus 1503 as an example. Optionally, the above-mentioned devices inside the communication device 1500 are connected to each other in a communication manner other than the bus 1503, for example, the above-mentioned devices inside the communication device 1500 are interconnected through a logical interface inside the communication device 1500.

The above devices may be arranged on separate chips, or at least partially or completely on the same chip. Whether each device is independently arranged on different chips or integrated on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices.

The communication device 1500 shown in Figure 15 is merely exemplary. During implementation, the communication device 1500 may also include other components, which are not listed here. The communication device 1500 shown in Figure 15 may implement data transmission by executing all or part of the steps of the method provided in the above embodiment.

The embodiment of the present application also provides a data transmission system, including: an SFU server and multiple nodes in a communication network. The multiple nodes include a first node and a second node, and the first node is located between the SFU server and the second node. The SFU server is used to send a message to the first node, the first node is used to execute the steps executed by node 11 in the above method embodiment, and the second node is used to execute the steps executed by node 21 in the above method embodiment.

Optionally, the plurality of nodes further include a third node, and the third node is located between the first node and the second node. The third node is used to execute the steps executed by the node 31 in the above method embodiment.

The present application also provides another data transmission system, including: an SFU server and a first node in a communication network. The SFU server is used to execute the steps executed by the node 11 in the above method embodiment, and the first node is used to execute the steps executed by the node 21 in the above method embodiment.

Optionally, the communication network further comprises a second node, and the second node is located between the SFU server and the first node. The second node is used to execute the steps executed by the node 31 in the above method embodiment.

The embodiment of the present application also provides a communication node, including: a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to call the computer program to implement the steps performed by node 11, node 21 or node 31 in the above method embodiment.

The embodiment of the present application further provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the steps executed by node 11, node 21 or node 31 in the above method embodiment are implemented.

The embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps executed by the node 11, the node 21 or the node 31 in the above method embodiment are implemented.

A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

In the embodiments of the present application, the terms “first”, “second” and “third” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

The term "and/or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character "/" in this article generally indicates that the associated objects before and after are in an "or" relationship.

It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present application should be included in the protection scope of the present application.

Claims (63)

A data transmission method, characterized in that the method comprises: A first node obtains a first message whose sender is a selective forwarding unit SFU server, and the first node is the first node that supports data deduplication on a transmission path of the first message; If a target message exists in the historical messages sent by the first node to the second node, and a payload part of the target message has duplicate content with a payload part of the first message, the first node performs deduplication processing on the payload part of the first message to obtain a second message, the second message does not include the duplicate content, and the second message carries a deduplication mark and indication information of the duplicate content, the deduplication mark is used to indicate that the second message is a deduplication message, and the second node is the next hop of the first message on the first node; The first node sends the second message to the second node. The method according to claim 1 is characterized in that the repeated content includes one or more repeated data blocks, the indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repeated data blocks, and each of the indications is used to indicate the hash value of the corresponding repeated data block. The method according to claim 2, characterized in that a data set is stored in the first node, the data set including a payload portion of a historical message sent by the first node to the second node, and the method further comprises: The first node performs content matching on a payload part of the first message and a payload part in the data set; If a target payload part having a repeated data block with the payload part of the first message exists in the data set, the first node determines that the target message exists in the historical message; The first node performs deduplication processing on the payload part of the first message, including: For each repeated data block between the payload part and the target payload part of the first message, the first node calculates a hash value of the repeated data block; The first node removes the repeated data block in the payload part of the first message, and adds an indication corresponding to the repeated data block in the payload part of the first message, wherein the indication is used to indicate a hash value of the repeated data block and a position of the repeated data block in the payload part of the first message. The method according to claim 3, characterized in that the method further comprises: If the data set does not contain a payload part having a duplicate data block with the payload part of the first message, the first node determines that the target message does not exist in the historical messages; The first node adds the payload part of the first message to the data set. The method according to claim 2 is characterized in that the payload part includes a protocol part and a data part, and the one or more repeated data blocks are located in the data part of the first message. The method according to claim 5 is characterized in that a sampling tag set is stored in the first node, the sampling tag set includes a hash value of a historical data block, and the historical data block is a data block obtained by sampling a preset position of a data part of a historical message sent by the first node to the second node; the method further comprises: The first node samples the preset position of the data part of the first message to obtain a sampled data block; The first node calculates a hash value of the sampled data block; If the sampling tag set includes the hash value of the sampling data block, the first node determines that the target message exists in the historical message; The first node performs deduplication processing on the payload part of the first message, including: The first node uses the sampling data block whose hash value belongs to the sampling tag set as a repeated data block, removes the repeated data block from the data part of the first message, and adds an indication corresponding to the repeated data block in the payload part of the first message, where the indication is used to indicate the hash value of the repeated data block. The method according to claim 6 is characterized in that there are multiple preset positions, and the first node obtains multiple sampled data blocks by sampling the preset positions of the data part of the first message, and the indication is also used to indicate the position of the repeated data block in the data part of the first message. The method according to claim 6 or 7, characterized in that the method further comprises: If the sampling tag set does not include the hash value of the sampling data block, the first node determines that the target message does not exist in the historical message; The first node adds the hash value of the sampled data block to the sampling tag set. The method according to any one of claims 6 to 8, characterized in that the sampling tag set also includes a historical data block indicated by a hash value, and if the sampling tag set includes the hash value of the sampling data block, the first node determines that the target message exists in the historical message, comprising: If the sampling tag set includes the hash value of the sampling data block, the first node performs content matching between the sampling data block and the historical data block indicated by the hash value of the sampling data block; When the content of the sampled data block is the same as the historical data block indicated by the hash value of the sampled data block, the first node determines that the target message exists in the historical message. According to the method described in any one of claims 5 to 9, it is characterized in that the first node also stores the protocol part of the historical message sent by the first node to the second node; the repeated content also includes the protocol information located in the protocol part of the first message, and the indication information also includes a difference indication, and the difference indication is used to indicate the difference between the protocol part of the first message and the protocol part of the target message. The method according to any one of claims 1 to 10, characterized in that the first node stores one or more flow grouping sets corresponding to the second node, each of the flow grouping sets includes flow identifiers of multiple flows flowing through the second node, and after the first node obtains the first message, the method further includes: If there is a target flow grouping set including the flow identifier of the flow to which the first message belongs in the flow grouping set corresponding to the second node, the first node determines whether the target message exists in the target historical message sent to the second node, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set; If all flow grouping sets corresponding to the second node do not include the flow identifier of the flow to which the first message belongs, the first node sends the first message to the second node. The method according to claim 11, characterized in that the method further comprises: The first node receives the grouping information sent by the second node, where the grouping information includes a correspondence between a node identifier of the second node and the one or more flow grouping sets. The method according to any one of claims 1 to 12, characterized in that the first node is not the SFU server, and the method further comprises: After the first node receives the third message whose destination port number is the SFU service port number, the first node sends a first node discovery message to the third node, where the third node is the next hop of the third message on the first node, and the destination of the third message is the SFU server. The first node discovery message carries an identifier of the SFU server, and the first node discovery message indicates that the first node is a subordinate node of the third node on a transmission path starting from the SFU server. In response to not receiving the first node discovery response message corresponding to the first node discovery message sent by the third node, the first node determines that the first node is the first node supporting data deduplication on the transmission path starting from the SFU server. The method according to claim 13, characterized in that the method further comprises: The first node generates the first node discovery message according to the third message, and the message header of the first node discovery message is the same as the The message header of the third message is the same, and the payload part of the first node discovery message carries an indication of the message type of the first node discovery message. The method according to any one of claims 1 to 14, characterized in that the method further comprises: The first node receives a second node discovery message sent by the fourth node, where the second node discovery message carries an identifier of the SFU server, and the second node discovery message indicates that the fourth node is a subordinate node of the first node on a transmission path starting from the SFU server; The first node determines that the fourth node supports data deduplication based on the second node discovery message, and sends a second node discovery response message corresponding to the second node discovery message to the fourth node, and the second node discovery response message indicates that the first node supports data deduplication. The method according to any one of claims 1 to 15, characterized in that the method further comprises: After the first node receives the fourth message whose source port number is the SFU service port number, the third node discovery message is sent to the fifth node, the fifth node is the next hop of the fourth message on the first node, the sender of the fourth message is the SFU server, the third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the upper node of the fifth node on the transmission path starting from the SFU server; In response to receiving a third node discovery response message corresponding to the third node discovery message sent by the fifth node, the first node determines that the fifth node supports data deduplication. The method according to any one of claims 1 to 16, characterized in that the method further comprises: If the target message does not exist in the historical messages sent by the first node to the second node, the first node sends the first message to the second node. A data transmission method, characterized in that the method comprises: The first node receives a first message sent by the second node, where the sender of the first message is a selective forwarding unit SFU server, the first message carries a deduplication mark and indication information of duplicate content, the deduplication mark is used to indicate that the first message is a deduplication message, and the first node is the last node that supports data deduplication on the transmission path of the first message; The first node determines that the first message is a deduplication message based on the deduplication mark; The first node obtains the repeated content from a data set according to the indication information, where the data set includes at least part of the payload part of the historical message received by the first node from the second node; The first node performs deduplication recovery processing on the payload part of the first message according to the repeated content to obtain a second message, wherein the payload part of the second message includes the repeated content; The first node sends the second message to a third node, and the third node is the next hop of the first message on the first node. The method according to claim 18 is characterized in that the repeated content includes one or more repeated data blocks, the indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repeated data blocks, and each of the indications is used to indicate the hash value of the corresponding repeated data block. The method according to claim 19, characterized in that each of the indications is further used to indicate the position of the corresponding repeated data block in the payload portion of the original message corresponding to the first message, and the data set includes the payload portion of the historical message received by the first node from the second node; The first node obtains the repeated content from the data set according to the indication information, including: For each indication in the indication information, the first node obtains, according to the position indicated by the indication, a data block to be matched at the position of the payload part in the data set; The first node calculates a hash value of the to-be-matched data block; The first node determines the to-be-matched data block whose hash value is consistent with the hash value indicated by the indication as the duplicate data block corresponding to the indication. data block. The method according to claim 19 is characterized in that the payload part includes a protocol part and a data part, and the one or more repeated data blocks are located in the data part. The method according to claim 21, characterized in that the data set includes a correspondence between a hash value of a historical data block and the historical data block, the historical data block being a data block obtained by sampling a preset position of a data portion of a historical message received by the first node from the second node; The first node obtains the repeated content from the data set according to the indication information, including: The first node determines the historical data block in the data set corresponding to the hash value indicated by the indication in the indication information as the repeated data block corresponding to the indication. The method according to claim 22, characterized in that the indication is also used to indicate the position of the corresponding repeated data block in the data portion of the original message corresponding to the first message; The first node performs deduplication recovery processing on the payload part of the first message according to the duplicate content, including: For each indication in the indication information, the first node adds a repeated data block corresponding to the indication at the position indicated by the indication in the data part of the first message. The method according to any one of claims 21 to 23 is characterized in that the repeated content also includes protocol information located in the protocol part, the indication information also includes a difference indication, the difference indication is used to indicate the difference between the protocol part of the original message corresponding to the first message and the protocol part of the target message, the target message is a historical message received by the first node from the second node, the data part of which and the data part of the original message have the one or more repeated data blocks; the data set also includes the protocol part of the message to which the historical data block belongs; The first node acquires the repeated content from the data set according to the indication information, further comprising: The first node obtains, from the data set, a protocol portion of the target message to which the one or more repeated data blocks belong; The first node performs deduplication recovery processing on the payload part of the first message according to the duplicate content, including: The first node modifies the protocol part of the target message according to the difference indication, and uses the modified protocol part of the target message as the protocol part of the second message. The method according to any one of claims 18 to 24, characterized in that the deduplication mark is located in the payload part of the first message, and one or more flow group sets are stored in the first node, each of the flow group sets includes flow identifiers of multiple flows flowing through the first node; Before the first node determines that the first message is a deduplication message based on the deduplication mark, the method further includes: The first node determines that there is a target flow group set including a flow identifier of a flow to which the first message belongs in the one or more flow group sets; The first node parses the payload part of the first message to obtain the deduplication mark. The method according to claim 25, characterized in that the first node obtains the repeated content from the data set according to the indication information, comprising: The first node obtains the repeated content from the payload content of the target historical message in the data set according to the indication information, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow grouping set. The method according to claim 25 or 26, characterized in that the method further comprises: The first node receives a third message, and the flow identifier of the flow to which the third message belongs does not belong to any of the flow grouping sets; The first node forwards the third message. The method according to any one of claims 25 to 27, characterized in that the method further comprises: The first node receives a fourth message, where the flow identifier of the flow to which the fourth message belongs belongs to the flow group set; The first node parses the payload part of the fourth message and determines that the payload part of the fourth message does not carry the deduplication mark; The first node adds at least part of the content of the payload part of the fourth message to the data set, and forwards the fourth message. The method according to any one of claims 25 to 28, characterized in that the method further comprises: The first node adds the flow identifiers of the flows to which the messages with duplicate contents in the payload part belong among the multiple messages received and belonging to different flows to the same flow grouping set, and the senders of the different flows are all the SFU servers. The method according to any one of claims 25 to 29, characterized in that the method further comprises: The first node sends grouping information to the second node, where the grouping information includes a correspondence between a node identifier of the first node and the one or more flow grouping sets. The method according to any one of claims 18 to 30, characterized in that the method further comprises: After receiving the fifth message whose destination port number is the SFU service port number, the first node sends a first node discovery message to the fourth node, the fourth node is the next hop of the fifth message on the first node, the destination of the fifth message is the SFU server, the first node discovery message carries an identifier of the SFU server, and the first node discovery message indicates that the first node is a subordinate node of the fourth node on the transmission path starting from the SFU server; In response to receiving a first node discovery response message corresponding to the first node discovery message sent by the fourth node, the first node determines that the fourth node supports data deduplication. The method according to claim 31, characterized in that the method further comprises: The first node generates the first node discovery message according to the fifth message, the message header of the first node discovery message is the same as the message header of the fifth message, and the payload part of the first node discovery message carries an indication of the message type of the first node discovery message. The method according to any one of claims 18 to 32, characterized in that the method further comprises: The first node receives a second node discovery message sent by the fifth node, where the second node discovery message carries an identifier of the SFU server, and the second node discovery message indicates that the fifth node is an upper-level node of the first node on a transmission path starting from the SFU server; The first node determines that the fifth node supports data deduplication based on the second node discovery message, and sends a second node discovery response message corresponding to the second node discovery message to the fifth node, and the second node discovery response message indicates that the first node supports data deduplication. The method according to any one of claims 18 to 33, characterized in that the method further comprises: After the first node receives the sixth message whose source port number is the SFU service port number, the third node discovery message is sent to the sixth node, the sixth node is the next hop of the sixth message on the first node, the sender of the sixth message is the SFU server, the third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the upper node of the sixth node on the transmission path starting from the SFU server; In response to not receiving the third node discovery response message corresponding to the third node discovery message sent by the sixth node, the first node determines that the first node is the last node supporting data deduplication on the transmission path starting from the SFU server. A data transmission method, characterized in that the method comprises: The first node receives a first message sent by the second node, where the sender of the first message is a selective forwarding unit SFU server, and the first node is an intermediate node supporting data deduplication on a transmission path of the first message; If the first message is a non-deduplicated message, and there is a first original message in the historical messages sent by the first node to the third node, and the payload part of the first original message has a first repeated content with the payload part of the first message, the first node performs deduplication processing on the payload part of the first message to obtain a second message, the second message does not include the first repeated content, and the second message carries A message having a deduplication mark and first indication information of the first repeated content, wherein the deduplication mark is used to indicate that the second message is a deduplication message, and the third node is the next hop of the first message on the first node; The first node sends the second message to the third node. The method according to claim 35, characterized in that the method further comprises: If the first message is a non-deduplicated message and the first original message does not exist in the historical messages sent by the first node to the third node, the first node sends the first message to the third node. The method according to claim 35 or 36 is characterized in that the first repeated content includes one or more repeated data blocks, the first indication information includes one or more indications, the one or more indications correspond one-to-one to the one or more repeated data blocks, and each of the indications is used to indicate the hash value of the corresponding repeated data block. The method according to claim 37, characterized in that a first data set is stored in the first node, the first data set includes a payload portion of a historical message sent by the first node to the third node, and the method further comprises: The first node performs content matching on a payload part of the first message and a payload part in the first data set; If the first data set contains a target payload part having a repeated data block with the payload part of the first message, the first node determines that the first original message exists in the historical message; If the first data set does not contain a payload part having a repeated data block with the payload part of the first message, the first node determines that the first original message does not exist in the historical message, and the first node adds the payload part of the first message to the first data set. The method according to claim 38, wherein the first data set includes the target payload part, and the first node performs deduplication processing on the payload part of the first message, comprising: For each repeated data block between the payload part and the target payload part of the first message, the first node calculates a hash value of the repeated data block; The first node removes the repeated data block in the payload part of the first message, and adds an indication corresponding to the repeated data block in the payload part of the first message, wherein the indication is used to indicate a hash value of the repeated data block and a position of the repeated data block in the payload part of the first message. The method according to claim 37 is characterized in that the payload part includes a protocol part and a data part, and the one or more repeated data blocks are located in the data part of the first message; the first node stores a sampling label set, and the sampling label set includes a hash value of a historical data block, and the historical data block is a data block obtained by sampling a preset position of the data part of the historical message sent by the first node to the third node; the method further includes: The first node samples the preset position of the data part of the first message to obtain a sampled data block; The first node calculates a hash value of the sampled data block; If the sampling tag set includes the hash value of the sampling data block, the first node determines that the first original message exists in the historical message; If the sampling tag set does not include the hash value of the sampling data block, the first node determines that the first original message does not exist in the historical message, and the first node adds the hash value of the sampling data block to the sampling tag set. The method according to claim 40, wherein the sampling tag set includes a hash value of the sampling data block, and the first node performs deduplication processing on the payload part of the first message, comprising: The first node uses the sampling data block whose hash value belongs to the sampling tag set as a repeated data block, removes the repeated data block from the data part of the first message, and adds an indication corresponding to the repeated data block in the payload part of the first message, where the indication is used to indicate the hash value of the repeated data block. The method according to claim 41 is characterized in that the first node also stores the first node's request to the third node The protocol part of the historical message sent by the point; the first repeated content also includes protocol information located in the protocol part of the first message, and the first indication information also includes a difference indication, which is used to indicate the difference between the protocol part of the first message and the protocol part of the first original message. The method according to claim 35, characterized in that the method further comprises: If the first message is a deduplicated message, the first message carries second indication information of the second repeated content, and there is a second original message in the historical message sent by the first node to the third node, the payload part of the second original message includes the second repeated content, and the first node sends the first message to the third node. The method according to claim 35, characterized in that the method further comprises: If the first message is a deduplicated message, the first message carries second indication information for the second duplicate content, and the second original message does not exist in the historical messages sent by the first node to the third node, the payload part of the second original message includes the second duplicate content, and the first node obtains the second duplicate content from a second data set according to the second indication information, and the second data set includes at least part of the payload part of the historical message received by the first node from the second node; The first node performs deduplication recovery processing on the payload part of the first message according to the second repeated content to obtain a third message, where the payload part of the third message includes the second repeated content; The first node sends the third message to the third node. The method according to claim 44 is characterized in that the second repeated content includes one or more repeated data blocks, and the second indication information includes one or more indications, and the one or more indications correspond one-to-one to the one or more repeated data blocks, and each of the indications is used to indicate the hash value of the corresponding repeated data block. The method according to claim 45, characterized in that each of the indications is further used to indicate the position of the corresponding repeated data block in the payload portion of the original message corresponding to the first message, and the second data set includes the payload portion of the historical message received by the first node from the second node; The first node acquires the second repeated content from the second data set according to the second indication information, including: For each indication in the second indication information, the first node acquires, according to the position indicated by the indication, a data block to be matched at the position of the payload part in the second data set; The first node calculates a hash value of the to-be-matched data block; The first node determines the to-be-matched data block whose hash value is consistent with the hash value indicated by the indication as the duplicate data block corresponding to the indication. The method according to claim 45 is characterized in that the payload part includes a protocol part and a data part, and the one or more repeated data blocks are located in the data part; the second data set includes a correspondence between a hash value of a historical data block and the historical data block, and the historical data block is a data block obtained by sampling a preset position of a data part of a historical message received by the first node from the second node; The first node acquires the second repeated content from the second data set according to the second indication information, including: The first node determines the historical data block in the second data set corresponding to the hash value indicated by the indication in the second indication information as the repeated data block corresponding to the indication. The method according to claim 47, characterized in that the indication is also used to indicate the position of the corresponding repeated data block in the data portion of the original message corresponding to the first message; The first node performs deduplication recovery processing on the payload part of the first message according to the second repeated content, including: For each indication in the second indication information, the first node adds a repeated data block corresponding to the indication at the position indicated by the indication in the data part of the first message. The method according to claim 47 or 48, characterized in that the second repeated content also includes a protocol message located in the protocol part. The second indication information further includes a difference indication, wherein the difference indication is used to indicate a difference between a protocol part of an original message corresponding to the first message and a protocol part of a target message, wherein the target message is a historical message received by the first node from the second node, in which a data part and a data part of the original message have the one or more repeated data blocks; the second data set also includes a protocol part of a message to which the historical data block belongs; The first node acquires the second repeated content from the second data set according to the second indication information, further comprising: The first node obtains, from the second data set, a protocol portion of the target message to which the one or more repeated data blocks belong; The first node performs deduplication recovery processing on the payload part of the first message according to the second repeated content, including: The first node modifies the protocol part of the target message according to the difference indication, and uses the modified protocol part of the target message as the protocol part of the third message. The method according to any one of claims 35 to 49, characterized in that one or more local flow grouping sets are stored in the first node, each of the local flow grouping sets includes flow identifiers of multiple flows flowing through the first node; the method further comprises: After determining that a target flow group set including a flow identifier of a flow to which the first message belongs exists in the one or more local flow group sets, the first node parses a payload part of the first message; If the payload part of the first message carries a deduplication flag, the first node determines that the first message is a deduplication message; If the payload part of the first message does not carry a deduplication mark, the first node determines that the first message is a non-deduplication message. The method according to claim 50, characterized in that the method further comprises: The first node adds the flow identifiers of the flows to which the messages with duplicate contents in the payload part belong among the received multiple messages belonging to different flows to the same local flow grouping set, and the senders of the different flows are all the SFU servers. The method according to claim 50 or 51, characterized in that the method further comprises: The first node sends first grouping information to the second node, where the first grouping information includes a correspondence between a node identifier of the first node and the one or more local flow grouping sets. The method according to any one of claims 35 to 52, characterized in that the first node stores one or more lower-level flow grouping sets corresponding to the third node, each of the lower-level flow grouping sets includes flow identifiers of multiple flows flowing through the third node, and after the first node receives the first message sent by the second node, the method further includes: If there is a target flow group set including the flow identifier of the flow to which the first message belongs in the lower-level flow group set corresponding to the third node, the first node determines whether there is a message having a payload part having duplicate content with the payload part of the first message in the target historical message sent to the third node, and the flow identifier of the flow to which the target historical message belongs belongs to the target flow group set; If all lower-level flow group sets corresponding to the third node do not include the flow identifier of the flow to which the first message belongs, the first node sends the first message to the third node. The method according to claim 53, characterized in that the method further comprises: The first node receives second grouping information sent by the third node, where the second grouping information includes a correspondence between a node identifier of the third node and the one or more lower-level flow grouping sets. The method according to any one of claims 35 to 54, characterized in that the method further comprises: After receiving the fourth message whose destination port number is the SFU service port number, the first node sends a first node discovery message to the fourth node, where the fourth node is the next hop of the fourth message on the first node, and the destination of the fourth message is the SFU server. The first node discovery message carries an identifier of the SFU server, and the first node discovery message indicates that the first node is a subordinate node of the fourth node on a transmission path starting from the SFU server. In response to receiving a first node discovery response message corresponding to the first node discovery message sent by the fourth node, the first node determines that the fourth node supports data deduplication. The method according to any one of claims 35 to 55, characterized in that the method further comprises: The first node receives a second node discovery message sent by the fifth node, where the second node discovery message carries an identifier of the SFU server, and the second node discovery message indicates that the fifth node is an upper-level node of the first node on a transmission path starting from the SFU server; The first node determines that the fifth node supports data deduplication based on the second node discovery message, and sends a second node discovery response message corresponding to the second node discovery message to the fifth node, and the second node discovery response message indicates that the first node supports data deduplication. The method according to any one of claims 35 to 56, characterized in that the method further comprises: After the first node receives the fifth message whose source port number is the SFU service port number, the third node discovery message is sent to the sixth node, the sixth node is the next hop of the fifth message on the first node, the sender of the fifth message is the SFU server, the third node discovery message carries the identifier of the SFU server, and the third node discovery message indicates that the first node is the upper node of the sixth node on the transmission path starting from the SFU server; In response to receiving a third node discovery response message corresponding to the third node discovery message sent by the sixth node, the first node determines that the sixth node supports data deduplication. The method according to any one of claims 35 to 57, characterized in that the method further comprises: The first node receives a fourth node discovery message sent by the seventh node, where the fourth node discovery message carries an identifier of the SFU server, and the fourth node discovery message indicates that the seventh node is a subordinate node of the first node on a transmission path starting from the SFU server; The first node determines that the seventh node supports data deduplication based on the fourth node discovery message, and sends a fourth node discovery response message corresponding to the fourth node discovery message to the seventh node, and the fourth node discovery response message indicates that the first node supports data deduplication. A data transmission system, characterized in that it comprises: a selective forwarding unit SFU server and a plurality of nodes in a communication network, the plurality of nodes comprising a first node and a second node, the first node being located between the SFU server and the second node; The SFU server is used to send a message to the first node, the first node is used to execute the method according to any one of claims 1 to 17, and the second node is used to execute the method according to any one of claims 18 to 34. The system according to claim 59 is characterized in that the multiple nodes also include a third node, the third node is located between the first node and the second node, and the third node is used to execute the method according to any one of claims 35 to 58. A data transmission system, characterized in that it comprises: a selective forwarding unit SFU server and a first node in a communication network; The SFU server is used to execute the method according to any one of claims 1 to 17, and the first node is used to execute the method according to any one of claims 18 to 34. The system according to claim 61 is characterized in that the communication network also includes a second node, the second node is located between the SFU server and the first node, and the second node is used to execute the method described in any one of claims 35 to 58. A communication node, characterized in that it comprises: a processor and a memory; The memory is used to store a computer program, wherein the computer program includes program instructions; The processor is used to call the computer program to implement the method according to any one of claims 1 to 58.