CN116684397A - Data item delimiting method, device, equipment and storage medium of data transmission protocol - Google Patents

Abstract

This specification provides a data item delimitation method, device, device and storage medium for data transmission protocols, which are applied to the data receiving end; receive the data sent by the data sending end, and cache the received data in the first Buffering, determining the first byte of the data cached in the first buffer as the target byte, and performing iteratively until the detection of the data cached in the first buffer is completed: starting from the target byte to detect and divide different data The delimited string of the item; if the delimited string is detected, determine the data after the target byte and before the first delimited string is detected as the data item, and copy the detected data item to the second buffer; The first byte of data after the first delimited string received is determined as the target byte. The solution provided in this manual can realize the correct delimitation of data items and improve the accuracy of data analysis when data transmission protocols are mixed to transmit data.

Description

Data item demarcation method, device, equipment and storage medium of data transmission protocol

technical field

One or more embodiments of this specification relate to the field of network communication technologies, and in particular, to a data item demarcation method, device, device, and storage medium of a data transmission protocol.

Background technique

Under the data transmission protocol, the data received by the data receiving end (such as the host) each time includes one or more data items, corresponding to one or more data types. When multiple data items of different data types are mixed and transmitted, the transmission content is a byte stream, and the data receiving end parses each data item from the received data stream. In some possible embodiments, since the data length of each data item is unpredictable, it is necessary to use the delimited character string as a boundary between different data items.

For example, in the case of simultaneously transmitting form data and file data based on the http protocol, since the data length of the form-type data items is relatively short, all of them can be cached in the memory buffer. However, the data length of the data item of the file type may be long, and the delimited character string needs to be received several times before it can be completely received, and the memory space is limited, so when the data length of the file data is unpredictable, it is not convenient to keep the file data being received. Buffered in the memory buffer until the reception is complete. Therefore, in related technologies, if form data and file data are transmitted at the same time, usually the default form data is first and the file data is last; correspondingly, the data receiver will first parse the form data from the received data by default, and then file data. That is, if the data receiving end parses the file data, the default form data has been parsed, and the rest is file data.

In this case, if the file data needs to be transmitted first, and then the form data, the data receiving end will mistakenly believe that all the transmission is file data, resulting in an error in parsing the data items of the form data.

Contents of the invention

The present application provides a method for delimiting data items of a data transmission protocol, the data transmission protocol is used to transmit data between a data sending end and a data receiving end; the method is applied to the data receiving end; the data receiving end The data received by the terminal includes one or more data items; the multiple data items correspond to one or more data types; the data length of each data item in the multiple data items is not fixed and unpredictable The data receiving end is configured with a first buffer for delimiting each data item included in the received data, and a second buffer for parsing out each data item from the received data; The methods described include:

receiving the data sent by the data sending end, buffering the received data into the first buffer, determining the first byte of the data buffered into the first buffer as the target byte, and performing the following steps iteratively , until the detection of the data buffered into the first buffer is completed:

starting from said target byte, detecting a delimited string for delimiting different data items;

If the delimited character string is detected, it is determined that the data including the target byte after the target byte and before the first detected delimited character string is a data item, and the detected data Items are copied to said second buffer;

determining the first byte of data after the first detected delimited character string as the target byte.

Optionally, the method also includes:

The receiving the data sent by the data sending end and buffering the data in the first buffer includes:

Cache the received data to the first buffer, point the first cursor to the memory address corresponding to the last byte of the data cached in the first buffer, and point the second cursor to the memory address cached in the first The memory address corresponding to the next byte of the last byte of the data in the buffer;

Wherein, the first cursor is used to indicate the memory address corresponding to the last byte of the detected data cached in the first buffer; the second cursor is used to indicate the memory address cached in the first buffer The memory address corresponding to the first byte of the unfinished detection data in .

Optionally, the received data includes a request message or a response message;

Before determining the first byte of the data buffered into the first buffer as the target byte, the method further includes:

Acquiring the delimited character string contained in the request header of the request message; or acquiring the delimited character string contained in the response header of the response message.

Optionally, starting from the target byte, detecting a delimited character string for dividing different data items includes:

Detect in the first buffer, whether the data before the memory address corresponding to the byte pointed to by the second cursor contains the delimited character string; if the delimited character string is included, point the first cursor to the detected The memory address corresponding to the first byte of the first delimited character string, and the data before the memory address pointed to by the first cursor are divided into data items.

Optionally, after copying the detected data item to the second buffer, the method further includes:

Overwrite the unfinished detection data in the memory space where the first buffer is located to the memory address corresponding to the first byte of the memory space where the first buffer is located.

Optionally, starting from the target byte, after detecting the delimited character string used to divide different data items, the method further includes:

If the delimited character string cannot be detected, point the first cursor to the memory address corresponding to the end byte of the delimited character string, and detect the byte pointed to by the first cursor and the byte pointed to by the second cursor Whether the number of bytes between is not less than the number of bytes occupied by the delimited string;

If it is not less than the number of bytes occupied by the delimited character string, the memory address of the byte pointed to by the first cursor and the forward offset of the second cursor are subtracted from the number of bytes occupied by the delimited character string The data between the memory address pointed to by the number of bytes of a byte is copied to the second buffer; then the remaining data not copied to the second buffer in the first buffer is overwritten Write to the memory address corresponding to the first byte of the memory space corresponding to the first buffer.

Optionally, the detecting whether the number of bytes between the byte pointed to by the first cursor and the byte pointed to by the second cursor is not less than the number of bytes occupied by the delimited string includes:

If it is less than the number of bytes occupied by the delimited character string, directly transfer the data between the memory address corresponding to the byte pointed to by the first cursor and the memory address corresponding to the byte pointed to by the second cursor Write overwriting to the memory address corresponding to the first byte of the memory space where the first buffer is located.

Optionally, the data transmission protocol includes:

HTTP protocol, HTTPS protocol, POP3 protocol and SMTP protocol, etc.

The present application also provides a data item delimitation device for a data transmission protocol, the data transmission protocol is used to transmit data between a data sending end and a data receiving end; the device is applied to the data receiving end; The data received by the data receiving end includes one or more data items; the multiple data items correspond to one or more data types; the data length of each data item in the multiple data items is not fixed and Unpredictable; the data receiving end is configured with a first buffer unit for delimiting each data item included in the received data, and a second buffer unit for parsing each data item from the received data ; the device comprises:

The first buffer unit is used to receive and buffer the data sent by the data sending end, determine the first byte of the data as the target byte, and perform the following steps iteratively until all the buffered data are detected: from the Starting from the target byte, detect the delimited character string used to divide different data items; The data before the first delimited character string is a data item, and the detected data item is copied to the second buffer unit; the first byte of the data after the detected first delimited character string is determined as the described target byte;

The second buffer unit is configured to receive the data item to be parsed sent by the first buffer unit, parse the data item, and provide it for use in a business process.

The present application also provides an electronic device, including a communication interface, a processor, a memory, and a bus, and the communication interface, the processor, and the memory are connected to each other through the bus;

The memory stores machine-readable instructions, and the processor executes the above method by invoking the machine-readable instructions.

The present application also provides a machine-readable storage medium, where the machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the above method is implemented.

Through the above method, on the one hand, by configuring the first buffer for the data receiving end, it is possible to copy the demarcated data items to the second buffer for data parsing after performing pre-parsing processing on the received data. When transmitting data of multiple data types at the same time, the data (such as file data) that may be longer in length and inconvenient to be completely received in the second buffer and demarcated and parsed (such as file data) may also be shorter in length, which is convenient in the second buffer. The data (such as form data) that is completely received and delimited and parsed in the memory is delimited in the memory space where the first buffer is located and analyzed in the memory space where the second buffer is located. When data items of multiple data types are transmitted at the same time, no matter in which order each data item is transmitted, the receiving end can accurately demarcate and analyze the data items, which improves the flexibility and reliability of data transmission. Wherein, the length of the memory occupied by the first buffer zone can realize the technical solution of the present application as long as it is greater than the length of the delimited character string, without caring about the possible length of a single data item, so the additional memory used by the technical solution of the present application can be very small , the memory usage changes very little when the program is running.

On the other hand, since the data before the delimited string is copied to the second buffer in the first buffer, it is equivalent to delimiting data items containing one or more data types in the first buffer (that is, pre-parsing), and then copy the pre-parsed data items to the second buffer, and the second buffer is used for the business process to analyze the pre-parsed data items. Such a solution can accurately demarcate and analyze each data item when the length of each data item in the data stream received by the data receiving end is unpredictable, thereby improving the reliability of data transmission.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments recorded in this specification. , for those skilled in the art, other drawings can also be obtained according to these drawings without paying creative labor.

Fig. 1 is a schematic diagram of the contents of data received by a data receiver receiving HTTP protocol transmission data shown in an exemplary embodiment;

Fig. 2 is a schematic structural diagram of an HTTP data transmission system shown in an exemplary embodiment;

Fig. 3 is a flow chart of a data item delimitation method for a data transmission protocol shown in an exemplary embodiment;

Fig. 4 is a schematic diagram of a first buffer indicating a target byte position shown in an exemplary embodiment;

Fig. 5 is a schematic diagram of the position indicated by the cursor when the first buffer of the data receiving end has not received data, shown in an exemplary embodiment;

Fig. 6 is a schematic diagram of the position indicated by the cursor when the first buffer of the data receiving end receives data according to an exemplary embodiment;

Fig. 7 is a schematic diagram of copying the demarcated data items in the first buffer to the second buffer according to an exemplary embodiment;

Fig. 8 is a schematic diagram showing that a delimited character string cannot be detected after the target byte in the first buffer according to an exemplary embodiment;

Fig. 9 is another schematic diagram showing that a delimited character string cannot be detected after the target byte in the first buffer according to an exemplary embodiment;

Fig. 10 is a hardware structural diagram of an electronic device where a data item demarcation device for a data transmission protocol is shown in an exemplary embodiment;

Fig. 11 is a block diagram of a device for delimiting data items for a data transmission protocol according to an exemplary embodiment.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described The embodiments are only some of the embodiments in this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

It should be noted that in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or less steps than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments describe.

Under the data transmission protocol, the data received by the data receiving end (such as the host) each time includes one or more data items, corresponding to one or more data types. When multiple data items of different data types are mixed and transmitted, the transmission content is a byte stream, and the data receiving end parses each data item from the received data stream. In some possible embodiments, since the data length of each data item is unpredictable, it is necessary to use the delimited character string as a boundary between different data items.

For example, in the case of simultaneously transmitting form data and file data based on the HTTP protocol, since the data length of the form-type data items is relatively short, all of them can be cached in the memory. However, the data length of the data item of the file type may be long, and the delimited character string needs to be received several times before it can be completely received, and the memory space is limited, so when the data length of the file data is unpredictable, it is not convenient to keep the file data being received. Buffered in memory until reception is complete. Therefore, in related technologies, if form data and file data are transmitted at the same time, usually the default form data is first and the file data is last; correspondingly, the data receiver will first parse the form data from the received data by default, and then file data. That is, if the data receiving end parses the file data, the default form data has been parsed, and the rest is file data.

Please refer to FIG. 1 . FIG. 1 is a schematic diagram of content of HTTP protocol transmission data received by a data receiving end in an exemplary embodiment. HTTP is an application layer protocol based on the TCP/IP protocol, because TCP is a streaming protocol, that is, the content of the protocol is a stream of bytes like water, and data is transmitted through continuous, gapless data streams, and each time The length of the data received by the request is variable, so it is necessary to divide the content of the transmission. Therefore, in related technologies, data with shorter lengths (such as form parameters) are generally transmitted first, and data with variable lengths that may be longer (such as file data) are transmitted later, and between different data items. The delimited string is used as a delimiter.

For example, as shown in Figure 1, "-----699156856224027134326940" is a delimited character string used to separate different data items. As shown in Figure 1, in HTTP data transmission, data with a shorter length (such as form data) is placed in front, and data with a variable length (such as file data) is placed in the back, and different data items are separated by character strings separated.

Wherein, in the HTTP protocol message header shown in FIG. 1 , Content-Type is used to represent media type information. When a client requests a service from the server, it only needs to transmit the request method and path. Commonly used request methods are GET, HEAD, and POST. Each method specifies a different type of client-server contact. The Authorization request header field is mainly used to prove that the client has the right to view a certain resource. User-Agent is a request header field that allows the client to tell the server its operating system, browser, and other attributes. It should be noted that FIG. 1 only exemplarily shows a schematic diagram of an interface for transmitting data through the HTTP protocol between a data receiving end and a data sending end, and does not mean that this specification makes a special limitation on the transmission interface.

In this specification, the delimiter character string is generally referred to as boundary, and different parts of data are separated by using a randomly generated delimiter character string. Because the randomly generated delimited string usually has enough randomness to guarantee its uniqueness in the data content. However, if the delimited string is repeated with the data content, an error may occur when the server parses the request, resulting in the inability to process the request correctly. In an actual HTTP request, the delimiter string is usually specified in the boundary parameter of the Content-Type header field. After the boundary string is agreed between the client and the server, different parts can be separated by this string. to ensure that each data item is parsed and processed correctly. The server will split and parse the data in different parts of the request body according to the delimiter string. For example, in the multipart/form-data format, the delimiter string is used to separate form fields and files.

In this case, if the file data needs to be transmitted first, and then the form data, the data receiving end will mistakenly believe that all the transmission is file data, resulting in an error in parsing the data items of the form data.

In view of this, this description aims to propose a data item delimitation method, after the data receiving end receives data containing one or more data items, cache the data in the first buffer, and in the second A technical solution for demarcating data in the first buffer and copying the demarcated data items to the second buffer for analysis.

In this specification, the data transmission protocol is used to transmit data between the data sending end and the data receiving end. The data received by the data receiving end includes one or more data items, the multiple data items correspond to one or more data types, and the data length of each data item in the multiple data items is not fixed and unpredictable. The data receiving end is configured with a first buffer for delimiting each data item included in the received data, and a second buffer for parsing out each data item from the received data.

When the data receiving end receives the data sent by the data sending end, buffer the received data into the first buffer, determine the first byte of the data buffered into the first buffer as the target byte, and iterate Perform the following steps until the data cached to the first buffer is detected: starting from the target byte, detect a delimited character string used to divide different data items; if the delimited character string is detected, determine The data including the target byte after the target byte and before the detected first delimited character string is a data item, and the detected data item is copied to the second buffer ; Further, determining the first byte of the data after the detected first delimited character string as the target byte.

It can be seen that, in the technical solution in this specification, on the one hand, by configuring the first buffer zone for the data receiving end, after the pre-parsing processing of the data boundary is performed on the received data, the data items that have completed the boundary can be copied To the second buffer for data analysis, so that when transmitting data of multiple data types at the same time, the length may be longer, and it is not convenient to completely receive and demarcate and analyze data in the second buffer (such as file data) It can also be combined with the data (such as form data) that is shorter in length and is convenient for complete reception in memory, data delimitation and analysis (such as form data) in the memory space where the first buffer is located and in the memory space where the second buffer is located. Do data analysis in. When data items of multiple data types are transmitted at the same time, no matter in which order each data item is transmitted, the receiving end can accurately demarcate and analyze the data items, which improves the flexibility and reliability of data transmission.

On the other hand, since the data before the delimited string is copied to the second buffer in the first buffer, it is equivalent to delimiting data items containing one or more data types in the first buffer (that is, pre-parsing), and then copy the pre-parsed data items to the second buffer, and the second buffer is used for the business process to analyze the pre-parsed data items. Such a solution can accurately demarcate and analyze each data item when the length of each data item in the data stream received by the data receiving end is unpredictable, thereby improving the reliability of data transmission.

The present application will be described below in combination with specific application scenarios by using specific embodiments and taking the HTTP protocol as an example for the data transmission protocol.

Please refer to FIG. 2 . FIG. 2 is a schematic structural diagram of an HTTP data transmission system shown in an exemplary embodiment. As shown in FIG. 2 , the data receiving end may be a host 20 , a mobile phone 21 and other terminals capable of receiving and analyzing data for the HTTP protocol, also known as an HTTP client. The data sending end may be the server 22 . The data receiving end and the data sending end are connected through the network, and data exchange is performed through the HTTP protocol.

In the case where the data sending end is a server and the data receiving end is a terminal, for example, as shown in FIG. document, wherein the HTTP request includes information such as a request header and a request body. After receiving the HTTP request, the server 22 will parse the request message and perform corresponding data processing, such as querying data from the database and generating a response message. After the server completes data processing, it will return a response message to the host 20 or mobile phone 21. The host 20 and mobile phone 21 receive the response message, analyze the response message body, and perform application-related processing on the analyzed data items.

Wherein, the server is a computer that provides services and has a large amount of concurrent processing capability and data storage and management capabilities. During the data transmission process, it mainly completes parsing HTTP request messages, processing HTTP requests, and generating HTTP response messages. The HTTP request message is a request message initiated by the client to the server, and includes two parts: a request header and a request body. The request header is used to describe the basic information of the HTTP request, such as request type, request path, request header fields, request parameters, etc. are stored in the HTTP request header. The request body is used to describe what data needs to be submitted to the server. For example, when a form is submitted, the form data is stored in the HTTP request body. The response message is a response message from the server to the client, including status code, response header, response body, etc., the status code represents the processing result of the request by the server, and the response header includes some additional HTTP protocol header information, The response body contains the actual data returned by the server to the client.

In some possible embodiments, the host computer 20 and the mobile phone 21 can be not only data receiving terminals, but also data transmitting terminals, sending user operation information (such as search keywords, order information, etc.) to the server 22 . This specification does not limit the types of devices specifically included in the data receiving end and the data sending end.

Please refer to FIG. 3 . FIG. 3 is a flow chart of a data item delimitation method for a data transmission protocol shown in an exemplary embodiment. The method can be applied to the data receiving end. The method may perform the following steps:

Step 302: Receive the data sent by the data sending end, buffer the received data in the first buffer, and determine the first byte of the data buffered in the first buffer as the target byte.

For example, please refer to FIG. 4 , which is a schematic diagram of a first buffer indicating a target byte position according to an exemplary embodiment. As shown in Figure 4, the rectangular frame represents the first buffer 400. When the data "abc123-xyz-def456-xyz-ghi789-xy" is received in the first buffer 400, the first byte of the data is "a", and the target The byte 402 is used to indicate the position of the first byte of the received data in the first buffer 400 , that is, the target byte 402 indicates the position of the data "a".

Wherein, the first byte of the data buffered in the first buffer refers to the first byte of the data in the data stream that has been received and stored in the first buffer. The first buffer refers to a data storage area used to receive data and perform data pre-parsing (that is, data demarcation). In this manual, the first buffer is set to divide the data items and copy them to the second buffer for data processing, so as to realize the demarcation in advance of the data whose length is unpredictable and not fixed, and which is not convenient to receive and cache all in the memory. In this specification, the target byte is used to indicate the position of a certain byte and used for subsequent operations. The target byte is used to divide the data area and detect whether there is a delimited character string in the data area. Setting the target byte is beneficial to indicate the process and steps of data demarcation more clearly.

In one embodiment shown, in order to indicate the detection position more intuitively, the receiving the data sent by the data sending end and buffering it in the first buffer includes:

Cache the received data to the first buffer, point the first cursor to the memory address corresponding to the first byte of the data cached in the first buffer, and point the second cursor to the memory address cached in the first The memory address corresponding to the next byte of the last byte of the data in the buffer;

For example, please refer to FIG. 5 . FIG. 5 is a schematic diagram of a position indicated by a cursor when the first buffer of the data receiving end has not received data in an exemplary embodiment. As shown in FIG. 5 , the rectangular frame represents the first buffer 400 , which is used to cache data that needs to be decomposed into data items. When the first buffer 400 has not received data, both the first cursor 502 and the second cursor 504 are used to indicate the position of the first byte in the first buffer 400 .

Wherein, the first cursor is used to indicate the memory address corresponding to the first byte of the unfinished detection data cached in the first buffer; the second cursor is used to indicate the memory address cached in the first buffer The memory address corresponding to the next byte of the last byte of the uncompleted detected data.

When data is stored in the first buffer, please refer to FIG. 6 . FIG. 6 is a schematic diagram of a position indicated by a cursor when the first buffer of the data receiving end receives data according to an exemplary embodiment. As shown in Figure 6, since the data has not yet been parsed, the first cursor 502 is used to indicate the memory address corresponding to the first byte "a" of the unfinished detection data cached in the first buffer, and the second cursor 504 It is used to indicate the memory address corresponding to the next byte of the last byte "y" of the unfinished detected data cached in the first buffer.

Wherein, the memory address corresponding to the first byte of the undetected data refers to the memory address where the first byte of the data that has been received but not detected in the first buffer is stored in the first buffer , that is, the memory address corresponding to the next byte of the last byte of the detected data in the first buffer. The cursor (Cursor) is a mark indicating a current position or an operating position, and is usually used for browsing, accessing or modifying data. A cursor can also indicate the current position in a data stream or resource, enabling precise access, reading, or modification of data at a specific location. Additionally, cursors can be used to traverse collections of data, processing each element or record individually.

In one embodiment shown, the received data includes a request message or a response message; before the first byte of the data buffered in the first buffer is determined as the target byte, the The method also includes:

Acquiring the delimited character string contained in the request header of the request message; or acquiring the delimited character string contained in the response header of the response message.

Wherein, when the data receiving end receives the HTTP request message, the delimited character string is obtained from the request header of the request message, and after the data receiving end sends the HTTP request message, the corresponding message is received In the case, the delimited character string is obtained from the response header of the response message. In order to obtain the delimited character string, the data receiving end first parses the request message, and the parsing request message uses the function provided by the corresponding network library or framework to parse the received request message into processable data structure; then obtain the request header part from the parsed request message and search for the specific request header field, and finally find the specific field containing the required delimited string; finally, according to the specific needs, parse the value of the field to Get the delimited string.

Step 304: Starting from the target byte, detect a delimited character string for dividing different data items.

For example, as shown in FIG. 4 , starting from the target byte 402 , it is detected whether the data stored in the first buffer 400 has a delimited character string for dividing different data items.

Wherein, the delimited character string is used to divide the boundary of data items, if the delimited character string is detected in the data stored in the first buffer 400, it means that the data in front of the delimited character string and the The data after the delimited string are two different data items, such as different form data, or one form data and one file data.

Step 306: Determine whether a delimited character string is detected; if yes, proceed to step 308.

Step 308: Determine that the data including the target byte after the target byte and before the detected first delimited character string is a data item, and copy the detected data item to the Second buffer.

For example, please refer to FIG. 7 . FIG. 7 is a schematic diagram of copying demarcated data items in the first buffer to the second buffer in an exemplary embodiment. As shown in FIG. 7, the delimited character string is a character string "-xyz-", starting from the target byte 402, detecting whether the data stored in the first buffer 400 has a delimited character string for dividing different data items, When the existence of the first delimited character string “-xyz-” is detected, the data item “abc123” before the detected first delimited character string “-xyz-” is copied to the second buffer 700 .

Wherein, the second buffer is the place where the file data items or parameter data items parsed from the first buffer are finally stored after the delimited string detection, and the data stored in the second buffer is used for subsequent application layer services Process usage.

Step 310: Determine the first byte of the data after the detected first delimited character string as the target byte.

For example, as shown in FIG. 7 , the first byte “d” of the data after the detected first delimited character string “-xyz-” is determined as the target byte 402 .

Wherein, the target byte is used to indicate a starting position for next detection of whether the delimited character string exists in the first buffer 400 . In this specification, the purpose of setting the target byte to move to the first byte of the data after the delimited character string after copying the data item is to avoid repeated detection and repeated copying operations on the same data item. After step 310 is executed, return to step 304 to continue to execute the flow of steps in a loop.

Step 312: Determine whether the detection is completed on the data buffered in the first buffer; if no, proceed to step 304; if yes, end the iteration.

In one embodiment shown, in order to indicate the detection position more intuitively, starting from the target byte, detecting the delimited character string used to divide different data items includes:

Detect in the first buffer, whether the data before the memory address corresponding to the byte pointed to by the second cursor contains the delimited character string; if the delimited character string is included, point the first cursor to the detected The first delimits the memory address corresponding to the first byte of the character string, and divides the data before the memory address pointed to by the first cursor into data items.

For example, as shown in FIG. 7 , detect whether the data before the memory address corresponding to the byte that the second cursor 504 points to in the first buffer 400 contains the delimited character string "-xyz-"; because in the first buffer 400 The cached data contains the delimited character string "-xyz-", and the first cursor 502 is pointed to the detected first delimited character string (ie the delimited character string "-xyz-" after the character string "abc123") The memory address corresponding to the first byte "-", and the data "abc123" before the memory address pointed to by the first cursor 502 is divided into data items.

Wherein, the second buffer is equivalent to the memory buffer in the background technology, and the first buffer newly added by the technical solution of the present application performs data delimitation pre-analysis on the received data, and demarcates the delimited The data items are copied to the second buffer for data parsing.

In one embodiment shown, in order to correctly identify the complete delimited character string and improve the accuracy of parameter demarcation when the delimited character string only appears in a part of the first buffer, starting from the target byte, After detecting a delimited string for delimiting different data items, the method further includes:

If the delimited character string cannot be detected, point the first cursor to the memory address corresponding to the end byte of the delimited character string, and detect the byte pointed to by the first cursor and the byte pointed to by the second cursor Whether the number of bytes between is not less than the number of bytes occupied by the delimited string;

If it is not less than the number of bytes occupied by the delimited character string, the memory address of the byte pointed to by the first cursor and the forward offset of the second cursor are subtracted from the number of bytes occupied by the delimited character string The data between the memory address pointed to by the number of bytes of a byte is copied to the second buffer; then the remaining data not copied to the second buffer in the first buffer is overwritten Write to the memory address corresponding to the first byte of the memory space corresponding to the first buffer.

For example, please refer to FIG. 8 , which is a schematic diagram of an exemplary embodiment showing that a delimited character string cannot be detected after the target byte in the first buffer. As shown in FIG. 8 , after the target byte 402 , only a part of “-xy” of the delimited character string appears in the first buffer 400 . Now the first cursor 502 points to the next byte "g" of the last delimited character string "-xyz-" in the first buffer 400, and the second cursor 504 points to the received The byte next to the end byte "y" of the data. Detect the number of bytes between the byte "g" pointed to by the first cursor 502 and the next byte of the byte "y" pointed to by the second cursor 504 (that is, the number of bytes from "g" to "y") Whether the number of bytes, excluding the next byte of "y" is not less than the number of bytes occupied by the delimited character string "-xyz-" (5 bytes).

Wherein, the number of bytes occupied by the delimited string "-xyz-" is 5 bytes, and the number of bytes occupied by the delimited string minus one byte is 4 bytes. The number of bytes between byte "g" and the next byte of byte "y" is 9 bytes, which is less than 5 bytes. Then the memory address of the byte "g" pointed to by the first cursor 502 and the forward offset of the second cursor 504 delimit the number of bytes (5 bytes) occupied by the character string "-xyz-" by one The data "ghi78" between the memory address of the byte "9" pointed to by the number of bytes (that is, 4 bytes) is copied to the second buffer 700, and the first cursor is updated synchronously to point to the byte "9 "The memory address where it is located.

In one embodiment shown, the detection whether the number of bytes between the byte pointed by the first cursor and the byte pointed by the second cursor is not less than the bytes occupied by the delimited string number, including:

If it is less than the number of bytes occupied by the delimited character string, directly transfer the data between the memory address corresponding to the byte pointed to by the first cursor and the memory address corresponding to the byte pointed to by the second cursor Overwriting writes to the memory address corresponding to the first byte of the memory space where the first buffer is located, and synchronously updates the first cursor and the second cursor to the address of the first byte of the target position and the next byte of the last byte respectively. The address where the byte is located.

For example, please refer to FIG. 9 . FIG. 9 is another schematic diagram showing that no delimited character string is detected after the target byte in the first buffer according to an exemplary embodiment. As shown in FIG. 9 , after the target byte 402 , only a part of “-xy” of the delimited character string appears in the first buffer 400 . The number of bytes between the byte "g" and the next byte of the byte "y" is 4 bytes, which is smaller than the number of bytes occupied by the delimited character string "-xyz-" (5 bytes). At this time, directly between the memory address of the byte "g" pointed to by the first cursor 502 and the memory address corresponding to the next byte of the byte "y" pointed to by the second cursor 504 The data "g-xy" (the next byte not including "y") is overwritten to the memory address corresponding to the first byte of the memory space where the first buffer 400 is located, and the first cursor is updated to point to the data The memory address where the byte "g" in "g-xy" is located, and the second cursor is updated to point to the memory address corresponding to the next byte of the byte "y" in the data "g-xy".

It should be noted that, in the embodiment shown above, it is equivalent to retaining the data that may have "a part of the delimited character string" in the first buffer 400, when the first buffer 400 subsequently receives When newly cached data, the presence of the complete delimited string can be detected. Wherein, the complete delimited string includes a hyphen connecting the delimited strings.

In an illustrated embodiment, the data transmission protocol includes:

HTTP protocol, HTTPS protocol, POP3 protocol and SMTP protocol, etc.

Among them, HTTP (Hypertext Transfer Protocol, Hypertext Transfer Protocol) protocol, HTTPS (HyperText Transfer Protocol Secure, secure Hypertext Transfer Protocol) protocol, POP3 (Post Office Protocol version 3, third edition Post Office Protocol) protocol and SMTP (Simple Mail Transfer Protocol, Simple Mail Transfer Protocol, etc., all need to delimit data items by delimiting strings to ensure that data is transmitted and parsed in the correct format.

Corresponding to the above embodiments of the data item delimitation method for the data transmission protocol, this specification also provides an embodiment of a data item delimitation device for the data transmission protocol.

Please refer to FIG. 10 . FIG. 10 is a hardware structural diagram of an electronic device in which a data item demarcation device for a data transmission protocol is located according to an exemplary embodiment. In terms of hardware, the device includes a processor 1002, an internal bus 1004, a network interface 1006, a memory 1008, and a non-volatile memory 1010, and may also include other required hardware. One or more embodiments of this specification may be implemented based on software, for example, the processor 1002 reads a corresponding computer program from the non-volatile memory 1010 into the memory 1008 and executes it. Of course, in addition to software implementations, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each A logic unit, which can also be a hardware or logic device.

Please refer to FIG. 11 . FIG. 11 is a block diagram of an apparatus for delimiting data items for a data transmission protocol according to an exemplary embodiment. The device for delimiting data items for a data transmission protocol can be applied to an electronic device as shown in FIG. 10 , so as to realize the technical solution of this specification. The data item demarcation device for the data transmission protocol is applied to the data receiving end; the data received by the data receiving end includes one or more data items; the multiple data items correspond to one or more data types ; The data length of each data item in the plurality of data items is not fixed and unpredictable; the data receiving end is configured with a first buffer for delimiting each data item included in the received data , and a second buffer for parsing out each data item from the received data; the means may include:

The first buffer unit 1102 is configured to receive and buffer the data sent by the data sending end, determine the first byte of the data as the target byte, and perform the following steps iteratively until all the buffered data are detected: from all Starting from the target byte, detect the delimited character string used to divide different data items; if the delimited character string is detected, then determine that the target byte, including the target byte, is located after the target byte and is located at the detected The data before the first delimited character string is a data item, and the detected data item is copied to the second buffer unit; the first byte of the data after the detected first delimited character string is determined as the the target byte;

The second buffering unit 1104 is configured to receive the data item to be parsed sent by the first buffering unit, parse the data item, and provide it for use in a business process;

For the implementation process of the functions and effects of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method for details, and will not be repeated here.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in a place, or can also be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution in this specification. It can be understood and implemented by those skilled in the art without creative effort.

The systems, devices, modules, or units described in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer, which may take the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, e-mail device, game control device, etc. desktops, tablets, wearables, or any combination of these.

In a typical configuration, a computer includes one or more processors (CPUs), input/output interfaces, network interfaces and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. Memory is an example of computer readable media.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by computing devices. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are authorized by the user or Information and data that have been fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The foregoing describes specific embodiments of this specification. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

Terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only, and are not intended to limit one or more embodiments of this specification. As used in one or more embodiments of this specification and the appended claims, the singular forms "a", "the", and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in one or more embodiments of the present specification to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of the present specification, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

The above descriptions are only preferred embodiments of one or more embodiments of this specification, and are not intended to limit one or more embodiments of this specification. Within the spirit and principles of one or more embodiments of this specification, Any modification, equivalent replacement, improvement, etc. should be included in the scope of protection of one or more embodiments of this specification.

Claims (10)

1. A method of delimiting data items for a data transmission protocol, wherein the data transmission protocol is used for transmitting data between a data transmitting end and a data receiving end; the method is applied to the data receiving end; the data received by the data receiving end comprises one or more data items; the plurality of data items corresponding to one or more data types; the data length of each of the plurality of data items is not fixed and unpredictable; the data receiving end is configured with a first buffer zone for delimiting each data item contained in the received data and a second buffer zone for analyzing each data item from the received data; the method comprises the following steps:

receiving data sent by the data sending end, caching the received data into the first buffer area, determining the first byte of the data cached into the first buffer area as a target byte, and iteratively executing the following steps until the data cached into the first buffer area is detected:

Starting from the target byte, detecting separation strings for dividing different data items;

if the separation character string is detected, determining that the data which contains the target byte and is positioned behind the target byte and before the first detected separation character string is a data item, and copying the detected data item to the second buffer area;

and determining the first byte of the data positioned after the detected first separation character string as the target byte.

2. The method of claim 1, wherein the receiving the data sent by the data sending end and buffering the data in the first buffer area includes:

caching the received data into the first buffer area, pointing a first cursor to a memory address corresponding to the first byte of the data cached into the first buffer area, and pointing a second cursor to a memory address corresponding to the next byte of the last byte of the data cached into the first buffer area;

the first cursor is used for indicating a memory address corresponding to the first byte of the incompletely detected data cached in the first buffer area; the second cursor is used for indicating a memory address corresponding to a next byte of the last byte of the data which is buffered in the first buffer zone and is not detected yet.

3. The method of claim 1, wherein the received data comprises a request message or a response message;

before determining the first byte of the data buffered in the first buffer as the target byte, the method further comprises:

acquiring the separation character strings contained in the request header of the request message; or, acquiring the separation character string contained in the response header of the response message.

4. The method of claim 2, wherein detecting a separation string for dividing different data items from the target byte comprises:

detecting whether the data before the memory address corresponding to the second cursor pointing byte contains the separation character string or not in the first buffer area; and if the separation character string is included, the first cursor points to the memory address corresponding to the first byte of the detected first separation character string, and the data before the memory address pointed by the first cursor is divided into data items.

5. The method of claim 2, wherein, starting from the target byte, after detecting a separation string for dividing different data items, the method further comprises:

If the separation character string cannot be detected, the first cursor points to a memory address corresponding to the last byte of the separation character string, and whether the number of bytes between the byte pointed by the first cursor and the byte pointed by the second cursor is not smaller than the number of bytes occupied by the separation character string is detected;

if the number of bytes occupied by the separation character string is not smaller than the number of bytes occupied by the separation character string, copying data between the memory address of the byte pointed by the first cursor and the memory address pointed by the byte number of one byte subtracted from the number of bytes occupied by the separation character string by shifting the second cursor forwards to the second buffer area; and then, the residual data which is not copied to the second buffer area in the first buffer area is written to a memory address corresponding to the first byte of the memory space corresponding to the first buffer area in an overlapping mode.

6. The method of claim 5, wherein detecting whether a number of bytes between the byte pointed to by the first cursor and the byte pointed to by the second cursor is not less than a number of bytes occupied by the separation string comprises:

if the number of bytes occupied by the separation character string is smaller than the number of bytes occupied by the separation character string, directly writing data between the memory address corresponding to the byte pointed by the first cursor and the memory address corresponding to the byte pointed by the second cursor into the memory address corresponding to the first byte of the memory space where the first buffer is located in a covering mode.

7. The method of claim 1, wherein the data transmission protocol comprises:

HTTP protocol, HTTPs protocol, POP3 protocol, and SMTP protocol.

8. A data item delimiting means for a data transmission protocol, characterized in that the data transmission protocol is used for transmitting data between a data transmitting end and a data receiving end; the device is applied to the data receiving end; the data received by the data receiving end comprises one or more data items; the plurality of data items corresponding to one or more data types; the data length of each of the plurality of data items is not fixed and unpredictable; the data receiving end is configured with a first buffer unit for delimiting each data item contained in the received data and a second buffer unit for analyzing each data item from the received data; the device comprises:

the first buffer unit is used for receiving and buffering the data sent by the data sending end, determining the first byte of the data as a target byte, and iteratively executing the following steps until the buffered data are detected: starting from the target byte, detecting separation strings for dividing different data items; if the separation character string is detected, determining that data which contains the target byte and is positioned behind the target byte and before the first detected separation character string is a data item, and copying the detected data item to the second buffer unit; determining a first byte of data located after the detected first separation string as the target byte;

The second buffer unit is used for receiving the data item to be analyzed sent by the first buffer unit, analyzing the data item and providing service flow for use.

9. An electronic device comprises a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are connected with each other through the bus;

the memory stores machine readable instructions, the processor executing the method of any of claims 1 to 7 by invoking the machine readable instructions.

10. A machine-readable storage medium storing machine-readable instructions which, when invoked and executed by a processor, implement the method of any one of claims 1 to 7.