WO2018161881A1 - Structuralized data processing method, data storage medium, and computer apparatus - Google Patents

Abstract

A structuralized data processing method comprises: determining, according to a combined submission strategy, whether a first write request and a second request stored in a write operation queue have a same write operation type, wherein the first write request comprises first structuralized data to be written and a corresponding write operation type, and the second write request comprises second structuralized data to be written and a corresponding write operation type; if the first write request and the second request have the same write operation type, combining the first write operation and the second write operation into a batch file of write requests comprising the first structuralized data and the second structuralized data; and storing, according to the batch file of the write requests, and in a data storage layer, the first structuralized data and the second structuralized data.

Description

Structured data processing method, storage medium, and computer device

This application claims to be filed on March 9, 2017, the Chinese Patent Office, the application number is 2017101385414, and the priority of the Chinese patent application entitled "Processing Data Processing Method and Distributed Processing System" is adopted. The citations are incorporated herein by reference.

Technical field

The present application relates to the field of computer technologies, and in particular, to a method for processing structured data, a storage medium, and a computer device.

Background technique

With the development of network technology, the requirements for the transmission and processing of information are getting higher and higher, and the information that can be represented by data or a unified structure is called structured data, such as numbers and symbols. Structured data has specific fields, that is, row data, which is stored in the database. The two-dimensional table structure can be used to logically express the implemented data. For example, the user uses social software to post a talk, with an identifier (Identifier, ID), time, title, body, and the like.

In the prior art, when processing structured data, a scheme is generally adopted in which a client sends a write request to a logical layer in a data processing system. To ensure atomicity, the data processing system routes the write request according to a list key (List Key). The write operation queue is queued and written to the storage layer in turn, and returns the result requested by the client in turn. At this time, the delay of writing data = storage layer processing delay + queuing delay. In the prior art, when processing structured data is processed, the data processing system uses queues to sequentially write data one by one. In a scenario where the amount of concurrent data is relatively large, many write requests are timed out to be written. The storage layer failed to cause the client to write data.

Summary of the invention

In accordance with various embodiments provided herein, a method of processing structured data, a storage medium, and a computer device are provided.

A method of processing structured data, comprising:

The computer device determines, according to the merge submission policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type, and the first write request includes: first structured data to be written and corresponding a write operation type, where the second write request includes: second structured data to be written and a corresponding write operation type;

If the first write request and the second write request have the same write operation type, the computer device merges the first write request and the second write request into one batch write request, the batch write The request includes: the first structured data and the second structured data; and

The computer device stores the first structured data and the second structured data into a data storage layer according to the bulk write request.

One or more non-volatile storage media storing computer readable instructions, when executed by one or more processors, cause one or more processors to perform the following steps:

Determining, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type, where the first write request includes: first structured data to be written and a corresponding write operation a type, the second write request includes: second structured data to be written and a corresponding write operation type;

And if the first write request and the second write request have the same write operation type, combining the first write request and the second write request into one batch write request, where the batch write request includes: Decoding the first structured data and the second structured data; and

The first structured data and the second structured data are stored into the data storage layer according to the bulk write request.

A computer device comprising a memory and a processor, the memory storing computer readable instructions, the computer readable instructions being executed by the processor such that the processor performs the following steps:

Determining, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type, where the first write request includes: first structured data to be written and a corresponding write operation a type, the second write request includes: second structured data to be written and a corresponding write operation type;

And if the first write request and the second write request have the same write operation type, combining the first write request and the second write request into one batch write request, where the batch write request includes: Decoding the first structured data and the second structured data; and

The first structured data and the second structured data are stored into the data storage layer according to the bulk write request.

Details of one or more embodiments of the present application are set forth in the accompanying drawings and description below. Other features, objects, and advantages of the invention will be apparent from the description and appended claims.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention. Other drawings may also be obtained from those skilled in the art from the drawings.

1 is a schematic block diagram of a method for processing structured data according to an embodiment of the present application;

2 is a schematic diagram of a data processing scenario of a method for processing structured data according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of composition content of index structure information according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an application scenario of a computer device according to an embodiment of the present disclosure;

FIG. 5-a is a schematic structural diagram of a computer device according to an embodiment of the present application;

FIG. 5-b is a schematic structural diagram of a component of a submitting module in a computer device according to an embodiment of the present disclosure;

FIG. 5-c is another schematic structural diagram of a computer device according to an embodiment of the present application;

FIG. 5-d is another schematic structural diagram of a computer device according to an embodiment of the present application;

FIG. 5-e is a schematic structural diagram of a structure of a queue storage module in a computer device according to an embodiment of the present application;

FIG. 5-f is another schematic structural diagram of a computer device according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure when implemented as a server; and

FIG. 7 is a schematic diagram showing the internal structure of a computer device in an embodiment.

detailed description

The embodiment of the present application provides a method for processing structured data, a storage medium, and a computer device, which are used to improve processing efficiency of a write request and reduce queuing delay of a write operation queue.

In order to make the object, the features and the advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. The described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the present application are within the scope of the present disclosure.

The terms "including" and "comprising", and any variations thereof, are intended to cover a non-exclusive inclusion in order to include a series of units of processes, methods, systems, or products. The devices are not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or devices.

Through the description of the following embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, dedicated hardware, dedicated CPU, dedicated memory, dedicated memory, Special components and so on. In general, functions performed by computer programs can be easily implemented with the corresponding hardware, and the specific hardware structure used to implement the same function can be various, such as analog circuits, digital circuits, or dedicated circuits. Circuits, etc. However, software program implementation is a better implementation for more applications in this application. Based on such understanding, the technical solution of the present application, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a readable storage medium, such as a floppy disk of a computer. , U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., including a number of instructions to make a computer device (may be A personal computer, server, or network device, etc.) performs the methods described in various embodiments of the present application. It should be understood that the computer device is implemented as a distributed processing system as described in the embodiments below to perform the various steps in the embodiments described below.

The details are described below separately.

An embodiment of the method for processing structured data of the present application may be specifically applied to batch processing of structured data to improve data processing efficiency. In the embodiment of the present application, the structured data refers to a data including a specific field, which can be logically expressed by a two-dimensional table structure. For example, a piece of information published on a social account is a structured data, which may include Publish the identifier (Identifier, ID), time, title, body and other fields. Referring to FIG. 1 , a method for processing structured data provided by an embodiment of the present application may include the following steps:

101. Determine, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type, where the first write request includes: the first structured data to be written and the corresponding write operation. The type, the second write request includes: second structured data to be written and a corresponding write operation type.

In the embodiment of the present application, a distributed submission system is configured with a merge submission policy for a write request, and the distributed processing system analyzes and determines a plurality of write requests stored in the write operation queue according to the merge submission policy, thereby determining Whether there are at least two write requests with the same write operation type in the write operation queue. The merge submission policy may include multiple implementation manners, for example, a polling read write request in the write operation queue may be periodically polled to determine multiple write requests that are simultaneously added or added to the write operation queue within a certain period of time. Whether it is possible to perform batch processing. The merge submission policy may be determined by an operation user of the distributed processing system, configured in a user-configured manner to the distributed processing system, or may be determined by the distributed processing system according to a queue storage condition of the write operation queue, for example, according to The number of write requests added to the write operation queue accounts for the capacity ratio of the write operation queue to determine whether to execute the merge submission policy in the embodiment of the present application. In the subsequent embodiment of the present application, the first write request and the second write request are stored in the write operation queue as an example to describe the implementation of the batch processing, and the actual write operation queue may also be added. More write requests.

The first write request and the second write request may be from the same client, or may be from two clients, that is, the write request in the write operation queue of the distributed processing system may be calculated according to the write frequency of the user. For example, if the personal computer and the mobile phone use the same user name to post, the personal computer and the mobile phone respectively submit multiple write requests to the distributed processing system as different clients. In the embodiment of the present application, each write request added to the write operation queue carries information of a write operation type. Specifically, the first write request includes: first structured data to be written and a corresponding write operation. The type, the second write request includes: second structured data to be written and a corresponding write operation type. For example, as follows, the write operation type may include an add operation, a modify operation, and a delete operation. Different write operation types are different operations on the structured data, so it is possible to determine whether multiple write requests can be merged by the judgment of the write operation type. Batch processing.

In some embodiments of the present application, the method provided by the embodiment of the present application further includes:

A1. Receive a first write request sent by the client.

A2: Adding the first write request to the write operation queue, and triggering to perform the following step 101: determining, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type.

It can be understood that steps A1 and A2 can be performed before step 101 according to the merge commit policy to determine whether the first write request and the second write request stored in the write operation queue have the same write operation type.

A communication link is established between the distributed processing system and the multiple clients. When the client needs to store the structured data, the client sends a write request to the distributed processing system. For example, the client sends the first write. The request, the distributed processing system may add the first write request to the write operation queue, so that the first write request waits for processing in the write operation queue, and according to the merge submission policy, the first write request newly added to the write operation queue and Whether the write request being queued in the write operation queue has the same write operation type. It should be noted that the same client may also send a second request, or another client sends a second request to the distributed processing system, and the distributed processing system processes the second write request in a similar manner, and details are not described herein.

Further, in some embodiments of the present application, step A2 adds the first write request to the write operation queue, including:

A21: Acquire, from the first write request, the first structured data and the write operation type, the service identifier, the list identifier (ListKey), and the row identifier (SubKey) corresponding to the first structured data;

A22. The first structured data and the write operation type, the service identifier, the ListKey, and the SubKey corresponding to the first structured data are stored in the first index structure information in the write operation queue.

The first write request sent by the client may include the first structured data, and the write request may include: a service identifier, a list identifier, and a row identifier, in addition to the write operation type of the first structured data, The structured data and the write operation type, the service identifier, the ListKey, and the SubKey corresponding to the first structured data are stored in the first index structure information in the write operation queue, and the index structure information can store the request content carried in the write request, thereby Through the index structure, structured storage of data can be realized, which facilitates the judgment of the type of write operation and the reading of the service identifier, the list identifier, and the row identifier of the structured data according to the merge submission policy. The service identifier is a character that can uniquely identify a service, and the ListKey is a string that can uniquely identify a list, and the row identifier can uniquely represent a row in the list.

102. If the first write request and the second write request have the same write operation type, merge the first write request and the second write request into one batch write request, where the batch write request includes: the first structured data and the second structure Data.

In the embodiment of the present application, by the judgment of step 101, the distributed processing system combines the first write request and the second write request into one if the first write request and the second write request have the same write operation type. A batch write request, wherein the bulk write request comprises: first structured data and second structured data. A batch write request is obtained by performing a merge batch process on multiple write requests having the same write operation type. The write operation type of the bulk write request refers to a merged new write operation type, for example, multiple add operations are merged into one batch add operation. Types of. In the embodiment of the present application, after the write request is sent by the client to the distributed processing system, the distributed processing system adds the write request to the write operation queue. In the embodiment of the present application, multiple write requests in the write operation queue are not required. The queue is processed in turn, and the batch processing request can be used to reduce the queue processing delay of multiple write requests.

In some embodiments of the present application, step 102 combines the first write request and the second write request into one bulk write request, including:

B1, extracting first structured data from the first write request, and extracting second structured data from the second write request;

B2. Generate a batch write request according to the first structured data and the second structured data.

B3. Add a bulk write request to the write operation queue, and delete the first write request and the second write request in the write operation queue.

The distributed processing system may extract the first structured data and the second structured data by parsing the first write request and the second write request in the write operation queue, and after the batch write request is generated, the write operation is performed. The original write request is deleted from the queue, which simplifies the management overhead of the write operation queue.

In some embodiments of the present application, the method provided by the embodiment of the present application further includes:

C1, establishing a first mapping relationship between the SubKey corresponding to the first structured data and the first link file descriptor, and a second mapping relationship between the SubKey and the second link file descriptor corresponding to the second structured data;

C2: Generate a return package table according to the first mapping relationship and the second mapping relationship, and store the return package table into the batch write request.

It will be appreciated that steps C1 and C2 may be performed after the first write request and the second write request are merged into one bulk write request in step 102.

Each structured data corresponds to a link file descriptor, and the link file descriptor is an index information identifying the connection of the user request result, and the return packet table can be generated by mapping the SubKey and the link file descriptor. . The function of the return package table is to find the connection that responds to the user result through the return package table after the user's request batch processing is completed.

103. Store the first structured data and the second structured data into the data storage layer according to the batch write request.

In the embodiment of the present application, after the first write request and the second write request are merged into one batch write request by the step 102, the batch write request includes the first structured data and the second structured data that need to be written. The distributed processing system can store the first structured data and the second structured data into the data storage layer according to the batch write request. In the embodiment of the present application, the plurality of structured data in the batch write request may be simultaneously stored in the data storage layer without being stored in sequence, and the storage delay of the plurality of structured data may be reduced by the batch write request.

In some embodiments of the present application, the method provided by the embodiment of the present application further includes:

D1. Obtain a batch execution result corresponding to the batch write request.

D2. The storage result of the first structured data and the storage result of the second structured data are respectively obtained from the batch execution result;

D3. Respond to the storage result of the first structured data to the client that sends the first write request, and reply the storage result of the second structured data to the client that sends the second write request.

It can be understood that steps D1, D2 and D3 can be performed after the first structured data and the second structured data are stored in the data storage layer according to the bulk write request in step 103.

After the processing of the plurality of structured data is completed in the data storage layer of the distributed processing system, the storage result of the plurality of structured data may be separately obtained, for example, the first structured data is separately obtained from the batch execution result. The storage result and the storage result of the second structured data are respectively sent back to the corresponding client for each write request, so that the client can know whether the requested structured data is successfully written. For example, in the foregoing implementation scenario of performing steps C1 to C2, the distributed processing system may obtain the first link file descriptor and the second link file descriptor through the return packet table, and reply the requested structure to the client. The result of storing the data.

According to the description of the embodiments of the present application, the first write request and the second write request stored in the write operation queue have the same write operation type, for the first write request and the second write. If the request has the same write operation type, the first write request and the second write request may be combined into one batch write request, and finally the first structured data and the second structured data are stored to the data storage layer according to the batch write request. in. In the embodiment of the present application, multiple write requests in the write operation queue do not need to be queued for sequential processing, and multiple structured writes to be written have the same write operation type, and may merge multiple write requests into one batch write request. Therefore, multiple structured data can be written into the data storage layer by one process, which reduces queue delay and processing delay, and improves the processing efficiency of structured data.

To facilitate a better understanding and implementation of the foregoing solutions of the embodiments of the present application, the following application scenarios are specifically illustrated.

The embodiment of the present application discloses a method for merging processing when a structured data is submitted. Specifically, the client submits a write request to the access module of the distributed processing system through the interface, and the access module writes the requested Listkey route according to the strip. To the logic module for queuing, the logic module merges the write requests within a certain period according to the merge commit policy, and submits a batch write request to the data storage layer. This kind of write operation merges the commit mechanism, which greatly reduces the number of interactions between the logic module and the data storage layer, saves the processing time required for the logic module and the system resources, and thus the above process improves the efficiency of processing the structured data. In the embodiment of the present application, the logic module may merge the multiple write requests in a certain period into a batch request according to the merge submission policy, and write to the data storage layer uniformly, and parse the returned result to respectively reply to the corresponding client, so that the logic Layer merging reduces the queuing delay and the data storage layer processing requests are merged, reducing latency. FIG. 2 is a schematic diagram of a data processing scenario of a method for processing structured data provided by an embodiment of the present application. The logic module merges the write request in a certain period into a batch write request to the data storage layer according to the merge submission policy, and parses the returned result, and respectively responds to the corresponding client, mainly including the following steps:

1) After the access module obtains the write request, parsing the signaling packet to obtain a ListKey. The listkey binary data is converted into an unsigned type, and the number of each device node in the modulo logic module, that is, through the hash calculation of the listkey, the write request is allocated to a device node of the logic module for logical processing. After the address is obtained, the data is forwarded to the node of the logic module.

Please refer to FIG. 3 , which is a schematic diagram of the composition content of the index structure information provided by the embodiment of the present application. When writing data, it is required to record a write operation type (Flag), a service identifier (BID), a list identifier, task information, service data, and the like. information. Among them, a service identifier such as a service name, a list identifier such as a list name, and the like. The information is stored in the index structure information, and its storage structure is as follows:

Among them, the task information Info structure is defined as follows:

2) After receiving the data, the logic module puts the write operation queue according to the BID and the ListKey. Regularly check the task attributes on the queue, as shown in Figure 2. If the same type of add operation, merge into a new operation type, avoiding the queuing delay of a single commit. The original operation is deleted from the write operation queue, and the mapping relationship between the subkey and the link file descriptor is established, thereby generating a return package table.

3) Submitting a batch write request of the merged data to the data storage layer, which reduces the processing delay of the data storage layer.

4) The data storage layer returns the execution result of the batch write request, and then splits the merged write request into a single write operation request.

5) The logic module replies the request result to multiple links according to the return packet table, and the client does not need to make any adaptation.

Next, please refer to FIG. 4 , which is a schematic diagram of an application scenario of a distributed processing system applied to a computer device according to an embodiment of the present application. The distributed structured data processing system is a storage logic platform for providing services for User Generated Content (UGC) data. It supports unlimited growth of user data, and can provide reading functions such as sorting, filtering, and classification, and is suitable for talking. Most UGC business scenarios such as message boards, WeChat friends, and Weibo. The internal structure of the computer device to which the distributed processing system is applied can be referred to the structure shown in FIG. Each of the modules described below can be implemented in whole or in part by software, hardware, or a combination thereof.

The five modules of the distributed processing system include: an access module, a logic processing module, a long list processing module, a node management module, and a repair module. The access module is responsible for access, the service directly requests access, the logical processing module is the core logic of the distributed processing system, the long list processing module is responsible for the processing of large user data sorting and filtering, and the node module is responsible for the configuration management of the entire system, repairing The module is responsible for fixing some scenarios where the logic process fails.

The method of batch merging is provided in the embodiment of the present application. Compared with a single write, batch consolidation processing can be implemented, the delay of queuing delay storage is reduced, and processing requests are also merged, and processing delay is reduced, and the same listkey is determined by actual measurement. The processing power can be increased by 25 times.

It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the present application is not limited by the described action sequence. Because certain steps may be performed in other sequences or concurrently in accordance with the present application. In the following, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

Please refer to FIG. 5-a, a computer device 500 provided by an embodiment of the present application. The internal structure of the computer device 500 can be referred to the structure shown in FIG. Each of the modules described below can be implemented in whole or in part by software, hardware, or a combination thereof. Referring to FIG. 5, the computer device 500 may include: a determining module 501, a merging module 502, and a submitting module 503, where

The determining module 501 is configured to determine, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type, where the first write request includes: a first structuring to be written Data and a corresponding write operation type, the second write request including: second structured data to be written and a corresponding write operation type.

The merging module 502 is configured to combine the first write request and the second write request into one batch write request if the first write request and the second write request have the same write operation type, The batch write request includes: the first structured data and the second structured data.

The submitting module 503 is configured to store the first structured data and the second structured data into the data storage layer according to the bulk write request.

In some embodiments of the present application, referring to FIG. 5-b, the merging module 502 includes:

The data extracting unit 5021 is configured to extract the first structured data from the first write request, and extract the second structured data from the second write request.

The write request aggregating unit 5022 is configured to generate a batch write request according to the first structured data and the second structured data.

The queue storage unit 5023 is configured to add the batch write request to the write operation queue, and delete the first write request and the second write request in the write operation queue.

In some embodiments of the present application, referring to FIG. 5-c, the computer device 500 further includes:

The result obtaining module 504 is configured to obtain a batch execution result corresponding to the batch write request.

The result parsing module 505 is configured to respectively obtain, from the batch execution result, a storage result of the first structured data and a storage result of the second structured data.

a result feedback module 506, configured to reply the storage result of the first structured data to the client that sends the first write request, and reply the second structured data to the client that sends the second write request Stored results.

In some embodiments of the present application, referring to FIG. 5-d, the computer device 500 further includes:

The access module 507 is configured to receive the first write request sent by the client.

The queue storage module 508 is configured to add the first write request to the write operation queue, and trigger the execution of the determining module 501.

In some embodiments of the present application, as shown in FIG. 5-e, the queue storage module 508 includes:

The information extraction module 5081 is configured to obtain, from the first write request, the first structured data and the write operation type, the service identifier, the list identifier ListKey, and the row identifier SubKey corresponding to the first structured data.

The index creation module 5082 is configured to store the first structured data and the write operation type corresponding to the first structured data, the corresponding service identifier, the ListKey, and the SubKey into the first index structure in the write operation queue. Information.

In some embodiments of the present application, please refer to FIG. 5-f. As shown in FIG. 5-a, the computer device 500 further includes:

The mapping module 509 is configured to establish a first mapping relationship between the SubKey corresponding to the first structured data and the first link file descriptor, and a SubKey and a second link file descriptor corresponding to the second structured data. The second mapping relationship.

The packet header generation module 510 is configured to generate a packet header table according to the first mapping relationship and the second mapping relationship, and store the packet header table in the batch write request.

As can be seen from the foregoing description of the embodiments of the present application, first, according to the merge submission policy, it is determined whether the first write request and the second write request stored in the write operation queue have the same write operation type, and the first write request and the second write request exist. In the case of the same write operation type, the first write request and the second write request may be combined into one batch write request, and finally the first structured data and the second structured data are stored into the data storage layer according to the batch write request. In the embodiment of the present application, multiple write requests in the write operation queue do not need to be queued for sequential processing, and multiple structured writes to be written have the same write operation type, and may merge multiple write requests into one batch write request. Therefore, multiple structured data can be written into the data storage layer by one process, which reduces queue delay and processing delay, and improves the processing efficiency of structured data.

FIG. 6 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure when a computer device is specifically implemented as a server. The server 1100 may have a large difference due to different configurations or performances, and may include one or more central processing units (central processing). Units, CPU) 1122 (eg, one or more processors) and memory 1132, one or more storage media 1130 that store application 1142 or data 1144 (eg, one or one storage device in Shanghai). The memory 1132 and the storage medium 1130 may be short-term storage or persistent storage. The program stored on storage medium 1130 may include one or more modules (not shown), each of which may include a series of instruction operations in the server. Still further, central processor 1122 can be configured to communicate with storage medium 1130, executing a series of instruction operations in storage medium 1130 on server 1100.

Server 1100 may also include one or more power sources 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and/or one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM and more.

The method steps performed by the distributed processing system in the above embodiments may be based on the server structure shown in FIG. 6.

FIG. 7 is a schematic diagram showing the internal structure of a computer device in an embodiment. The computer device can be the server 1100 shown in FIG. As shown in FIG. 7, the computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein, the memory comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device can store operating system and computer readable instructions. The computer readable instructions, when executed, may cause the processor to perform a method of processing structured data. The processor is used to provide computing and control capabilities to support the operation of the entire computer device. Computer readable instructions may be stored in the internal memory of the computer device, and when the computer readable instructions are executed by the processor, the processor may be caused to perform a method of processing structured data. It will be understood by those skilled in the art that the structure shown in FIG. 7 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation of the computer device to which the solution of the present application is applied. The specific computer device may It includes more or fewer components than those shown in the figures, or some components are combined, or have different component arrangements.

It should be further noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be Physical units can be located in one place or distributed to multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, and specifically may be implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement without any creative effort.

One of ordinary skill in the art can understand that all or part of the process of implementing the above embodiments can be completed by a computer program to instruct related hardware, and the program can be stored in a non-volatile computer readable storage medium. Wherein, the program, when executed, may include the flow of an embodiment of the methods as described above. Any reference to a memory, storage, database or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. Non-volatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of formats, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronization chain. Synchlink DRAM (SLDRAM), Memory Bus (Rambus) Direct RAM (RDRAM), Direct Memory Bus Dynamic RAM (DRDRAM), and Memory Bus Dynamic RAM (RDRAM).

The technical features of the above embodiments may be arbitrarily combined. For the sake of brevity of description, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, It is considered to be the range described in this specification.

In summary, the above embodiments are only used to explain the technical solutions of the present application, and are not limited thereto; although the present application is described in detail with reference to the above embodiments, those skilled in the art should understand that they can still The technical solutions described in the above embodiments are modified, or the equivalents of the technical features are replaced by the same, and the modifications and substitutions do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (18)

A method of processing structured data, comprising: The computer device determines, according to the merge submission policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type, and the first write request includes: first structured data to be written and corresponding a write operation type, where the second write request includes: second structured data to be written and a corresponding write operation type; If the first write request and the second write request have the same write operation type, the computer device merges the first write request and the second write request into one batch write request, the batch write The request includes: the first structured data and the second structured data; and The computer device stores the first structured data and the second structured data into a data storage layer according to the bulk write request. The method of claim 1, wherein the computer device merges the first write request and the second write request into a batch write request, comprising: The computer device extracts the first structured data from the first write request, and extracts the second structured data from the second write request; The computer device generates a batch write request according to the first structured data and the second structured data; and The computer device adds the bulk write request to the write operation queue and deletes the first write request and the second write request in the write operation queue. The method of claim 1 further comprising: Obtaining, by the computer device, a batch execution result corresponding to the batch write request; Obtaining, by the computer device, the storage result of the first structured data and the storage result of the second structured data, respectively, from the batch execution result; The computer device replies to the client that sends the first write request to the storage result of the first structured data, and replies to the client that sends the second write request to the storage result of the second structured data . The method of claim 1 further comprising: Receiving, by the computer device, the first write request sent by a client; and The computer device adds the first write request to the write operation queue, and triggers the following steps: determining, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation Types of. The method according to claim 4, wherein the adding the first write request to the write operation queue comprises: Obtaining, by the first write request, the first structured data and the write operation type, the service identifier, the list identifier ListKey, and the row identifier SubKey corresponding to the first structured data; The computer device stores the first structured data and a write operation type corresponding to the first structured data, a corresponding service identifier, a ListKey, and a SubKey into the first index structure information in the write operation queue. The method of claim 5, wherein the method further comprises: The computer device establishes a first mapping relationship between the SubKey corresponding to the first structured data and the first link file descriptor, and a second key corresponding to the second structured data and a second link file descriptor Mapping relationship; and The computer device generates a return package table according to the first mapping relationship and the second mapping relationship, and stores the return package table into the batch write request. One or more non-volatile storage media storing computer readable instructions, when executed by one or more processors, cause one or more processors to perform the following steps: Determining, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type, where the first write request includes: first structured data to be written and a corresponding write operation a type, the second write request includes: second structured data to be written and a corresponding write operation type; And if the first write request and the second write request have the same write operation type, combining the first write request and the second write request into one batch write request, where the batch write request includes: Decoding the first structured data and the second structured data; and The first structured data and the second structured data are stored into the data storage layer according to the bulk write request. The storage medium according to claim 7, wherein the combining the first write request and the second write request into one batch write request comprises: Extracting the first structured data from the first write request, and extracting the second structured data from the second write request; Generating a batch write request according to the first structured data and the second structured data; and Adding the bulk write request to the write operation queue, and deleting the first write request and the second write request in the write operation queue. The storage medium of claim 7, wherein when the computer readable instructions are executed by one or more processors, the one or more processors further cause the following steps: Obtaining a batch execution result corresponding to the batch write request; Obtaining, from the batch execution result, a storage result of the first structured data and a storage result of the second structured data; and Responding to the storage result of the first structured data to the client that sends the first write request, and replying to the storage result of the second structured data to the client that sends the second write request. The storage medium of claim 7, wherein when the computer readable instructions are executed by one or more processors, the one or more processors further cause the following steps: Receiving the first write request sent by the client; and Adding the first write request to the write operation queue, and triggering the following step: determining, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type. The storage medium according to claim 10, wherein the adding the first write request to the write operation queue comprises: Acquiring, by the first write request, the first structured data and the write operation type, the service identifier, the list identifier ListKey, and the row identifier SubKey corresponding to the first structured data; and And storing the first structured data and the write operation type corresponding to the first structured data, the corresponding service identifier, the ListKey, and the SubKey into the first index structure information in the write operation queue. The storage medium of claim 11 wherein when said computer readable instructions are executed by one or more processors, the one or more processors further cause the following steps: Establishing a first mapping relationship between the SubKey corresponding to the first structured data and the first link file descriptor, and a second mapping relationship between the SubKey and the second link file descriptor corresponding to the second structured data; and Generating a return package table according to the first mapping relationship and the second mapping relationship, and storing the return package table into the batch write request. A computer device comprising a memory and a processor, the memory storing computer readable instructions, the computer readable instructions being executed by the processor such that the processor performs the following steps: Determining, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type, where the first write request includes: first structured data to be written and a corresponding write operation a type, the second write request includes: second structured data to be written and a corresponding write operation type; And if the first write request and the second write request have the same write operation type, combining the first write request and the second write request into one batch write request, where the batch write request includes: Decoding the first structured data and the second structured data; and The first structured data and the second structured data are stored into the data storage layer according to the bulk write request. The computer device according to claim 13, wherein the combining the first write request and the second write request into one batch write request comprises: Extracting the first structured data from the first write request, and extracting the second structured data from the second write request; Generating a batch write request according to the first structured data and the second structured data; and Adding the bulk write request to the write operation queue, and deleting the first write request and the second write request in the write operation queue. The computer apparatus according to claim 13 wherein said computer readable instructions, when executed by said processor, further cause said processor to perform the steps of: Obtaining a batch execution result corresponding to the batch write request; Obtaining, from the batch execution result, a storage result of the first structured data and a storage result of the second structured data; and Responding to the storage result of the first structured data to the client that sends the first write request, and replying to the storage result of the second structured data to the client that sends the second write request. The computer apparatus according to claim 13 wherein said computer readable instructions, when executed by said processor, further cause said processor to perform the steps of: Receiving the first write request sent by the client; and Adding the first write request to the write operation queue, and triggering the following step: determining, according to the merge commit policy, whether the first write request and the second write request stored in the write operation queue have the same write operation type. The computer device according to claim 16, wherein the adding the first write request to the write operation queue comprises: Acquiring, by the first write request, the first structured data and the write operation type, the service identifier, the list identifier ListKey, and the row identifier SubKey corresponding to the first structured data; and And storing the first structured data and the write operation type corresponding to the first structured data, the corresponding service identifier, the ListKey, and the SubKey into the first index structure information in the write operation queue. The computer apparatus according to claim 17, wherein said computer readable instructions, when executed by said processor, further cause said processor to perform the following steps: Establishing a first mapping relationship between the SubKey corresponding to the first structured data and the first link file descriptor, and a second mapping relationship between the SubKey and the second link file descriptor corresponding to the second structured data; and Generating a return package table according to the first mapping relationship and the second mapping relationship, and storing the return package table into the batch write request.

Images