CN103699494A - Data storage method, data storage equipment and distributed storage system - Google Patents

Abstract

The invention discloses a data storage method, data storage equipment and a distributed storage system. The method, the equipment and the system belong to the technical field of computers. The data storage method comprises the steps that access nodes of the distributed storage system cut a data block which is written in by a user into a plurality of data fragments with preset sizes; a plurality of check fragments corresponding to the data fragments which are obtained by cutting are calculated through a redundancy check algorithm; the data fragments and the check fragments are stored into the data nodes of the distributed storage system, and each data fragment and each check fragment only store one copy. According to the data storage method, the data storage equipment and the distributed storage system, the proportion of redundant data can be reduced on the premise of ensuring data reliability, so as to save the storage space.

Description

A data storage method, data storage device and distributed storage system

technical field

The invention relates to the technical field of computers, in particular to a data storage method, a data storage device and a distributed storage system.

Background technique

Cassandra is a typical distributed storage system with a ring structure without central nodes, which relies on distributed hash table (Distributed Hash Table, DHT) technology. DHT is a distributed storage technology. Without the need of a central node, each storage node is responsible for a small range of routing and is responsible for storing a small part of data, thereby realizing the addressing and storage of the entire DHT distributed cluster.

Cassandra's data storage space can be abstracted into a ring structure, and data is dispersed in this ring storage space through hash. Each node of the distributed storage system is responsible for managing a certain continuous range (also called Range) on this ring storage space, and the data falling in this Range space is stored on this node. As shown in Figure 1, in a cluster with four nodes of A/B/C/D, the storage space is divided into four Ranges of R0/R1/R2/R3, and each node is responsible for the Range distribution as shown in the following table Show:

Range node R3 A R0 B R1 C R2 D.

For example, when the hash of data falls on R0, this data is stored on node B.

There are two roles of Cassandra nodes: access nodes and data nodes. The access node is responsible for the distribution of computing data and copies, and the data node is responsible for the storage of data copies. In the Cassandra system, each node exists as both an access node and a data node.

In order to ensure data reliability, Cassandra generally writes multiple copies of data and distributes them on different data nodes. When a copy is lost due to abnormalities such as disk failure or machine failure, other copies can also provide access normally, and can complete the recovery of the lost copy under certain conditions after failure recovery.

In the production environment of Cassandra, a 3-copy strategy is generally adopted: that is, for each data record (or data block), in addition to storing the original data in one data node, the other 2 copies of the data are also stored in other in the two data nodes. As shown in Figure 2, if the hash of the data block falls within the R0 space range, the original data is stored in node B, and the other two copies are stored in node C and node D respectively.

From the realization of existing technology, a data record is stored in Cassandra. In fact, three data copies need to be stored, which occupies a large amount of storage space. The overhead of redundant copies is too large, resulting in waste of space and high storage service costs. And if the number of copies is reduced, such as changing to two copies for storage, the reliability of the data will be greatly reduced.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a data storage method, a data storage device and a distributed storage system that overcome the above problems or at least partially solve the above problems.

According to one aspect of the present invention, a data storage method is provided, comprising:

The access node of the distributed storage system divides the data block written by the user into several data fragments of a predetermined size;

Using the redundancy check algorithm to calculate the check slices corresponding to the sliced data slices;

The data fragments and the verification fragments are stored in the data nodes of the distributed storage system, and only one copy of each data fragment and each verification fragment is stored.

Optionally, the storing the data fragments and the verification fragments in the data nodes of the distributed storage system includes:

Calculate the digital signatures of each data fragment and each verification fragment respectively;

Each data segment and each verification segment are respectively stored in a data node of the distributed storage system according to their respective digital signatures.

Optionally, the data storage method also includes:

Encode all digital signatures to form metadata;

The metadata is stored in multiple data nodes of the distributed storage system.

Optionally, the calculation of the number of check pieces corresponding to the number of data pieces obtained by using the redundancy check algorithm includes:

Dividing the obtained data fragments into several groups of data fragments, wherein each group of data fragments includes a first predetermined number of data fragments;

A second predetermined number of check slices corresponding to each group of data slices are respectively calculated by using a redundancy check algorithm.

Optionally, the data storage method also includes:

Using each group of data fragments and corresponding verification fragments as a stripe unit, encoding all digital signatures corresponding to the stripe unit to form metadata;

The metadata is stored in multiple data nodes of the distributed storage system.

Optionally, the encoding is JSON encoding.

Optionally, the distributed storage system is a Cassandra system.

According to another aspect of the present invention, a data storage device is provided, which is located in an access node of a distributed storage system, and the data storage device includes:

The data segmentation unit is suitable for dividing the data block written by the user into several data fragments of a predetermined size;

The verification calculation unit is adapted to use a redundancy verification algorithm to calculate a plurality of verification fragments corresponding to the plurality of data fragments obtained by segmentation;

The storage allocation unit is adapted to store the data shards and the check shards in the data nodes of the distributed storage system, and only one copy of each data shard and each check shard is stored.

Optionally, the storage allocation unit is further adapted to:

Calculate the digital signatures of each data fragment and each verification fragment respectively;

Each data segment and each verification segment are respectively stored in a data node of the distributed storage system according to their respective digital signatures.

Optionally, the data storage device also includes:

an encoding unit adapted to encode all digital signatures to form metadata;

The metadata storage unit is adapted to store the metadata in multiple data nodes of the distributed storage system.

Optionally, the verification calculation unit is further adapted to:

Dividing the obtained data fragments into several groups of data fragments, wherein each group of data fragments includes a first predetermined number of data fragments;

A second predetermined number of check slices corresponding to each group of data slices are respectively calculated by using a redundancy check algorithm.

Optionally, the data storage device also includes:

An encoding unit adapted to use each group of data fragments and corresponding verification fragments as a stripe unit, and encode all digital signatures corresponding to the stripe unit to form metadata;

The metadata storage unit is adapted to store the metadata in multiple data nodes of the distributed storage system.

Optionally, the encoding is JSON encoding.

Optionally, the distributed storage system is a Cassandra system.

According to still another aspect of the present invention, a distributed storage system including the above-mentioned data storage device is provided.

According to the above-mentioned one or more technical solutions of the present invention, in the distributed storage system, by using the erasable code (Eraser Code, EC) verification method to replace the existing multi-copy storage method, the premise of ensuring data reliability can be Under this circumstance, the proportion of redundant data is reduced, so as to achieve the purpose of saving storage space.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same components. In the attached picture:

Fig. 1 shows the schematic diagram of the storage space division of the Cassandra system;

Fig. 2 shows the schematic diagram of the data storage method according to the Cassandra system of prior art;

Fig. 3 shows the flowchart of the data storage method according to one embodiment of the present invention;

FIG. 4 shows the correspondence between metadata and data blocks in a data storage method according to an embodiment of the present invention;

Fig. 5 shows a structural diagram of a data storage device according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the existing problems of using multi-copy storage in the distributed storage system in the prior art, excessive redundancy copy overhead and high storage service cost, the embodiment of the present invention provides a data storage solution, in the distributed storage system such as The access node of the Cassandra system divides the data block into several data fragments, and calculates several verification fragments, and stores the data fragments and verification fragments in different data nodes in Cassandra. These fragments form a The stripe unit is managed. When a limited number of fragments in the stripe unit are lost due to a fault, the faulty data fragment can be corrected and restored with the help of check information, which can reduce redundant data while ensuring data reliability. ratio, so as to achieve the purpose of saving storage space.

Fig. 3 shows a flowchart of a data storage method according to an embodiment of the present invention. Referring to Figure 3, the data storage method may include:

Step 302, the access node of the distributed storage system divides the data block written by the user into several data fragments of a predetermined size;

The distributed storage system may be a Cassandra system. As mentioned above, Cassandra nodes have two roles: access nodes and data nodes. The access node is responsible for the distribution of computing data and copies, and the data node is responsible for the storage of data copies. In the Cassandra system, each node exists as both an access node and a data node.

In the prior art, when the access node receives the data block written by the user, it first calculates the hash value of the data block (or the key (Key) of the data block), and stores the data block in the Among the corresponding multiple data nodes, it is usually stored in 3 data nodes (the data node corresponding to the Range where the hash value is located, and the other 2 data nodes located clockwise after the data node in the ring storage space).

One difference from the prior art is that, in the embodiment of the present invention, when the access node receives the data block written by the user, it first divides the data block according to a fixed size, that is, The size is fixed (it is convenient to calculate the check information according to the redundancy check algorithm). Of course, the last fragment obtained by segmentation may not reach the predetermined size. At this time, it can be filled with empty data. Among them, the fragment size can be specified as a parameter when the storage is created, and stored in the systemtable of the Cassandra system.

It should be noted that, in the embodiment of the present invention, the distributed storage system is not limited to the Cassandra system, and may also be any other distributed storage system such as the BigTable system, the Dynamo system, and the Hadoop Hbase system.

Step 304, using a redundancy check algorithm to calculate a number of check slices corresponding to the sliced data slices;

Redundancy check algorithm is usually applied in erasable code (Eraser Code, EC) technology, EC is a form of forward error correction (Forward Error Correction, FEC) technology, by calculating and verifying multiple blocks of data, forming Several verification information can automatically correct and restore the faulty data when some data is faulty. In this way, with the help of EC technology, good storage persistence can be achieved with relatively low storage overhead.

Wherein, the redundancy check algorithm may use a parity check algorithm, a cyclic redundancy check algorithm (Cyclic Redundancy Check, CRC) and the like. Generally speaking, for M data fragments, N (N≥1) check fragments can be calculated according to the redundancy check algorithm. Among M+N fragments, if K (K≤N) fragments If a shard fails, it can be recovered by the remaining M+N-K shards that have not failed. N can be selected according to needs. The larger the value of N, the higher the reliability of the data, but the higher the storage space occupied. Correspondingly, the smaller the value of N, the lower the reliability of the data, but the higher the storage space occupied. Also low.

Step 306, storing the data shards and the verification shards in the data nodes of the distributed storage system, and storing only one copy of each data shard and each verification shard.

Another difference from the prior art is that in the embodiment of the present invention, only one copy of each data fragment and each verification fragment is stored in the distributed storage system, that is, each data fragment is stored in In a corresponding data node, each check fragment is also stored in a corresponding data node. In this way, by adopting EC technology, after one or more data fragments in the data block fail, it can be recovered through the unfailed data fragments and check fragments, ensuring the reliability of the data; Only one copy of the shard is stored, which reduces the proportion of redundant data and saves storage space.

There are multiple ways to store only one copy of each data fragment and each verification fragment, and one of the realization manners is given below.

According to an implementation manner of the embodiment of the present invention, in step 306, storing the data fragment and the verification fragment in the data node of the distributed storage system may include:

Step S02, calculating the digital signatures of each data segment and each verification segment respectively;

There are many algorithms for calculating the digital signature of data, and the embodiment of the present invention does not limit the specific algorithm. For example, the secure hash algorithm (secure hash algorithm, SHA1) or the fifth edition of the message digest algorithm (Message Digest Algorithm, MD5).

Step S04, storing each data segment and each verification segment in a data node of the distributed storage system according to their respective digital signatures.

The digital signature calculated in step S02 can be used as the key of the fragment. In step S04, the data node where the fragment is located can be found according to the key, so that the fragment is stored in the corresponding data node.

In order to facilitate addressing of each data fragment when reading a data block (that is, locate the data node storing the data fragment), the method in the embodiment of the present invention may further include: encoding all digital signatures to form metadata, and The metadata is stored in multiple data nodes of the distributed storage system.

For example, suppose the data block database written by the user is divided into 10 data fragments, and the redundancy check algorithm is used to calculate the 2 check fragments corresponding to the 10 data fragments. In this way, the calculation can The 12 digital signatures corresponding to the 12 fragments are: K0, K1, ..., K9, Ec0, Ec1. As shown in Fig. 4, these 12 digital signatures are organized together in sequence and encoded to form metadata (Meta), and the data nodes corresponding to each fragment can be located according to the metadata.

In the implementation of the present invention, the encoding can adopt JSON (JavaScript Object Notation) encoding, of course, other encoding algorithms can also be used, which is not limited in the embodiment of the present invention.

In addition, the metadata may be stored in multiple data nodes of the distributed storage system by using an implementation method of the distributed storage system itself. For example, in the Cassandra system, three copies of the metadata can be stored, that is, the metadata are stored in three data nodes respectively. Specifically, you can first calculate the hash value of the metadata, store a copy of the metadata in the data node corresponding to the Range where the hash value is located according to the calculated hash value, and store the other two copies of the metadata respectively to the other 2 data nodes located clockwise after this data node.

Since the sizes of the multiple data blocks written by the user are usually inconsistent, in order to realize the standardization of data encoding and storage, in step 304 of the embodiment of the present invention, the redundant check algorithm is used to calculate the number of data blocks obtained by segmentation. A number of verification slices corresponding to slices can include:

Step S12, dividing the several data fragments obtained by the segmentation into several groups of data fragments, wherein each group of data fragments includes a first predetermined number of data fragments;

The first predetermined number can be set as required, for example, the first predetermined number is set to 5, that is, each group of data fragments includes 10 data fragments. Of course, if after the division, the number of data fragments included in the last group of data fragments may be less than the first predetermined number, at this time, "0" (empty fragments) can be used to fill up to the A first predetermined number.

For example, assuming that 38 data fragments are obtained after dividing the data block, these 38 data fragments can be divided into 4 groups of data fragments according to the above steps, and the data fragments included in each group of data fragments The number of is 10, and the last set of data shards includes 2 empty shards.

Step S14, using a redundancy check algorithm to respectively calculate a second predetermined number of check slices corresponding to each group of data slices.

The second predetermined number may be based on a trade-off between data reliability factors and storage space occupancy factors, for example, when the first predetermined number is 10, the second predetermined number is 2 , that is, each group of 10 data fragments corresponds to 2 check fragments.

Afterwards, the first predetermined number of data fragments and the corresponding second predetermined number of verification fragments of each group may be managed as a stripe unit. Correspondingly, the aforementioned metadata corresponds to a stripe unit, that is, the data storage method in the embodiment of the present invention may further include: using each group of data fragments and corresponding verification fragments as a stripe unit, and storing the All digital signatures corresponding to the stripe units are encoded to form metadata, and the metadata is stored in multiple data nodes of the distributed storage system. For the specific implementation of encoding and storage of metadata, reference may be made to the descriptions of Step S02 and Step S04 above, which will not be repeated here.

An application example of the data storage method of the embodiment of the present invention is given below. In this application example, the distributed storage system is a Cassandra system, and the encoding is JSON encoding.

(1) The user writes a data block at the access node, the key of the data block is 'dummy.key', and the value of the data block includes 10000 bytes of data, then the data block can be According to the fixed size (1000 bytes), it is divided into 10 pieces: 1...1000 bytes, 1001...2000 bytes,..., 9001...10000 byte.

(2) Calculate key values (such as SHA1 signatures) for the 10 data fragments, as follows:

34f11e0febf1a6d43f66f5af738667fec9ac6e5c

e27efcfe1450f97d21b51c612587488dae3d356e

e4a192a7d494f3691bd1fff1a8195eecdf2ed76b

f8d9a3e143bc275e4859736f37de0ce280aae034

12116b77653d73825530056f4d9f322abc771e34

87e1e55c6684ab33eb4ea104b31bc5595340e934

61306782f48c25332729e2fca1efa29d3709067d

c04b64f9b709488fbee7282f9c7f1c6f443293ab

e401b1b341e0bfe2a1989f99c22353b8e7b68427

b29ab838829fa69443230f20e8fc5fbc1fbbfe28

(3) Use the redundancy verification algorithm to calculate 2 pieces of verification fragments, and calculate the key value (such as SHA1 signature) for these 2 verification fragments, respectively as follows:

8936986814f5f38a83d2f0809f2e36f4572996c2

09e77c8ef46aaccaa76c4c71fa757fb68ccd0a7d

(4) Treat 10 data fragments and 2 verification fragments as a stripe unit, and the JSON encoding of the 12 key values corresponding to the stripe unit is:

[″34f11e0febf1a6d43f66f5af738667fec9ac6e5c″，″e27efcfe1450f97d21b51c612587488dae3d356e″，″e4a192a7d494f3691bd1fff1a8195eecdf2ed76b″，″f8d9a3e143bc275e4859736f37de0ce280aae034″，″12116b77653d73825530056f4d9f322abc771e34″，″87e1e55c6684ab33eb4ea104b31bc5595340e934″，″61306782f48c25332729e2fcalefa29d3709067d″，″c04b64f9b709488fbee7282f9c7f1c6f443293ab″，″e401b1b341e0bfe2a1989f99c22353b8e7b68427″，″b29ab838829fa69443230f20e8fc5fbc1fbbfe28″，″8936986814f5f38a83d2f0809f2e36f4572996c2″，″09e77c8ef46aaccaa76c4c71fa757fb68ccd0a7d″]

(5) Store the 10 data fragments and 2 check fragments contained in the stripe unit in the corresponding data nodes of Cassandra according to the corresponding key values. Specifically, the fragments can only be stored in the Range where the key is located. in the data node without storing another 2 copies;.

(6) Write the JSON encoding of the 12 key values into the Cassandra cluster as metadata. Metadata still adopts a multi-copy redundancy scheme.

In this way, when one piece of data is lost due to disk failure of the node where it resides, it can be repaired by the other 11 pieces of data, thereby ensuring data reliability.

In this application example, 2 verification shards are calculated from 10 data shards, and the data is expanded by 20% after being persisted to the disk, that is, the actual storage is 1.2 times the original data. However, according to the existing technical solution, in the case of using 3 copies, the data is expanded by 200%, that is, the actual storage is 3 times of the original data.

Therefore, compared with the original technical solution of multi-copy storage, this technical solution can greatly reduce the proportion of redundant data on the premise of ensuring data reliability, thereby greatly saving storage space.

An embodiment of the present invention also provides a data storage device for implementing the above data storage method.

Fig. 5 shows a structural diagram of a data storage device according to an embodiment of the present invention, the data storage device is located in an access node of a distributed storage system, and the distributed storage system may be a Cassandra system or any other type Distributed storage system. Referring to FIG. 5, the data storage device may include a data segmentation unit 502, a verification calculation unit 504, and a storage allocation unit 506, wherein:

The data segmenting unit 502 is adapted to segment the data block written by the user into several data segments of a predetermined size. If the last fragment obtained by splitting may not reach the predetermined size, it can be filled with empty data. Among them, the fragment size can be specified as a parameter when the storage is created, and stored in the systemtable of the Cassandra system.

The check calculation unit 504 is adapted to use a redundancy check algorithm to calculate a number of check segments corresponding to the number of data segments obtained by segmentation. The redundancy check algorithm may use a parity check algorithm or a cyclic redundancy check algorithm or the like.

The storage allocation unit 506 is adapted to store the data shards and the verification shards in the data nodes of the distributed storage system, and only one copy of each data shard and each verification shard is stored.

In the embodiment of the present invention, only one copy of each data fragment and each verification fragment is stored in the distributed storage system, that is, each data fragment is stored in a data node corresponding to it, and each The checksum shard is also stored in a corresponding data node. In this way, when one or more data fragments in the data block fail, they can be recovered through the unfailed data fragments and check fragments, which ensures the reliability of the data; One copy reduces the proportion of redundant data and saves storage space.

As an implementation manner, the storage allocation unit 506 may select data nodes for data storage in the following manner:

First, calculate the digital signatures of each data fragment and each verification fragment respectively;

Then, each data segment and each verification segment are respectively stored in a data node of the distributed storage system according to their respective digital signatures.

In order to facilitate addressing of each data fragment when reading a data block (that is, locate the data node storing the data fragment), the data storage device in the embodiment of the present invention may also include an encoding unit (not shown in the figure) and metadata A storage unit (not shown in the figure), which encodes all digital signatures (for example, using JSON encoding) through the encoding unit to form metadata, and stores the metadata to multiple data nodes of the distributed storage system through the metadata storage unit middle. Among them, the storage of metadata may adopt a multi-copy scheme.

Since the sizes of the multiple data blocks written by the user are usually inconsistent, in order to realize the standardization of data encoding and storage, as an implementation, the verification calculation unit 504 further divides the several data fragments obtained by segmentation into A plurality of groups of data fragments, wherein each group of data fragments includes a first predetermined number of data fragments, and a second predetermined number of verification fragments corresponding to each group of data fragments are respectively calculated using a redundancy check algorithm.

Correspondingly, the encoding unit uses each group of data fragments and corresponding verification fragments as a stripe unit, and encodes all digital signatures corresponding to the stripe unit (for example, using JSON encoding) to form metadata; The metadata storage unit stores metadata corresponding to each group into multiple data nodes of the distributed storage system. Among them, the storage of metadata may adopt a multi-copy scheme.

In addition, an embodiment of the present invention also provides a distributed storage system including the above-mentioned data storage device. Specifically, the distributed storage system includes an access node and a data node, and the data storage device is located in the access node. Of course, when the distributed storage system is a Cassandra system, each node exists as both an access node and a data node.

To sum up, the embodiment of the present invention replaces the existing multi-copy storage method with an erasable code (verification method) in a distributed storage system, which can reduce the proportion of redundant data on the premise of ensuring data reliability. , so as to achieve the purpose of saving storage space.

A1, a data storage method, comprising:

The access node of the distributed storage system divides the data block written by the user into several data fragments of a predetermined size;

Using the redundancy check algorithm to calculate the check slices corresponding to the sliced data slices;

The data fragments and the verification fragments are stored in the data nodes of the distributed storage system, and only one copy of each data fragment and each verification fragment is stored.

A2. The data storage method according to claim A1, wherein the storing the data fragments and the verification fragments in the data nodes of the distributed storage system comprises:

Calculate the digital signatures of each data fragment and each verification fragment respectively;

Each data segment and each verification segment are respectively stored in a data node of the distributed storage system according to their respective digital signatures.

A3. The data storage method according to claim A2, further comprising:

Encode all digital signatures to form metadata;

The metadata is stored in multiple data nodes of the distributed storage system.

A4, the data storage method as claimed in claim A2, wherein, said employing the redundancy check algorithm to calculate the number of check pieces corresponding to the number of data slices obtained by segmentation, comprising:

Dividing the obtained data fragments into several groups of data fragments, wherein each group of data fragments includes a first predetermined number of data fragments;

A second predetermined number of check slices corresponding to each group of data slices are respectively calculated by using a redundancy check algorithm.

A5. The data storage method according to claim A4, further comprising:

Using each group of data fragments and corresponding verification fragments as a stripe unit, encoding all digital signatures corresponding to the stripe unit to form metadata;

The metadata is stored in multiple data nodes of the distributed storage system.

A6. The data storage method according to claim A3 or A5, wherein the encoding is JSON encoding.

A7. The data storage method according to claim A2, wherein the distributed storage system is a Cassandra system.

B8. A data storage device located in an access node of a distributed storage system, the data storage device comprising:

The data segmentation unit is suitable for dividing the data block written by the user into several data fragments of a predetermined size;

The verification calculation unit is adapted to use a redundancy verification algorithm to calculate a plurality of verification fragments corresponding to the plurality of data fragments obtained by segmentation;

The storage allocation unit is adapted to store the data shards and the check shards in the data nodes of the distributed storage system, and only one copy of each data shard and each check shard is stored.

B9. The data storage device of claim B8, wherein the storage allocation unit is further adapted to:

Calculate the digital signatures of each data fragment and each verification fragment respectively;

Each data segment and each verification segment are respectively stored in a data node of the distributed storage system according to their respective digital signatures.

B10. The data storage device as claimed in claim B9, further comprising:

an encoding unit adapted to encode all digital signatures to form metadata;

The metadata storage unit is adapted to store the metadata in multiple data nodes of the distributed storage system.

B11. The data storage device according to claim B9, wherein the verification calculation unit is further adapted to:

Dividing the obtained data fragments into several groups of data fragments, wherein each group of data fragments includes a first predetermined number of data fragments;

A second predetermined number of check slices corresponding to each group of data slices are respectively calculated by using a redundancy check algorithm.

B12. The data storage device according to claim B11, further comprising:

An encoding unit adapted to use each group of data fragments and corresponding verification fragments as a stripe unit, and encode all digital signatures corresponding to the stripe unit to form metadata;

The metadata storage unit is adapted to store the metadata in multiple data nodes of the distributed storage system.

B13. The data storage device according to claim B10 or B12, wherein the encoding is JSON encoding.

B14. The data storage method according to claim B9, wherein the distributed storage system is a Cassandra system.

B15. A distributed storage system comprising the data storage device according to any one of claims B8-B14.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other device. Various generic systems can also be used with the teachings based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not specific to any particular programming language. It should be understood that various programming languages can be used to implement the content of the present invention described herein, and the above description of specific languages is for disclosing the best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components in the data storage device according to the embodiments of the present invention. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

Claims (10)

1. a date storage method, comprising:

The data block that the access node of distributed memory system writes user is cut into the data fragmentation of some pre-sizings;

Adopt redundancy check algorithm to calculate some verification bursts corresponding to some data fragmentations that cutting obtains;

Described data fragmentation and described verification burst are stored in the back end of described distributed memory system, and each data fragmentation is only stored a copy with each verification burst.

2. date storage method as claimed in claim 1, wherein, describedly stores described data fragmentation and described verification burst in the back end of described distributed memory system into, comprising:

Calculate respectively the digital signature of each data fragmentation and each verification burst;

According to digital signature separately, each data fragmentation and each verification burst are stored into respectively in a back end of described distributed memory system.

3. date storage method as claimed in claim 2, wherein, also comprises:

All digital signature are encoded and formed metadata;

By described metadata store in a plurality of back end of described distributed memory system.

4. date storage method as claimed in claim 2, wherein, described employing redundancy check algorithm calculates some verification bursts corresponding to some data fragmentations that cutting obtains, and comprising:

If the individual data fragmentation that cutting is obtained is divided into some groups of data fragmentations, wherein, every group of data fragmentation comprises the first predetermined number data fragmentation;

Adopt redundancy check algorithm to calculate respectively and respectively organize the second predetermined number verification burst that data fragmentation is corresponding.

5. date storage method as claimed in claim 4, wherein, also comprises:

Using every group of data fragmentation and corresponding verification burst as a stripe cell, all digital signature corresponding to described stripe cell are encoded and formed metadata;

By described metadata store in a plurality of back end of described distributed memory system.

6. the date storage method as described in claim 3 or 5, wherein, described in be encoded to JSON coding.

7. date storage method as claimed in claim 2, wherein, described distributed memory system is Cassandra system.

8. a data storage device, is arranged in the access node of distributed memory system, and described data storage device comprises:

Data cutting unit, the data block that is suitable for user to write is cut into the data fragmentation of some pre-sizings;

Verification computing unit, is suitable for adopting redundancy check algorithm to calculate some verification bursts corresponding to some data fragmentations that cutting obtains;

Unit of memory allocation, is suitable for described data fragmentation and described verification burst to store in the back end of described distributed memory system, and each data fragmentation is only stored a copy with each verification burst.

9. data storage device as claimed in claim 8, wherein, described unit of memory allocation is further adapted for:

Calculate respectively the digital signature of each data fragmentation and each verification burst;

According to digital signature separately, each data fragmentation and each verification burst are stored into respectively in a back end of described distributed memory system.

10. a distributed memory system that comprises the data storage device as described in any one in claim 8～9.

Images