CN111399755A - Data storage management method and device - Google Patents
Data storage management method and device Download PDFInfo
- Publication number
- CN111399755A CN111399755A CN201910855037.5A CN201910855037A CN111399755A CN 111399755 A CN111399755 A CN 111399755A CN 201910855037 A CN201910855037 A CN 201910855037A CN 111399755 A CN111399755 A CN 111399755A
- Authority
- CN
- China
- Prior art keywords
- data
- security level
- segment
- preset
- check
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
- G06F11/1044—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices with specific ECC/EDC distribution
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Storage Device Security (AREA)
Abstract
The application discloses a data storage management method, and belongs to the technical field of data storage. The method comprises the following steps: receiving target data to be stored, dividing the target data into a plurality of data segments based on a preset security level, determining check data of each data segment, and storing each data segment and corresponding check data; after each data segment and corresponding check data are stored, when a preset adjustment period is reached, adjusting the current security level of the target data based on the lifting step length of the preset segmented data quantity to obtain an adjusted security level; and based on the adjusted security level, dividing the target data into a plurality of data segments, determining the check data of each divided data segment, and replacing each stored data segment and the corresponding check data by using each newly divided data segment and the corresponding check data. By adopting the method and the device, the technical problem that the storage equipment can only be continuously increased when the storage space is insufficient can be effectively solved.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a method and an apparatus for data storage management.
Background
In a conventional cloud storage system, data of a user is divided into a plurality of data segments, the data segments are stored in a plurality of nodes of a storage server, and in order to prevent data loss when a node fails, a Redundant Array of Independent Disks (RAID) is often used to perform cross-node protection on the data of the user.
At present, a common implementation manner of a network RAID is mainly based on an EC (Erasure Code) technology, the EC technology is a data protection method and is also a coding fault-tolerant technology, data is divided into a plurality of data segments, redundant coding is performed on the plurality of data segments, check data corresponding to each data segment is obtained, and each data segment and the check data corresponding to the data segment respectively include a plurality of data units with a preset data amount. Each data segment and the data unit contained in the corresponding check data are respectively stored in different nodes, and when part of the nodes fail, the data can still be recovered. According to the amount of the check data, the data often has different security levels, the security level of the data is often characterized by using a formula N + M, where N represents a segmented data amount, specifically, the number of data units included in a single data segment, M represents a check data amount, specifically, the number of data units included in the check data, M and N are integers greater than 0, and M is less than or equal to N. If the security level of the data is N + M, the data can still be recovered in case the corruption is no more than M nodes.
In the course of implementing the present application, the inventors found that the related art has at least the following problems:
in a conventional cloud storage project, after a user sets a security level N + M of data, a space utilization rate of user data is fixed, that is, N/(N + M), with continuous increase of traffic, when a storage space is insufficient, only storage devices can be continuously increased, so that an operation cost is also continuously increased with increase of traffic.
Disclosure of Invention
In order to solve technical problems in the related art, embodiments of the present application provide a method and an apparatus for improving a storage space utilization rate. The technical scheme of the method and the device for improving the utilization rate of the storage space is as follows:
in a first aspect, a method for improving storage space utilization is provided, where the method includes:
receiving target data to be stored, dividing the target data into a plurality of data segments based on a preset security level, determining check data of each data segment, and storing each data segment and corresponding check data, wherein the security level corresponds to a segmented data volume and a check data volume;
after each data segment and corresponding check data are stored, when a preset adjustment period is reached, adjusting the current security level of the target data based on the lifting step length of a preset segmented data volume to obtain an adjusted security level;
and based on the adjusted security level, dividing the target data into a plurality of data segments again, determining check data of each data segment which is divided again, and replacing each data segment which is stored and the corresponding check data by using each data segment which is divided again and the corresponding check data.
Optionally, the adjusting the current security level of the target data based on the preset lifting step size of the segmented data amount to obtain an adjusted security level includes:
on the basis of keeping the check data volume corresponding to the current security level unchanged, the segment data volume corresponding to the current security level is promoted by the promotion step length of the preset segment data volume, and the adjusted security level is obtained.
Optionally, after storing each data segment and corresponding check data, when a preset adjustment period is reached, adjusting the current security level of the target data based on a preset lifting step of the segmented data amount to obtain an adjusted security level, including:
and after each data segment and corresponding check data are stored, when a preset adjustment period is reached, if the current space utilization rate of the target data is smaller than a preset upper limit of the space utilization rate, adjusting the current security level of the target data based on a preset lifting step length of the segmented data quantity to obtain an adjusted security level.
Optionally, the dividing, based on a preset security level, the target data into a plurality of data segments, determining check data corresponding to each data segment, and storing each data segment and corresponding check data, where the security level corresponds to a segmented data volume and a check data volume, includes:
the target data is segmented into a plurality of data units with preset data volume, and the data units are divided into a plurality of groups based on the segmented data volume corresponding to the preset security level, wherein each group of data units is a data segment, and the data volume of each data segment is the same as the segmented data volume;
determining the check data of each data segment based on the check data volume corresponding to the preset security level, wherein the check data volume of each data segment is the same as the check data volume;
each data segment and corresponding check data is stored.
Optionally, the segmented data amount is the number of data units of a preset data amount included in the divided data segment, and the check data amount is the number of data units of the preset data amount included in the check data;
the repartitioning, based on the adjusted security level, the target data into a plurality of data segments, determining check data of each repartitioned data segment, and replacing each stored data segment and corresponding check data with each repartitioned data segment and corresponding check data includes:
if the sum of the numerical value of the segmented data volume corresponding to the adjusted security level and the numerical value of the check data volume is smaller than the number of storage devices in the cluster, the target data is divided into a plurality of data segments again based on the adjusted security level, check data of each divided data segment is determined, and each data segment and corresponding check data which are stored are replaced by each data segment and corresponding check data which are divided again.
In a second aspect, an apparatus for improving storage space utilization is provided, the apparatus comprising:
the storage module is used for receiving target data to be stored, dividing the target data into a plurality of data segments based on a preset security level, determining check data of each data segment, and storing each data segment and corresponding check data, wherein the security level corresponds to a segmented data volume and a check data volume;
the adjusting module is used for adjusting the current security level of the target data based on the lifting step length of the preset segmented data volume when a preset adjusting period is reached after each data segment and corresponding check data are stored, so as to obtain the adjusted security level;
and the replacing module is used for re-dividing the target data into a plurality of data segments based on the adjusted security level, determining the check data of each re-divided data segment, and replacing each stored data segment and the corresponding check data by using each re-divided data segment and the corresponding check data.
Optionally, the adjusting module is configured to:
on the basis of keeping the check data volume corresponding to the current security level unchanged, the segment data volume corresponding to the current security level is promoted by the promotion step length of the preset segment data volume, and the adjusted security level is obtained.
Optionally, the adjusting module is configured to:
and after each data segment and corresponding check data are stored, when a preset adjustment period is reached, if the current space utilization rate of the target data is smaller than a preset upper limit of the space utilization rate, adjusting the current security level of the target data based on a preset lifting step length of the segmented data quantity to obtain an adjusted security level.
Optionally, the storage module is configured to:
the target data is segmented into a plurality of data units with preset data volume, and the data units are divided into a plurality of groups based on the segmented data volume corresponding to the preset security level, wherein each group of data units is a data segment, and the data volume of each data segment is the same as the segmented data volume;
determining the check data of each data segment based on the check data volume corresponding to the preset security level, wherein the check data volume of each data segment is the same as the check data volume;
each data segment and corresponding check data is stored.
Optionally, the segmented data amount is the number of data units of a preset data amount included in the divided data segment, and the check data amount is the number of data units of the preset data amount included in the check data;
the replacement module is configured to:
if the sum of the numerical value of the segmented data volume corresponding to the adjusted security level and the numerical value of the check data volume is smaller than the number of storage devices in the cluster, the target data is divided into a plurality of data segments again based on the adjusted security level, check data of each divided data segment is determined, and each data segment and corresponding check data which are stored are replaced by each data segment and corresponding check data which are divided again.
In a third aspect, there is provided a server comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for data storage management as described in the first aspect above.
In a fourth aspect, there is provided a computer readable storage medium having stored therein at least one instruction which is loaded and executed by the processor to implement the method of data storage management as described in the first aspect above.
The beneficial effects brought by the technical scheme provided by the embodiment of the application at least comprise:
the method for managing the data storage provided by the embodiment of the application can be applied to management of the data storage. After target data to be stored are received, the target data are divided into a plurality of data segments according to a preset security level, check data of each data segment are determined, and each data segment and corresponding check data are stored. And then, every time a preset adjustment period is reached, obtaining the adjusted security level based on the preset lifting step length of the segmented data volume. And finally, based on the adjusted security level, the target data is divided into a plurality of data segments again, check data of each divided data segment is determined, and each data segment and corresponding check data which are stored are replaced by each data segment and corresponding check data which are divided again. Therefore, the space utilization rate of the target data is not fixed, and the space utilization rate of the target data can be automatically improved when a preset adjustment period is reached, so that when the storage space is insufficient, the problem can be solved by improving the space utilization rate without increasing storage equipment, and the operation cost is reduced.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flow chart of a method for data storage management according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of an apparatus for data storage management according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a server provided in an embodiment of the present application;
fig. 4 is a schematic diagram of periodically adjusting a security level according to an embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
The embodiment of the application provides a data storage management method, which can be realized by a server.
The execution subject of each step in the embodiment of the present application may be a server in a cloud storage system. The cloud storage system is a system which integrates a large number of storage devices of different types in a network through application software to cooperatively work through functions of cluster application, a grid technology or a distributed file system and the like, and provides data storage and service access functions to the outside. The cloud storage system includes: the storage server may comprise a plurality of storage units, that is, nodes, and the nodes may be disks, data blocks or other forms of storage media. The nodes may store data, and the management server may manage the data stored in the nodes of the storage server.
As shown in fig. 1, the processing flow of the method may include the following steps:
in step 101, target data to be stored is received, the target data is divided into a plurality of data segments based on a preset security level, check data of each data segment is determined, and each data segment and corresponding check data are stored.
The security level corresponds to the segmented data volume and the check data volume. The data volume of the segments is the number of data units contained in each data segment, and the check data volume is the number of data units contained in the check data corresponding to each data segment. The security level may be expressed in the form of N + M, N representing the value of the amount of fragmented data, M representing the value of the amount of parity data, N and M each being an integer greater than 0. For example, the security level N + M corresponds to a segment data amount of N and a check data amount of M. Based on the preset security level N + M, each divided data segment includes N data units, and the check data corresponding to each data segment includes M data units.
The target data to be stored may be data uploaded by the user. The preset security level is a security level preset in the cloud storage system, and different security levels can be set according to different importance of the target data.
In implementation, firstly, a user uploads target data to be stored to a server in a cloud storage system, then, the server receives the target data to be stored, and codes the target data by adopting an erasure coding algorithm based on a preset security level, and then, stores the coded target data.
The specific process can be as follows:
after a server in a cloud storage system receives target data to be stored, firstly, the target data is divided into a plurality of data segments based on a preset security level, wherein the data volume of each data segment is the same as the data volume of a segment corresponding to the preset security level. And then, determining the check data of each data segment, wherein the data volume of the check data of each data segment is the same as the check data volume corresponding to the preset security level. Finally, the verification data is appended to the end of the corresponding data segment, so that each data segment contains both the content of the target data and the content of the verification data.
And then, averagely distributing the data in the data segment added with the check data to different storage devices in the cluster for storage, wherein the number of the storage devices for storing the data segment is equal to the sum of the segment data quantity and the check data quantity corresponding to the preset security level.
For example, if the preset security level is N + M, after a server in the cloud storage system receives target data to be stored, the target data is first divided into a plurality of data segments based on the preset security level N + M, where the data volume of each data segment is N. Then, the check data of each data segment is determined, wherein the data quantity of the check data of each data segment is M. And finally, adding the verification data to the tail of the corresponding data segment, so that each data segment contains the content of the target data and the content of the verification data, and the data volume of each data segment is N + M.
And then, averagely distributing the data in the data segment added with the check data to N + M storage devices in the cluster for storage.
Optionally, the target data may be divided into a plurality of data units with a preset data amount, then the plurality of data units are divided into a plurality of data segments, the check data corresponding to each data segment is calculated, and finally, each data segment and the corresponding check data are stored. The corresponding processing procedure of step 101 may be as follows: the method comprises the steps of segmenting target data into a plurality of data units, and dividing the plurality of data units into a plurality of groups based on a segmented data volume corresponding to a preset security level, wherein each group of data units is a data segment, and the data volume of each data segment is the same as the segmented data volume. And determining the check data of each data segment based on the check data volume corresponding to the preset security level, wherein the data volume of the check data of each data segment is the same as the check data volume. Each data segment and corresponding check data is stored.
The preset data amount may be 1 million, that is, each data unit may contain 1 million data.
In implementation, firstly, a user uploads target data to be stored to a server in a cloud storage system, then, the server receives the target data to be stored, and based on a preset security level, encodes the target data by adopting an erasure coding algorithm, and then, stores the encoded target data.
The specific process can be as follows:
the preset security level is N + M, after a server in the cloud storage system receives target data to be stored, the target data is firstly divided into a plurality of data units, and then the plurality of data units are divided into a plurality of groups based on a preset segmented data amount N, wherein each group of data units is a data segment, the data amount of each data segment is N, that is, each data segment contains N data units. Then, based on a preset check data amount M, the check data of each data segment is determined, where the data amount of the check data corresponding to each data segment is M, that is, the check data corresponding to each data segment includes M data units. And finally, adding the check data to the tail of the corresponding data segment, wherein each data segment contains the content of the target data and the content of the check data, and the data volume of each data segment is N + M, namely each data segment contains N + M data units.
And then, averagely distributing the data in the data segment added with the check data to N + M storage devices in the cluster for storage.
For example, the preset security level is 4+2, and each data unit includes 1MB of data. After a server in the cloud storage system receives 100MB to be stored, firstly, target data is divided into 100 data units, each data unit contains 1MB of data, and then, based on a preset segmented data amount of 4, 100 data units are divided into 100/4 groups, wherein each group of data units is one data segment, that is, 25 data segments in total, and each data segment contains 4 data units. Then, based on a preset check data amount 2, check data of each data segment is determined, wherein the check data corresponding to each data segment comprises 2 data units. Finally, the check data is appended to the end of the corresponding data segment, so that each data segment contains both the content of the target data and the content of the check data, and each data segment contains 4+2 data units and 6MB data.
And then, averagely distributing the data in the data segment added with the check data to 6 storage devices in the cluster for storage, wherein each storage device stores 1 data unit.
Then, after each data segment and the corresponding check data are stored, the process of step 102 is performed.
In step 102, after each data segment and corresponding check data are stored, each time a preset adjustment period is reached, the current security level of the target data is adjusted based on a preset lifting step of the segmented data amount, so as to obtain an adjusted security level.
The preset adjustment period may be set according to actual conditions, for example, the preset adjustment period is two days, that is, the current security level of the target data is adjusted once every two days.
The preset lifting step size of the segmented data volume is an increased value of the segmented data volume each time, and the preset lifting step size of the segmented data volume can be set arbitrarily according to actual conditions.
In implementation, the values of the segment data amount and the check data amount determine the space utilization rate of the target data, so that the space utilization rate of the target data can be adjusted by adjusting the values of the segment data amount and the check data amount, that is, the space utilization rate of the target data can be improved by adjusting the security level of the target data.
The space utilization rate corresponding to the security level N + M is N/(N + M). As can be seen from the expression of the space utilization, the increase of the space utilization can be achieved by adjusting the values of N and M in the security level. That is, the improvement of the space utilization rate can be realized by adjusting the values of the segment data volume and the check data volume corresponding to the security level.
In the specific implementation of realizing the improvement of the space utilization rate based on the adjustment of the security level, the improvement of the space utilization rate can be realized by improving the amount of the segmented data on the basis of keeping the amount of the check data corresponding to the security level unchanged; the space utilization rate can be improved by reducing the data volume of the check on the basis of keeping the data volume of the sections corresponding to the security level unchanged; the space utilization rate can be improved by reducing the check data volume and improving the segmented data volume; the improvement of the space utilization rate can be realized by simultaneously reducing the check data volume and the segmentation data volume, but the reduced numerical values of the check data volume and the segmentation data volume improve the space utilization rate; the space utilization rate can be improved by a mode of simultaneously improving the data volume of the sections and the data volume of the check, but the improved numerical values of the sections and the data volume ensure that the space utilization rate is improved.
That is, for the security level N + M, in order to improve the spatial rate N/(N + M), the spatial utilization rate can be improved by improving N while keeping M unchanged; the space utilization rate can be improved by reducing M on the basis of keeping N unchanged; the space utilization rate can be improved by increasing N and reducing M; the improvement of the space utilization rate can also be realized by simultaneously reducing N and M, but the reduction amplitude of N and M increases N/(N + M); the improvement of the space utilization rate can also be realized by simultaneously improving N and M, but the increasing amplitude of N and M should be made N/(N + M) larger.
Optionally, on the basis that the check data amount corresponding to the security level is not changed, the segment data amount corresponding to the security level may be promoted, and the corresponding processing procedure in step 102 may be as follows: on the basis of keeping the check data volume corresponding to the current security level unchanged, the segment data volume corresponding to the current security level is promoted by the promotion step length of the preset segment data volume, and the adjusted security level is obtained.
The preset lifting step length of the segment data volume is the added value of the segment data volume each time.
In implementation, the current security level is set to be N + M, the preset lifting step size of the segment data size is i, and the adjusted security level is (N + i) + M. Namely, each time the preset adjustment period is reached, the segment data amount (namely N) corresponding to the current security level is increased by the preset segment data amount increasing step length (namely i). As shown in fig. 4, the initial security level is 4+2, the lifting step of the preset segmented data amount is 1, and each time the preset adjustment period is reached, the segmented data amount corresponding to the security level is subjected to the processing of adding 1.
Optionally, when the space utilization of the target data is increased to a certain upper limit, the increase of the space utilization is stopped, that is, the adjustment of the security level of the target data is stopped. The corresponding process of step 102 may be as follows: after each data segment and corresponding check data are stored, when a preset adjustment period is reached, if the current space utilization rate of the target data is smaller than a preset upper limit of the space utilization rate, the current security level of the target data is adjusted based on a preset lifting step length of the segmented data volume, and the adjusted security level is obtained.
The preset space utilization rate may be set according to an actual situation, for example, may be set according to the number of storage devices in the cluster.
In implementation, as can be seen from the expression N/(N + M) of the space utilization, when the space utilization rises to a large value, even if the security level of the target data is adjusted, the value (i.e., N) of the amount of the segmented data corresponding to the security level is raised, and the raised value of the space utilization is small. Moreover, the security is reduced to a certain extent due to the excessively high space utilization rate, and therefore, when the space utilization rate is increased to a certain upper limit value, the increase of the space utilization rate, that is, the adjustment of the security level of the target data, should be stopped.
Furthermore, the space utilization rate cannot be infinitely increased, the higher the space utilization rate is, the more the number of storage devices is required, and the number of storage devices in the cluster is limited, so that when the space utilization rate is increased to a certain upper limit value, the adjustment of the target data security level should be stopped. The greater the number of storage devices in a cluster, the higher the space utilization that can be supported. Therefore, when the number of storage devices in the cluster is large, the preset upper limit of the space utilization rate can be set to be high; when the number of storage devices in the cluster is small, the preset upper limit of space utilization should be set low.
After the adjusted security level is obtained, the process of step 103 is performed.
In step 103, based on the adjusted security level, the target data is subdivided into a plurality of data segments, and the check data of each data segment subdivided is determined, and each data segment and the corresponding check data stored in the memory are replaced by each data segment and the corresponding check data subdivided.
In implementation, after the adjusted security level is obtained, first, target data is obtained from different storage devices in the cluster, then, based on the adjusted security level, the erasure correction coding algorithm is adopted again for coding the target data, and then, the target data subjected to the re-coding processing is used for replacing the target data stored before.
Based on the adjusted security level, the specific process of encoding the target data by using the erasure coding algorithm again may refer to the specific process in step 101, and is not described herein again.
If the previous security level is N + M and the preset step size for increasing the amount of the segmented data is i, the adjusted security level is (N + i) + M, and the target data encoded based on the adjusted security level is replaced with the target data encoded based on the previous security level, the increase value of the storage space utilization rate is the difference between (N + i)/(N + i + M) and N/(N + M).
For example, if the previous security level is 4+2 and the adjusted security level is 6+2, the space utilization improvement value is the difference between 6/8 and 4/6.
Optionally, when the number of storage devices required by the adjusted security level is greater than the number of storage devices in the cluster, the increase of the space utilization rate is stopped, and the corresponding processing procedure in step 103 may be as follows: and if the sum of the numerical value of the segmented data volume corresponding to the adjusted security level and the numerical value of the check data volume is smaller than the number of the storage devices in the cluster, re-dividing the target data into a plurality of data segments based on the adjusted security level, determining the check data of each re-divided data segment, and replacing each stored data segment and the corresponding check data by using each re-divided data segment and the corresponding check data.
The data volume of the segments is the number of data units of the preset data volume included in the divided data segments, and the check data volume is the number of data units of the preset data volume included in the check data.
In implementation, after the target data is encoded by the erasure coding algorithm based on the security level N + M, at least N + M storage devices are required to store the encoded target data. Therefore, when the sum of the numerical values of the segmented data amount corresponding to the adjusted security level and the check data amount is smaller than the number of the storage devices in the cluster, the number of the storage devices in the cluster is not enough to achieve the adjusted security level, and therefore the target data is not encoded based on the adjusted security level any more, and is still stored in the original security level.
When the number of the storage devices in the cluster is increased to exceed the sum of the numerical values of the segmented data amount and the check data amount, the target data is continuously divided into a plurality of data segments again based on the adjusted security level, the check data of each divided data segment is determined, and each stored data segment and the corresponding check data are replaced by each divided data segment and the corresponding check data.
The method for managing the data storage provided by the embodiment of the application can be applied to management of the data storage. After target data to be stored are received, the target data are divided into a plurality of data segments according to a preset security level, check data of each data segment are determined, and each data segment and corresponding check data are stored. And then, every time a preset adjustment period is reached, obtaining the adjusted security level based on the preset lifting step length of the segmented data volume. And finally, based on the adjusted security level, the target data is divided into a plurality of data segments again, check data of each divided data segment is determined, and each data segment and corresponding check data which are stored are replaced by each data segment and corresponding check data which are divided again. Therefore, the space utilization rate of the target data is not determined, and when a preset adjustment period is reached, as shown in fig. 4, the security level of the target data is automatically adjusted, so that the space utilization rate of the target data is periodically improved, and when the storage space is insufficient, the problem that the space utilization rate is improved by adjusting the security level can be solved without increasing storage equipment, so that the cost is reduced.
Based on the same technical concept, an embodiment of the present application further provides an apparatus for data storage management, where the apparatus may be a server in the foregoing embodiment, and as shown in fig. 2, the apparatus includes:
the storage module 201 is configured to receive target data to be stored, divide the target data into a plurality of data segments based on a preset security level, determine check data of each data segment, and store each data segment and corresponding check data, where the security level corresponds to a segmented data amount and a check data amount;
the adjusting module 202 is configured to, after each data segment and corresponding check data are stored, adjust a current security level of the target data based on a preset lifting step size of the segmented data volume whenever a preset adjusting period is reached, so as to obtain an adjusted security level;
and a replacing module 203, configured to re-divide the target data into a plurality of data segments based on the adjusted security level, determine check data of each re-divided data segment, and replace each stored data segment and corresponding check data with each re-divided data segment and corresponding check data.
Optionally, the adjusting module 202 is configured to:
on the basis of keeping the check data volume corresponding to the current security level unchanged, the segment data volume corresponding to the current security level is promoted by the promotion step length of the preset segment data volume, and the adjusted security level is obtained.
Optionally, the adjusting module 202 is configured to:
after each data segment and corresponding check data are stored, when a preset adjustment period is reached, if the current space utilization rate of the target data is smaller than a preset upper limit of the space utilization rate, the current security level of the target data is adjusted based on a preset lifting step length of the segmented data volume, and the adjusted security level is obtained.
Optionally, the storage module 201 is configured to:
dividing target data into a plurality of data units with preset data quantity, and dividing the plurality of data units into a plurality of groups based on the segmented data quantity corresponding to the preset security level, wherein each group of data units is a data segment, and the data quantity of each data segment is the same as the segmented data quantity;
determining the check data of each data segment based on the check data volume corresponding to the preset security level, wherein the data volume of the check data of each data segment is the same as the check data volume;
each data segment and corresponding check data is stored.
Optionally, the segmented data amount is the number of data units of a preset data amount included in the divided data segment, and the check data amount is the number of data units of the preset data amount included in the check data;
a replacement module 203 for:
and if the sum of the numerical value of the segmented data volume corresponding to the adjusted security level and the numerical value of the check data volume is smaller than the number of the storage devices in the cluster, re-dividing the target data into a plurality of data segments based on the adjusted security level, determining the check data of each re-divided data segment, and replacing each stored data segment and the corresponding check data by using each re-divided data segment and the corresponding check data.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
It should be noted that: in the data storage management apparatus provided in the foregoing embodiment, only the division of the functional modules is illustrated when performing data storage management, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the data storage management apparatus provided in the foregoing embodiment and the data storage management method embodiment belong to the same concept, and specific implementation processes thereof are described in the method embodiment and are not described herein again.
Fig. 3 is a schematic structural diagram of a server according to an embodiment of the present application, where the server 300 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 301 and one or more memories 302, where the memory 302 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 301 to implement the above-mentioned method for data storage management.
In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the method for data storage management in the above-described embodiments. For example, the computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (12)
1. A method of data storage management, the method comprising:
receiving target data to be stored, dividing the target data into a plurality of data segments based on a preset security level, determining check data of each data segment, and storing each data segment and corresponding check data, wherein the security level corresponds to a segmented data volume and a check data volume;
after each data segment and corresponding check data are stored, when a preset adjustment period is reached, adjusting the current security level of the target data based on the lifting step length of a preset segmented data volume to obtain an adjusted security level;
and based on the adjusted security level, dividing the target data into a plurality of data segments again, determining check data of each data segment which is divided again, and replacing each data segment which is stored and the corresponding check data by using each data segment which is divided again and the corresponding check data.
2. The method according to claim 1, wherein the adjusting the current security level of the target data based on the lifting step of the preset segmented data amount to obtain an adjusted security level comprises:
on the basis of keeping the check data volume corresponding to the current security level unchanged, the segment data volume corresponding to the current security level is promoted by the promotion step length of the preset segment data volume, and the adjusted security level is obtained.
3. The method according to claim 1, wherein the adjusting the current security level of the target data to obtain the adjusted security level based on a preset step size of a segment data amount each time a preset adjustment period is reached after storing each data segment and corresponding check data comprises:
and after each data segment and corresponding check data are stored, when a preset adjustment period is reached, if the current space utilization rate of the target data is smaller than a preset upper limit of the space utilization rate, adjusting the current security level of the target data based on a preset lifting step length of the segmented data quantity to obtain an adjusted security level.
4. The method according to any one of claims 1 to 3, wherein the dividing the target data into a plurality of data segments based on a preset security level, determining check data corresponding to each data segment, and storing each data segment and corresponding check data, wherein the security level corresponds to a segmented data amount and a check data amount includes:
the target data is segmented into a plurality of data units with preset data volume, and the data units are divided into a plurality of groups based on the segmented data volume corresponding to the preset security level, wherein each group of data units is a data segment, and the data volume of each data segment is the same as the segmented data volume;
determining the check data of each data segment based on the check data volume corresponding to the preset security level, wherein the check data volume of each data segment is the same as the check data volume;
each data segment and corresponding check data is stored.
5. The method according to any one of claims 1 to 3, wherein the segmented data amount is the number of data units of a preset data amount included in the divided data segment, and the check data amount is the number of data units of the preset data amount included in the check data;
the repartitioning, based on the adjusted security level, the target data into a plurality of data segments, determining check data of each repartitioned data segment, and replacing each stored data segment and corresponding check data with each repartitioned data segment and corresponding check data includes:
if the sum of the numerical value of the segmented data volume corresponding to the adjusted security level and the numerical value of the check data volume is smaller than the number of storage devices in the cluster, the target data is divided into a plurality of data segments again based on the adjusted security level, check data of each divided data segment is determined, and each data segment and corresponding check data which are stored are replaced by each data segment and corresponding check data which are divided again.
6. An apparatus for data storage management, the apparatus comprising:
the storage module is used for receiving target data to be stored, dividing the target data into a plurality of data segments based on a preset security level, determining check data of each data segment, and storing each data segment and corresponding check data, wherein the security level corresponds to a segmented data volume and a check data volume;
the adjusting module is used for adjusting the current security level of the target data based on the lifting step length of the preset segmented data volume when a preset adjusting period is reached after each data segment and corresponding check data are stored, so as to obtain the adjusted security level;
and the replacing module is used for re-dividing the target data into a plurality of data segments based on the adjusted security level, determining the check data of each re-divided data segment, and replacing each stored data segment and the corresponding check data by using each re-divided data segment and the corresponding check data.
7. The apparatus of claim 6, wherein the adjustment module is configured to:
on the basis of keeping the check data volume corresponding to the current security level unchanged, the segment data volume corresponding to the current security level is promoted by the promotion step length of the preset segment data volume, and the adjusted security level is obtained.
8. The apparatus of claim 6, wherein the adjustment module is configured to:
and after each data segment and corresponding check data are stored, when a preset adjustment period is reached and the current space utilization rate of the target data is smaller than a preset upper limit of the space utilization rate, adjusting the current security level of the target data based on a preset lifting step length of the segmented data quantity to obtain an adjusted security level.
9. The apparatus of any one of claims 6-8, wherein the storage module is configured to:
the target data is segmented into a plurality of data units with preset data volume, and the data units are divided into a plurality of groups based on the segmented data volume corresponding to the preset security level, wherein each group of data units is a data segment, and the data volume of each data segment is the same as the segmented data volume;
determining the check data of each data segment based on the check data volume corresponding to the preset security level, wherein the check data volume of each data segment is the same as the check data volume;
each data segment and corresponding check data is stored.
10. The apparatus according to any one of claims 6-8, wherein the segmented data amount is the number of data units of a preset data amount included in the divided data segment, and the check data amount is the number of data units of the preset data amount included in the check data;
the replacement module is configured to:
if the sum of the numerical value of the segmented data volume corresponding to the adjusted security level and the numerical value of the check data volume is smaller than the number of storage devices in the cluster, the target data is divided into a plurality of data segments again based on the adjusted security level, check data of each divided data segment is determined, and each data segment and corresponding check data which are stored are replaced by each data segment and corresponding check data which are divided again.
11. A server, comprising a processor and a memory, the memory having stored therein at least one instruction, the at least one instruction being loaded and executed by the processor to implement a method of data storage management according to any one of claims 1 to 5.
12. A computer-readable storage medium having stored thereon at least one instruction, which is loaded and executed by a processor, to implement the method for improving storage space utilization according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910855037.5A CN111399755A (en) | 2019-09-10 | 2019-09-10 | Data storage management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910855037.5A CN111399755A (en) | 2019-09-10 | 2019-09-10 | Data storage management method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111399755A true CN111399755A (en) | 2020-07-10 |
Family
ID=71435873
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910855037.5A Pending CN111399755A (en) | 2019-09-10 | 2019-09-10 | Data storage management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111399755A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113132409A (en) * | 2021-04-29 | 2021-07-16 | 工银科技有限公司 | Data transmission method and device |
| CN114339723A (en) * | 2020-10-10 | 2022-04-12 | 深圳长城开发科技股份有限公司 | Air upgrading method and system based on LoRa |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS63200238A (en) * | 1987-02-17 | 1988-08-18 | Fujitsu Ltd | Error recovering device |
| CN103118133A (en) * | 2013-02-28 | 2013-05-22 | 浙江大学 | Mixed cloud storage method based on file access frequency |
| CN103631666A (en) * | 2012-08-24 | 2014-03-12 | 中兴通讯股份有限公司 | Data redundancy fault-tolerance adaptation management device, service device, system and method |
| US20140281804A1 (en) * | 2009-10-30 | 2014-09-18 | Cleversafe, Inc. | Distributed storage network for modification of a data object |
| CN104541251A (en) * | 2012-06-13 | 2015-04-22 | 卡林戈公司 | Erasure coding and replication in storage clusters |
| CN108717383A (en) * | 2018-04-03 | 2018-10-30 | 华为技术有限公司 | A kind of method and apparatus of update check block |
| CN109150792A (en) * | 2017-06-15 | 2019-01-04 | 杭州海康威视数字技术股份有限公司 | A kind of raising data storage safety method and device |
| CN109144766A (en) * | 2017-06-28 | 2019-01-04 | 杭州海康威视数字技术股份有限公司 | A kind of storage of data, reconstructing method and device and electronic equipment |
-
2019
- 2019-09-10 CN CN201910855037.5A patent/CN111399755A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS63200238A (en) * | 1987-02-17 | 1988-08-18 | Fujitsu Ltd | Error recovering device |
| US20140281804A1 (en) * | 2009-10-30 | 2014-09-18 | Cleversafe, Inc. | Distributed storage network for modification of a data object |
| CN104541251A (en) * | 2012-06-13 | 2015-04-22 | 卡林戈公司 | Erasure coding and replication in storage clusters |
| CN103631666A (en) * | 2012-08-24 | 2014-03-12 | 中兴通讯股份有限公司 | Data redundancy fault-tolerance adaptation management device, service device, system and method |
| CN103118133A (en) * | 2013-02-28 | 2013-05-22 | 浙江大学 | Mixed cloud storage method based on file access frequency |
| CN109150792A (en) * | 2017-06-15 | 2019-01-04 | 杭州海康威视数字技术股份有限公司 | A kind of raising data storage safety method and device |
| CN109144766A (en) * | 2017-06-28 | 2019-01-04 | 杭州海康威视数字技术股份有限公司 | A kind of storage of data, reconstructing method and device and electronic equipment |
| CN108717383A (en) * | 2018-04-03 | 2018-10-30 | 华为技术有限公司 | A kind of method and apparatus of update check block |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114339723A (en) * | 2020-10-10 | 2022-04-12 | 深圳长城开发科技股份有限公司 | Air upgrading method and system based on LoRa |
| CN114339723B (en) * | 2020-10-10 | 2024-01-26 | 深圳长城开发科技股份有限公司 | Air upgrading method and system based on LoRa |
| CN113132409A (en) * | 2021-04-29 | 2021-07-16 | 工银科技有限公司 | Data transmission method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230236741A1 (en) | Managing Correlated Outages in a Dispersed Storage Network | |
| CN107943421B (en) | Partition division method and device based on distributed storage system | |
| CN110175212B (en) | Block chain distributed storage method, data reading method, device and storage medium | |
| CN102142006B (en) | File processing method and device of distributed file system | |
| US9959169B2 (en) | Expansion of dispersed storage network (DSN) memory | |
| US10031669B2 (en) | Scheduling migration related traffic to be non-disruptive and performant | |
| US10191808B2 (en) | Systems and methods for storing, maintaining, and accessing objects in storage system clusters | |
| US10929065B2 (en) | Method, device and computer program product for coordinating access operations | |
| CN113687975B (en) | Data processing method, device, equipment and storage medium | |
| CN115268773B (en) | Erasure code data storage method, device, equipment and medium | |
| CN113655969B (en) | Data balanced storage method based on streaming distributed storage system | |
| US10360103B2 (en) | Focused storage pool expansion to prevent a performance degradation | |
| CN110737391A (en) | Method, apparatus and computer program product for managing a storage system | |
| US12111807B2 (en) | Optimizing storage and retrieval of compressed data | |
| EP4170499A1 (en) | Data storage method, storage system, storage device, and storage medium | |
| CN110874284B (en) | Data processing method and device | |
| CN111399755A (en) | Data storage management method and device | |
| CN107797884A (en) | A kind of cloud storage fault-tolerance approach adaptively switched based on duplication and correcting and eleting codes | |
| US10528282B2 (en) | Modifying and utilizing a file structure in a dispersed storage network | |
| CN115599315B (en) | Data processing method, device, system, equipment and medium | |
| US10554752B2 (en) | Efficient transfer of encoded data slice sets to new or alternate storage units | |
| US20210117088A1 (en) | Method, device, and computer program product for storage management | |
| CN116010161A (en) | Method, apparatus and program product for managing blocks in a storage system | |
| CN117234433B (en) | Data migration method and device, RAID storage device, storage medium and electronic device | |
| CN119883127B (en) | Distributed storage method, device and electronic device based on data modification frequency |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200710 |
|
| RJ01 | Rejection of invention patent application after publication |