CN114756172A - Service data storage method and device - Google Patents
Service data storage method and device Download PDFInfo
- Publication number
- CN114756172A CN114756172A CN202210393314.7A CN202210393314A CN114756172A CN 114756172 A CN114756172 A CN 114756172A CN 202210393314 A CN202210393314 A CN 202210393314A CN 114756172 A CN114756172 A CN 114756172A
- Authority
- CN
- China
- Prior art keywords
- storage
- service data
- data
- stored
- slave
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种业务数据存储方法和装置，涉及计算机技术领域。该方法的一具体实施方式包括：接收待存储的业务数据，在分布式存储集群的主节点中存储业务数据；确定分布式存储集群中多个从节点的存储效率；利用主节点向多个从节点发送业务数据，以在多个从节点中分别存储业务数据副本；其中，发送至任一从节点的待存储业务数据数量不超过第一数量阈值，且存储效率最高的从节点中待存储业务数据数量与存储效率最低的从节点中待存储业务数据数量之差不超过第二数量阈值。该实施方式保障了数据可靠性，提高了存储效果，提升了用户体验。
The invention discloses a business data storage method and device, and relates to the technical field of computers. A specific implementation of the method includes: receiving business data to be stored, and storing the business data in the master node of the distributed storage cluster; determining the storage efficiency of multiple slave nodes in the distributed storage cluster; The node sends service data to store copies of the service data in multiple slave nodes respectively; wherein, the quantity of service data to be stored sent to any slave node does not exceed the first quantity threshold, and the service to be stored in the slave node with the highest storage efficiency The difference between the quantity of data and the quantity of service data to be stored in the slave node with the lowest storage efficiency does not exceed the second quantity threshold. This implementation method ensures data reliability, improves storage effect, and improves user experience.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for storing service data.
Background
In order to prevent loss of service data, in addition to storing service data in a master node of the distributed storage cluster, a copy of the service data is also stored in each slave node.
Taking a Raft (a distributed consistency protocol) cluster as an example, if the Raft cluster includes a master node (leader) and two slave nodes (FollowerA, FollowerB), service data written by a user is sent to the leader of the distributed storage cluster for storage, and then the leader distributes the service data to the FollowerA, FollowerB, so as to store a copy of the service data in the slave nodes, thereby ensuring distributed consistency.
The prior art has at least the following problems:
in the existing service data storage method, when a master node sends service data to a slave node, the performance and the load capacity of different nodes are not considered, so that the data reliability is low, the storage effect is poor, and the user experience is poor.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for storing service data, which can combine storage efficiencies of different slave nodes to store service data copies in multiple slave nodes, thereby ensuring data reliability, improving storage effect, and improving user experience.
In order to achieve the above object, according to a first aspect of the embodiments of the present invention, there is provided a service data storage method, including:
receiving service data to be stored, and storing the service data in a main node of a distributed storage cluster;
Determining storage efficiency of a plurality of slave nodes in a distributed storage cluster;
transmitting, by the master node, the traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the number of the business data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the business data to be stored in the slave node with the highest storage efficiency and the number of the business data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold.
Further, the step of transmitting the traffic data to the plurality of slave nodes by using the master node to store the traffic data in the plurality of slave nodes, respectively, includes:
numbering the service data stored in the main node;
acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first quantity threshold; the slave node storage schedule indicates the current quantity of the service data to be stored of a plurality of slave nodes;
calculating the quantity difference between the quantity of the current service data to be stored in the candidate slave nodes and the quantity of the current service data to be stored in the slave nodes with the lowest storage efficiency, judging whether the quantity difference does not exceed a second quantity threshold value, if so, determining the candidate slave nodes as target slave nodes, and sending the target service data to the target slave nodes by using the master nodes; and the sum of the quantity of the target service data and the quantity of the service data to be stored currently in the target slave node does not exceed a first quantity threshold value.
Further, the step of transmitting the traffic data to the plurality of slave nodes by using the master node to store the traffic data in the plurality of slave nodes, respectively, further includes:
determining the data volume corresponding to the numbered service data;
acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first data volume threshold; the slave node storage schedule indicates the data volume corresponding to the current business data to be stored of a plurality of slave nodes;
calculating a data volume difference value between the data volume of the current service data to be stored in the candidate slave nodes and the data volume of the current service data to be stored in the slave nodes with the lowest storage efficiency, judging whether the data volume difference value does not exceed a second data volume threshold value, if so, determining the candidate slave nodes as target slave nodes, and sending the target service data to the target slave nodes by using the master node, wherein the sum of the data volume of the target service data and the data volume of the current service data to be stored in the target slave nodes does not exceed the first data volume threshold value.
Further, before the step of obtaining the slave node storage schedule, the method further comprises:
acquiring the storage states of service data to be stored in a plurality of slave nodes in real time;
And constructing a slave node storage schedule according to the storage state.
Further, the method also comprises the following steps:
setting a storage efficiency threshold, and determining the time length for which the storage efficiency of the slave node is less than the storage efficiency threshold;
and if the duration exceeds a duration threshold, adjusting the slave nodes of the distributed storage cluster.
Further, the step of determining the storage efficiency of a plurality of slave nodes in the distributed storage cluster comprises:
and determining the storage efficiency of the plurality of slave nodes according to the heartbeats between the master node and the plurality of slave nodes respectively.
Further, still include:
and updating the storage efficiency according to the slave node storage schedule.
According to a second aspect of the embodiments of the present invention, there is provided a service data storage apparatus, including:
a receiving module to: receiving service data to be stored, and storing the service data in a main node of a distributed storage cluster;
a storage efficiency determination module to: determining storage efficiency of a plurality of slave nodes in a distributed storage cluster;
a service data storage module, configured to: transmitting, by the master node, traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the number of the service data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the service data to be stored in the slave node with the highest storage efficiency and the number of the service data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold.
According to a third aspect of the embodiments of the present invention, there is provided an electronic device for storing service data, including:
one or more processors;
a storage device to store one or more programs,
when executed by one or more processors, cause the one or more processors to implement any of the business data storage methods described above.
According to a fourth aspect of the embodiments of the present invention, there is provided a computer-readable medium, on which a computer program is stored, the program, when executed by a processor, implementing any one of the above-mentioned service data storage methods.
One embodiment of the above invention has the following advantages or benefits: the method comprises the steps that business data to be stored are received and stored in a main node of a distributed storage cluster; determining storage efficiency of a plurality of slave nodes in a distributed storage cluster; transmitting, by the master node, the traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the method comprises the steps that the number of the service data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the service data to be stored in the slave node with the highest storage efficiency and the number of the service data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold, so that the technical problems that in the existing service data storage method, when a master node sends the service data to the slave nodes, the data reliability is low, the storage effect is poor, and the user experience is poor are solved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a service data storage method according to an embodiment of the present invention;
fig. 2a is a schematic diagram of a main flow of a business data storage method according to another embodiment of the present invention;
fig. 2b is a schematic diagram of a main flow of a business data storage method according to still another embodiment of the present invention;
FIG. 3 is a schematic diagram of the main modules of a business data storage apparatus provided in accordance with another embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a service data storage method according to an embodiment of the present invention; as shown in fig. 1, the service data storage method provided in the embodiment of the present invention is mainly applied to a distributed storage cluster including multiple nodes, and mainly includes:
step S101, receiving service data to be stored, and storing the service data in a main node of the distributed storage cluster.
Specifically, according to the embodiment of the present invention, the service data may be service data to be stored, which is directly written by a user, or may be service data to be stored, which is written by a service system. The master node of the distributed storage cluster receives the service data to be stored, and the master node distributes the service data to the slave nodes of the distributed storage cluster in the follow-up process so as to store service data copies in the slave nodes, thereby realizing that one service data is respectively stored in all the nodes of the distributed storage cluster and avoiding data loss.
Step S102, determining the storage efficiency of a plurality of slave nodes in the distributed storage cluster.
Specifically, according to the embodiment of the present invention, the step of determining the storage efficiency of a plurality of slave nodes in a distributed storage cluster includes:
And determining the storage efficiency of the plurality of slave nodes according to the heartbeats between the master node and the plurality of slave nodes respectively.
In specific implementation, if the service data copy is not stored in the slave node, the storage efficiency cannot be determined according to the actual storage condition, and the storage efficiency can be determined by the central hop between the master node and the slave node.
Preferably, according to an embodiment of the present invention, the method further includes:
and updating the storage efficiency according to the slave node storage schedule.
After all or part of the service data copies are stored in the slave node, the storage efficiency of the slave node can be determined according to the storage condition (recording the number of the stored data copies in a certain time interval) recorded in the slave node storage schedule. According to the storage efficiency of the slave nodes, the number or data volume of the service data to be stored, which are sent to different slave nodes, is controlled, so that the efficiency of storing service data copies in a plurality of slave nodes can be kept consistent as much as possible, the data consistency among different nodes is realized, and the storage efficiency is improved.
Step S103, transmitting the service data to a plurality of slave nodes by using the master node so as to respectively store the service data copies in the plurality of slave nodes; the number of the business data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the business data to be stored in the slave node with the highest storage efficiency and the number of the business data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold. Wherein, the first quantity threshold value can be set to 1000 as a limiting condition for limiting the maximum storage quantity in a single time; the second number threshold may be set to 256 as a limiting condition for determining a maximum difference in storage progress between nodes. It should be noted that the above values are only examples, and the specific values of the first quantity threshold and the second quantity threshold may be adjusted according to the storage performance of the actual node and the requirement on data consistency.
Specifically, according to an embodiment of the present invention, the step of sending the service data to the plurality of slave nodes by using the master node to store the service data in the plurality of slave nodes respectively includes:
numbering the service data stored in the main node;
acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first quantity threshold; the slave node storage schedule indicates the current quantity of the business data to be stored of a plurality of slave nodes;
calculating the quantity difference between the quantity of the current service data to be stored in the candidate slave nodes and the quantity of the current service data to be stored in the slave nodes with the lowest storage efficiency, judging whether the quantity difference does not exceed a second quantity threshold value, if so, determining the candidate slave nodes as target slave nodes, and sending the target service data to the target slave nodes by using the master nodes; and the sum of the quantity of the target service data and the quantity of the service data to be stored currently in the target slave node does not exceed a first quantity threshold value.
In the foregoing, it is mentioned that the storage efficiency of the slave nodes can be determined according to the storage status recorded in the slave node storage schedule, and meanwhile, since the slave node storage schedule also indicates the current amount of the service data to be stored of the plurality of slave nodes, the slave node with the highest storage efficiency can be selected as the candidate slave node from the slave nodes whose current amount of the service data to be stored does not exceed the first number threshold. Through the arrangement, the service data stored in the master node are numbered, so that the quantity difference between the quantity of the service data to be stored in any slave node and the quantity of the service data to be stored in the slave node with the lowest storage efficiency is controlled to be smaller than the second data threshold value from the perspective of the quantity of the service data subsequently, the fact that a plurality of slave nodes finish storing all the service data in a similar time period is guaranteed, data consistency is guaranteed, meanwhile, the sum of the quantity of the target service data sent to the slave nodes by the master node and the quantity of the current service data to be stored is controlled not to exceed the first quantity threshold value, and overload of the slave nodes is avoided.
Optionally, according to an embodiment of the present invention, the step of sending the service data to the multiple slave nodes by using the master node to store the service data in the multiple slave nodes respectively further includes:
determining the data volume corresponding to the numbered service data;
acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first data volume threshold; the slave node storage schedule indicates the data volume corresponding to the current business data to be stored of a plurality of slave nodes;
calculating a data volume difference value between the data volume of the current service data to be stored in the candidate slave nodes and the data volume of the current service data to be stored in the slave nodes with the lowest storage efficiency, judging whether the data volume difference value does not exceed a second data volume threshold value, if so, determining the candidate slave nodes as target slave nodes, and sending the target service data to the target slave nodes by using the master node, wherein the sum of the data volume of the target service data and the data volume of the current service data to be stored in the target slave nodes does not exceed the first data volume threshold value.
Through the setting, the data volume corresponding to the service data after numbering is determined, so that the data volume difference value between the data volume of the service data to be stored in any slave node and the data volume of the service data to be stored in the slave node with the lowest storage efficiency is controlled to be smaller than the second data volume threshold value from the perspective of the data volume of the service data, the storage of all service data in a plurality of slave nodes is guaranteed to be completed in a similar time period, the data consistency is guaranteed, meanwhile, the sum of the data volume of the target service data sent to the slave node by the master node and the data volume of the current service data to be stored is controlled not to exceed the first data volume threshold value, and the overload of the slave nodes is avoided.
Further, according to an embodiment of the present invention, before the step of obtaining the slave node storage schedule, the method further includes:
acquiring the storage states of the service data to be stored in the plurality of slave nodes in real time;
and constructing a slave node storage schedule according to the storage state.
Through the setting, the storage schedule of the slave nodes is established, the current quantity of the service data to be stored of the slave nodes or the data volume of the current service data to be stored can be known in real time, and the subsequent master node is facilitated to adjust the quantity of the service data or the data volume sent to the slave nodes, so that the data consistency is guaranteed.
Exemplarily, according to an embodiment of the present invention, the method further includes:
setting a storage efficiency threshold, and determining the time length for which the storage efficiency of the slave node is less than the storage efficiency threshold;
and if the duration exceeds a duration threshold, adjusting the slave nodes of the distributed storage cluster.
Through the setting, in order to further guarantee the storage effect of the distributed storage cluster, if the storage efficiency of one or more slave nodes is lower than the storage efficiency threshold within a certain time (the time threshold), the slave nodes can be maintained or newly added, that is, a storage efficiency threshold is set as a trigger condition for replacing or maintaining the slave nodes, the value of the storage efficiency threshold can be set according to the actual situation, and if the requirement on the storage efficiency of the slave nodes is not high, a lower storage efficiency threshold can be set; if the requirement on the storage efficiency of the slave node is high, a high storage efficiency threshold value needs to be set.
According to the technical scheme of the embodiment of the invention, the service data to be stored are received and stored in the main node of the distributed storage cluster; determining storage efficiency of a plurality of slave nodes in a distributed storage cluster; transmitting, by the master node, the traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the method comprises the steps that the number of the business data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the business data to be stored in the slave node with the highest storage efficiency and the number of the business data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold, so that the technical problems of low data reliability, poor storage effect and poor user experience caused by the fact that the performance and the load capacity of different nodes are not considered when a master node sends the business data to the slave nodes in the existing business data storage method are solved, and the storage efficiency of different slave nodes is combined, business data copies are stored in multiple slave nodes, so that the data reliability is guaranteed, the storage effect is improved, and the technical effect of the user experience is improved.
Fig. 2a is a schematic diagram of a main flow of a business data storage method according to another embodiment of the present invention; as shown in fig. 2a, the service data storage method provided in the embodiment of the present invention is applied to a distributed storage cluster, and mainly includes:
Step S201, receiving service data to be stored, and storing the service data in the master node of the distributed storage cluster.
Specifically, according to the embodiment of the present invention, the service data may be service data to be stored, which is directly written by a user, or may be service data to be stored, which is written by a service system. The master node of the distributed storage cluster receives the service data to be stored, and the master node distributes the service data to the slave nodes of the distributed storage cluster in the follow-up process, so that service data copies are stored in the slave nodes, one service data copy is respectively stored in all the nodes of the distributed storage cluster, and data loss is avoided.
Step S202, determining the storage efficiency of a plurality of slave nodes according to the heartbeat between the master node and the plurality of slave nodes respectively.
In specific implementation, if the service data copy is not stored in the slave node, the storage efficiency cannot be determined according to the actual storage condition, and the storage efficiency can be determined by the central hop between the master node and the slave node.
Step S203, numbering the service data stored in the main node; acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first quantity threshold; wherein, the slave node storage schedule indicates the current quantity of the service data to be stored of a plurality of slave nodes.
Step S204, calculating a quantity difference between the quantity of the current business data to be stored in the candidate slave node and the quantity of the current business data to be stored in the slave node with the lowest storage efficiency.
Step S205, judging whether the quantity difference value does not exceed a second quantity threshold value; if yes, i.e. the number difference does not exceed the second number threshold, step S206 is executed; if not, i.e. the number difference exceeds the second number threshold, go to step S203.
Step S206, determining the candidate slave node as a target slave node, and sending target service data to the target slave node by using the master node; and the sum of the quantity of the target service data and the quantity of the service data to be stored currently in the target slave node does not exceed a first quantity threshold value.
Through the setting, the service data stored in the master node is numbered, so that the quantity difference between the quantity of the service data to be stored in any slave node and the quantity of the service data to be stored in the slave node with the lowest storage efficiency is controlled to be smaller than the second data threshold value from the perspective of the quantity of the service data subsequently, the fact that a plurality of slave nodes complete storage of all service data in a similar time period is guaranteed, data consistency is guaranteed, meanwhile, the sum of the quantity of the target service data sent to the slave node by the master node and the quantity of the current service data to be stored is controlled not to exceed the first quantity threshold value, and overload of the slave nodes is avoided.
Further, according to an embodiment of the present invention, before the step of obtaining the slave node storage schedule, the method further includes:
acquiring the storage states of service data to be stored in a plurality of slave nodes in real time;
and constructing a slave node storage schedule according to the storage state.
Through the setting, the storage schedule of the slave nodes is established, the current quantity of the service data to be stored of the slave nodes or the data volume of the current service data to be stored can be known in real time, and the subsequent master node is facilitated to adjust the quantity of the service data or the data volume sent to the slave nodes, so that the data consistency is guaranteed.
Illustratively, according to an embodiment of the present invention, the method further includes:
setting a storage efficiency threshold, and determining the time length of the slave node with the storage efficiency smaller than the storage efficiency threshold;
and if the duration exceeds a duration threshold, adjusting the slave nodes of the distributed storage cluster.
Through the setting, in order to further guarantee the storage effect of the distributed storage cluster, if the storage efficiency of one or more slave nodes is lower than the storage efficiency threshold within a certain time (the time threshold), the slave nodes can be maintained or newly added.
Referring to fig. 2b, an embodiment of the present invention provides another service data storage method, which mainly includes:
Step S211, receiving the service data to be stored, and storing the service data in the master node of the distributed storage cluster.
And step S212, determining the storage efficiency of the plurality of slave nodes according to heartbeats between the master node and the plurality of slave nodes and the slave node storage schedule.
Step S213, numbering the service data stored in the main node; acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first quantity threshold; wherein, the slave node storage schedule indicates the current quantity of the service data to be stored of a plurality of slave nodes.
Step S214, calculating a quantity difference between the quantity of the current business data to be stored in the candidate slave node and the quantity of the current business data to be stored in the slave node with the lowest storage efficiency.
Step S215, judging whether the quantity difference value does not exceed a second quantity threshold value; if yes, i.e. the number difference does not exceed the second number threshold, go to step S216; if not, i.e. the number difference exceeds the second number threshold, go to step S213.
Step S216, determining the data volume corresponding to the numbered service data; acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first data volume threshold; the slave node storage schedule indicates the data amount corresponding to the current service data to be stored of the plurality of slave nodes.
Step S217, calculating a data amount difference between the data amount of the current service data to be stored in the candidate slave node and the data amount of the current service data to be stored in the slave node with the lowest storage efficiency.
Step S218, determining whether the data amount difference does not exceed the second data amount threshold, if yes, performing step S219; if not, go to step S216;
step S219, determining candidate slave nodes as target slave nodes, and sending target service data to the target slave nodes by using the master node, wherein the sum of the data volume of the target service data and the data volume of the current service data to be stored in the target slave nodes does not exceed a first data volume threshold; the sum of the quantity of the target service data and the quantity of the service data to be stored currently in the target slave node does not exceed a first quantity threshold value.
Through the setting, the service data stored in the master node is numbered, and the data volume corresponding to the numbered service data is determined, so that from the two aspects of the number of the service data and the data volume, that is, the difference between the number of the service data to be stored in any slave node and the number of the service data to be stored in the slave node with the lowest storage efficiency is controlled to be smaller than the second data threshold, and the difference between the data volume of the service data to be stored in any slave node and the data volume of the service data to be stored in the slave node with the lowest storage efficiency is controlled to be smaller than the second data threshold, so that the storage of all the service data in a plurality of slave nodes in a similar time period is ensured, and the data consistency is ensured.
Meanwhile, the sum of the target service data quantity and the current service data quantity to be stored sent to the slave node by the master node is controlled not to exceed a first quantity threshold value, and the sum of the data quantity of the target service data and the current service data quantity to be stored sent to the slave node by the master node is controlled not to exceed the first data quantity threshold value, so that overload of the slave node is obviously avoided.
According to the embodiment of the present invention, it can be understood that, from the perspective of the data volume of the service data, it is also ensured that all the service data are stored in the plurality of slave nodes in a similar time period, so as to achieve data consistency.
According to the technical scheme of the embodiment of the invention, the service data to be stored are received and stored in the main node of the distributed storage cluster; determining storage efficiency of a plurality of slave nodes in a distributed storage cluster; transmitting, by the master node, the traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the method comprises the steps that the number of the service data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the service data to be stored in the slave node with the highest storage efficiency and the number of the service data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold, so that the technical problems that in the existing service data storage method, when a master node sends the service data to the slave nodes, the data reliability is low, the storage effect is poor, and the user experience is poor are solved.
FIG. 3 is a schematic diagram of the main modules of a business data storage apparatus provided in accordance with another embodiment of the present invention; as shown in fig. 3, a service data storage apparatus 300 provided in the embodiment of the present invention is mainly disposed in a distributed storage cluster including a plurality of nodes, and mainly includes:
a receiving module 301, configured to: and receiving the service data to be stored, and storing the service data in the main node of the distributed storage cluster.
Specifically, according to the embodiment of the present invention, the service data may be service data to be stored, which is directly written by a user, or may be service data to be stored, which is written by a service system. The master node of the distributed storage cluster receives the service data to be stored, and the master node distributes the service data to the slave nodes of the distributed storage cluster in the follow-up process so as to store service data copies in the slave nodes, thereby realizing that one service data is respectively stored in all the nodes of the distributed storage cluster and avoiding data loss.
A storage efficiency determination module 302 to: the storage efficiency of a plurality of slave nodes in a distributed storage cluster is determined.
Specifically, according to an embodiment of the present invention, the storage efficiency determining module 302 is further configured to:
And determining the storage efficiency of the plurality of slave nodes according to the heartbeats between the master node and the plurality of slave nodes respectively.
In specific implementation, if the service data copy is not stored in the slave node, the storage efficiency cannot be determined according to the actual storage condition, and the storage efficiency can be determined by the central hop between the master node and the slave node.
Preferably, according to an embodiment of the present invention, the storage efficiency determining module 302 is further configured to:
and updating the storage efficiency according to the slave node storage schedule.
After all or part of the service data copies are stored in the slave node, the storage efficiency of the slave node can be determined according to the storage condition (the number of the stored data copies in a certain time interval is recorded) recorded in the slave node storage schedule. According to the storage efficiency of the slave nodes, the number or data volume of the service data to be stored sent to different slave nodes is controlled, and the efficiency of storing service data copies in the slave nodes can be kept consistent as much as possible, so that the data consistency among different nodes is realized, and the storage efficiency is improved.
A service data storage module 303, configured to: transmitting, by the master node, traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the number of the service data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the service data to be stored in the slave node with the highest storage efficiency and the number of the service data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold. Wherein, the first quantity threshold value can be set to 1000 as a limiting condition for limiting the maximum storage quantity in a single time; the second number threshold may be set to 256 as a limiting condition for determining a maximum difference in storage progress between nodes. It should be noted that the above values are only examples, and the specific values of the first quantity threshold and the second quantity threshold may be adjusted according to the storage performance of the actual node and the requirement on data consistency.
Specifically, according to the embodiment of the present invention, the service data storage module 303 is further configured to:
numbering the service data stored in the main node;
acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first quantity threshold; the slave node storage schedule indicates the current quantity of the business data to be stored of a plurality of slave nodes;
calculating the quantity difference between the quantity of the current service data to be stored in the candidate slave nodes and the quantity of the current service data to be stored in the slave nodes with the lowest storage efficiency, judging whether the quantity difference does not exceed a second quantity threshold value, if so, determining the candidate slave nodes as target slave nodes, and sending the target service data to the target slave nodes by using the master nodes; and the sum of the quantity of the target service data and the quantity of the service data to be stored currently in the target slave node does not exceed a first quantity threshold value.
In the foregoing, it is mentioned that the storage efficiency of the slave nodes can be determined according to the storage status recorded in the slave node storage schedule, and meanwhile, since the slave node storage schedule also indicates the current amount of the service data to be stored of the plurality of slave nodes, the slave node with the highest storage efficiency can be selected as the candidate slave node from the slave nodes whose current amount of the service data to be stored does not exceed the first number threshold. Through the arrangement, the service data stored in the master node are numbered, so that the quantity difference between the quantity of the service data to be stored in any slave node and the quantity of the service data to be stored in the slave node with the lowest storage efficiency is controlled to be smaller than the second data threshold value from the perspective of the quantity of the service data subsequently, the fact that a plurality of slave nodes finish storing all the service data in a similar time period is guaranteed, data consistency is guaranteed, meanwhile, the sum of the quantity of the target service data sent to the slave nodes by the master node and the quantity of the current service data to be stored is controlled not to exceed the first quantity threshold value, and overload of the slave nodes is avoided.
Optionally, according to the embodiment of the present invention, the service data storage module 303 is further configured to:
determining the data volume corresponding to the numbered service data;
acquiring a slave node storage schedule, and determining candidate slave nodes according to the slave node storage schedule and a first data volume threshold; the slave node storage schedule indicates the data volume corresponding to the current business data to be stored of a plurality of slave nodes;
calculating a data volume difference between the data volume of the current service data to be stored in the candidate slave node and the data volume of the current service data to be stored in the slave node with the lowest storage efficiency, judging whether the data volume difference is not more than a second data volume threshold value, if so, determining the candidate slave node as a target slave node, and sending the target service data to the target slave node by using the master node, wherein the sum of the data volume of the target service data and the data volume of the current service data to be stored in the target slave node is not more than a first data volume threshold value.
Through the setting, the data volume corresponding to the service data after numbering is determined, so that the data volume difference between the data volume of the service data to be stored in any slave node and the data volume of the service data to be stored in the slave node with the lowest storage efficiency is controlled to be smaller than the second data volume threshold from the perspective of the data volume of the service data, the storage of all service data in a plurality of slave nodes within a similar time period is guaranteed, the data consistency is guaranteed, meanwhile, the sum of the data volume of the target service data sent to the slave node by the master node and the data volume of the current service data to be stored is controlled not to exceed the first data volume threshold, and overload of the slave nodes is avoided.
Further, according to the embodiment of the present invention, the service data storage apparatus 300 further includes a slave node storage schedule constructing module, configured to:
before the step of obtaining the slave node storage schedule, the method further comprises:
acquiring the storage states of the service data to be stored in the plurality of slave nodes in real time;
and constructing a slave node storage schedule according to the storage state.
Through the setting, the storage schedule of the slave nodes is established, the current quantity of the service data to be stored of the slave nodes or the data volume of the current service data to be stored can be known in real time, and the subsequent master node is facilitated to adjust the quantity of the service data or the data volume sent to different slave nodes, so that the data consistency is guaranteed.
Illustratively, according to the embodiment of the present invention, the service data storage apparatus 300 further includes a slave node adjusting module, configured to:
setting a storage efficiency threshold, and determining the time length for which the storage efficiency of the slave node is less than the storage efficiency threshold;
and if the duration exceeds a duration threshold, adjusting the slave nodes of the distributed storage cluster.
Through the setting, in order to further guarantee the storage effect of the distributed storage cluster, if the storage efficiency of one or more slave nodes is lower than the storage efficiency threshold within a certain time (the time threshold), the slave nodes can be maintained or newly added, that is, a storage efficiency threshold is set as a trigger condition for replacing or maintaining the slave nodes, the value of the storage efficiency threshold can be set according to the actual situation, and if the requirement on the storage efficiency of the slave nodes is not high, a lower storage efficiency threshold can be set; if the requirement on the storage efficiency of the slave node is high, a high storage efficiency threshold value needs to be set.
According to the technical scheme of the embodiment of the invention, the service data to be stored is received and stored in the main node of the distributed storage cluster; determining storage efficiency of a plurality of slave nodes in a distributed storage cluster; transmitting, by the master node, traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the method comprises the steps that the number of the service data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the service data to be stored in the slave node with the highest storage efficiency and the number of the service data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold, so that the technical problems that in the existing service data storage method, when a master node sends the service data to the slave nodes, the data reliability is low, the storage effect is poor, and the user experience is poor are solved.
Fig. 4 shows an exemplary system architecture 400 of a service data storage method or service data storage apparatus to which an embodiment of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405 (this architecture is merely an example, and the components included in a particular architecture may be adapted according to application specific circumstances). The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have installed thereon various communication client applications, such as a business data storage-type application, a web browser application, a search-type application, an instant messaging tool, a database cluster, social platform software, etc. (by way of example only).
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server that provides various services, such as a server (for example only) for (storing/processing of) traffic data by users using the terminal devices 401, 402, 403. The server may analyze and perform other processing on the received data such as the service data to be stored, and feed back a processing result (e.g., storage efficiency, service data — just an example) to the terminal device.
It should be noted that, the service data storage method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the service data storage device is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for an implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device or server implementing embodiments of the present invention. The terminal device or the server shown in fig. 5 is only an example, and should not bring any limitation to the functions and the use range of the embodiment of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a receiving module, a storage efficiency determination module, and a traffic data storage module. The names of these modules do not in some cases form a limitation on the modules themselves, for example, the receiving module may also be described as a "module for receiving the service data to be stored and storing the service data in the master node of the distributed storage cluster".
As another aspect, the present invention also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving service data to be stored, and storing the service data in a main node of a distributed storage cluster; determining storage efficiency of a plurality of slave nodes in a distributed storage cluster; transmitting, by the master node, the traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the number of the service data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the service data to be stored in the slave node with the highest storage efficiency and the number of the service data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold.
According to the technical scheme of the embodiment of the invention, the service data to be stored is received and stored in the main node of the distributed storage cluster; determining storage efficiency of a plurality of slave nodes in a distributed storage cluster; transmitting, by the master node, traffic data to the plurality of slave nodes to store traffic data copies in the plurality of slave nodes, respectively; the method comprises the steps that the number of the service data to be stored sent to any slave node does not exceed a first number threshold, and the difference between the number of the service data to be stored in the slave node with the highest storage efficiency and the number of the service data to be stored in the slave node with the lowest storage efficiency does not exceed a second number threshold, so that the technical problems that in the existing service data storage method, when a master node sends the service data to the slave nodes, the data reliability is low, the storage effect is poor, and the user experience is poor are solved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210393314.7A CN114756172B (en) | 2022-04-15 | 2022-04-15 | Business data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210393314.7A CN114756172B (en) | 2022-04-15 | 2022-04-15 | Business data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114756172A true CN114756172A (en) | 2022-07-15 |
| CN114756172B CN114756172B (en) | 2025-07-15 |
Family
ID=82331665
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210393314.7A Active CN114756172B (en) | 2022-04-15 | 2022-04-15 | Business data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114756172B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115599807A (en) * | 2022-10-28 | 2023-01-13 | 中国农业银行股份有限公司（Cn） | Data access method, device, application server and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2015088109A (en) * | 2013-11-01 | 2015-05-07 | 株式会社エヌ・ティ・ティ・データ | Storage system, storage control method, and storage control program |
| KR101626067B1 (en) * | 2015-09-08 | 2016-05-31 | 주식회사 브이시스템즈 | Apparatus and method for facilitating a transfer of container between slave nodes |
| CN105630416A (en) * | 2015-12-24 | 2016-06-01 | 创新科存储技术（深圳）有限公司 | Disc kicking method and device in cloud storage system |
| CN110781156A (en) * | 2019-09-20 | 2020-02-11 | 苏州浪潮智能科技有限公司 | A method, device and medium for allocating data nodes |
| CN111552441A (en) * | 2020-04-29 | 2020-08-18 | 重庆紫光华山智安科技有限公司 | Data storage method and device, main node and distributed system |
-
2022
- 2022-04-15 CN CN202210393314.7A patent/CN114756172B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2015088109A (en) * | 2013-11-01 | 2015-05-07 | 株式会社エヌ・ティ・ティ・データ | Storage system, storage control method, and storage control program |
| KR101626067B1 (en) * | 2015-09-08 | 2016-05-31 | 주식회사 브이시스템즈 | Apparatus and method for facilitating a transfer of container between slave nodes |
| CN105630416A (en) * | 2015-12-24 | 2016-06-01 | 创新科存储技术（深圳）有限公司 | Disc kicking method and device in cloud storage system |
| CN110781156A (en) * | 2019-09-20 | 2020-02-11 | 苏州浪潮智能科技有限公司 | A method, device and medium for allocating data nodes |
| CN111552441A (en) * | 2020-04-29 | 2020-08-18 | 重庆紫光华山智安科技有限公司 | Data storage method and device, main node and distributed system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115599807A (en) * | 2022-10-28 | 2023-01-13 | 中国农业银行股份有限公司（Cn） | Data access method, device, application server and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114756172B (en) | 2025-07-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101871383B1 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| US10250480B2 (en) | Polling parameter adjustment | |
| CN109542352B (en) | Method and apparatus for storing data | |
| CN113742389A (en) | Service processing method and device | |
| JP2023545985A (en) | Managing task flows in edge computing environments | |
| CN110019158A (en) | A kind of method and apparatus of monitoring data quality | |
| CN109428926B (en) | Method and device for scheduling task nodes | |
| CN114612212A (en) | Business processing method, device and system based on risk control | |
| CN114296953A (en) | Multi-cloud heterogeneous system and task processing method | |
| CN112073395B (en) | File distribution method and device | |
| CN107818027B (en) | Method and device for switching main name node and standby name node and distributed system | |
| CN114756172B (en) | Business data storage method and device | |
| CN108875035A (en) | The date storage method and relevant device of distributed file system | |
| CN113947434B (en) | A business processing method and device for a multi-business component combination scenario | |
| CN111210349A (en) | Virtual asset transfer hosting method and device, electronic equipment and storage medium | |
| CN113778993A (en) | Service data processing method and device | |
| CN118368259B (en) | Network resource allocation method, device, electronic equipment and storage medium | |
| CN113760179B (en) | A data processing method and apparatus | |
| CN113722548A (en) | Method and device for processing reference relationship in business system | |
| CN115328674A (en) | Data writing method, device and system | |
| JP2025518158A (en) | Method, apparatus and system for managing cluster access rights - Patents.com | |
| US11526499B2 (en) | Adaptively updating databases of publish and subscribe systems using optimistic updates | |
| CN115022053A (en) | Method, apparatus and system for accessing a blockchain network | |
| CN113778660A (en) | System and method for managing hot spot data | |
| CN114090524A (en) | A kind of Excel file distributed export method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |