CN113760179A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN113760179A CN113760179A CN202110153654.8A CN202110153654A CN113760179A CN 113760179 A CN113760179 A CN 113760179A CN 202110153654 A CN202110153654 A CN 202110153654A CN 113760179 A CN113760179 A CN 113760179A
- Authority
- CN
- China
- Prior art keywords
- storage
- data processing
- configuration information
- storage device
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种数据处理方法和装置，涉及计算机技术领域。该方法的一具体实施方式包括：响应于业务服务器发起的数据处理请求，获取存储实例列表；其中，数据处理请求包括数据读取请求和/或数据写入请求；判断存储实例列表中是否包括多个存储实例；其中，存储实例指示了业务服务器与存储装置之间的对应关系；若是，确定数据处理请求对应的待处理数据的存储键值，获取存储配置信息，根据存储键值和存储配置信息确定目标存储实例，并将数据处理请求发送至目标存储实例指示的存储装置，以使得存储装置执行相应的数据处理。该实施方式能够均衡各存储装置的负载，缩短数据处理耗时，降低系统风险，提高数据处理效率，提升用户体验。
The invention discloses a data processing method and device, and relates to the technical field of computers. A specific implementation of the method includes: obtaining a storage instance list in response to a data processing request initiated by a service server; wherein the data processing request includes a data read request and/or a data write request; and determining whether the storage instance list includes multiple A storage instance; wherein, the storage instance indicates the corresponding relationship between the service server and the storage device; if so, determine the storage key value of the data to be processed corresponding to the data processing request, obtain the storage configuration information, and obtain the storage configuration information according to the storage key value and the storage configuration information. A target storage instance is determined, and a data processing request is sent to the storage device indicated by the target storage instance, so that the storage device performs corresponding data processing. This embodiment can balance the load of each storage device, shorten data processing time, reduce system risks, improve data processing efficiency, and improve user experience.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
The distributed cache system supports multiple data types and capacity expansion, and is often used for solving the bottleneck between a database server and a Web server. However, with the continuous growth of internet services, one distributed cache system cluster often cannot meet service requirements, and needs to be expanded into two or more clusters. How to dynamically route the request of the service server to the corresponding distributed cache system to realize data processing is a problem which needs to be solved at present.
The prior art has at least the following problems:
in the existing data processing method, because the access requests of the service server need to be sent to a certain storage device in a unified manner, or sent after being respectively polled, the load capacity of the storage device is unbalanced, the processing time consumption is long, the system risk is high, downtime is easily caused, the data processing efficiency is low, the user experience is poor, and the like.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method and apparatus, which can balance loads of each storage apparatus, shorten data processing time, reduce system risk, improve data processing efficiency, and improve user experience.
To achieve the above object, according to a first aspect of embodiments of the present invention, there is provided a data processing method including:
responding to a data processing request initiated by a service server, and acquiring a storage instance list; the data processing request comprises a data reading request and/or a data writing request;
judging whether the storage instance list comprises a plurality of storage instances or not; the storage instance indicates the corresponding relation between the service server and the storage device;
if so, determining a storage key value of the to-be-processed data corresponding to the data processing request, acquiring storage configuration information, determining a target storage instance according to the storage key value and the storage configuration information, and sending the data processing request to the storage device indicated by the target storage instance, so that the storage device executes corresponding data processing.
Further, determining a storage key value of the to-be-processed data corresponding to the data processing request further includes: and acquiring service requirements, and determining a storage key value according to the service requirements and the data type of the data to be processed.
Further, sending the data processing request to the storage device storing the indication of the instance, further comprises:
and determining a corresponding storage device according to the service server number corresponding to the data processing request and the storage instance so as to send the data processing request to the storage device.
Further, the storage configuration information includes at least one of the following information: the configuration information of the business server, the configuration information of the storage key value and the configuration information of the storage device.
Further, if the storage instance list includes a plurality of storage instances, determining a target storage instance according to the storage key value and the storage configuration information, further comprising:
setting matching priority for the configuration information of the business server, the configuration information of the storage key value and the configuration information of the storage device;
and matching the storage key value with the matching priority of the plurality of information in the storage configuration information in sequence to determine a target storage instance.
Further, the method further comprises: the method comprises the steps of obtaining the current load capacity of a plurality of storage devices, determining a target storage device according to the current load capacity, and sending a data processing request to the target storage device.
Further, the storage device is a distributed cache system cluster, and the method further comprises:
and adjusting the distributed cache system corresponding to the data processing request according to the load capacity of a plurality of distributed cache systems in the distributed cache system cluster.
According to a second aspect of the embodiments of the present invention, there is provided a data processing apparatus including:
the acquisition module is used for responding to a data processing request initiated by the service server and acquiring a storage instance list; the data processing request comprises a data reading request and/or a data writing request;
the judging module is used for judging whether the storage instance list comprises a plurality of storage instances; the storage instance indicates the corresponding relation between the service server and the storage device;
and if the storage instance list comprises a plurality of storage instances, the data processing module is used for determining a storage key value of the to-be-processed data corresponding to the data processing request, acquiring storage configuration information, determining a target storage instance according to the storage key value and the storage configuration information, and sending the data processing request to the storage device indicated by the target storage instance, so that the storage device executes corresponding data processing.
According to a third aspect of embodiments of the present invention, there is provided an electronic apparatus, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement any of the data processing methods described above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing any one of the data processing methods described above.
One embodiment of the above invention has the following advantages or benefits: because the data processing request initiated by responding to the service server is adopted, the storage instance list is obtained; the data processing request comprises a data reading request and/or a data writing request; judging whether the storage instance list comprises a plurality of storage instances or not; the storage instance indicates the corresponding relation between the service server and the storage device; if the data processing method is adopted, the storage key value of the to-be-processed data corresponding to the data processing request is determined, the storage configuration information is obtained, the target storage instance is determined according to the storage key value and the storage configuration information, and the data processing request is sent to the storage device indicated by the target storage instance, so that the storage device executes the corresponding data processing technical means.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a data processing method provided according to a first embodiment of the present invention;
fig. 2 is a schematic diagram of a main flow of a data processing method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of the main blocks of a data processing apparatus according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a data processing method provided according to a first embodiment of the present invention; as shown in fig. 1, the data processing method provided in the embodiment of the present invention mainly includes:
step S101, responding to a data processing request initiated by a service server, and acquiring a storage instance list; the data processing request comprises a data reading request and/or a data writing request.
Specifically, according to the embodiment of the present invention, a large number of users send data reading requests or data writing requests based on the service servers, and in the face of a high-concurrency user request, a storage instance list is obtained, where the storage instance list records the corresponding relationship between each service server and the storage device, so as to match the corresponding storage device for the user request, which is beneficial to improving data processing efficiency and improving the technical effect of user experience.
Step S102, judging whether the storage example list comprises a plurality of storage examples; wherein the storage instance indicates a correspondence between the service server and the storage device. If yes, that is, the storage instance list includes a plurality of (greater than or equal to two) storage instances, execute step S103; if not, the step S104 is executed if the storage instance list includes at most one storage instance.
Specifically, according to the embodiment of the present invention, if the storage instance list only includes one storage instance, that is, for the service server, the storage instance list only corresponds to one storage device, in such a situation, after receiving the data processing request of the service server, only the data processing request needs to be sent to the unique storage device, and no additional routing call operation is required. If the storage instance list includes a plurality of (greater than or equal to two) storage instances, that is, for the service server, it needs to match the target storage device for the data processing request sent by the service server according to the subsequent storage configuration information, so as to complete data processing, thereby achieving the purposes of improving data processing efficiency and improving user experience.
Step S103, if the storage instance list includes a plurality of storage instances, determining a storage key value of the to-be-processed data corresponding to the data processing request, acquiring storage configuration information, determining a target storage instance according to the storage key value and the storage configuration information, and sending the data processing request to the storage device indicated by the target storage instance, so that the storage device executes corresponding data processing.
Specifically, according to the embodiment of the present invention, the step of determining the storage key value of the to-be-processed data corresponding to the data processing request further includes: and acquiring service requirements, and determining a storage key value according to the service requirements and the data type of the data to be processed.
Through the setting, different storage key values are set for the data types of the data to be processed according to the service requirements, the data processing requests can be matched with the target storage devices according to the storage key values subsequently, and the load balance among the storage devices can be optimized.
Further, according to an embodiment of the present invention, the storage configuration information includes at least one of the following information: the configuration information of the business server, the configuration information of the storage key value and the configuration information of the storage device.
The configuration information of the service server mainly refers to configuration information of the same machine room, namely, the configuration information of the service server is a storage device in the same machine room.
The storage key value configuration information refers to setting the grade of the data to be processed according to the storage key value, wherein the storage key value can be a key value in a specific interval, and can also be set in the form of a specific prefix and a specific suffix to distinguish the data to be processed in different grades, so that the target storage devices are matched according to the storage key value to optimize load balance among the storage devices, the data in the same grade is written into the same storage device, and subsequent reading and other operations are facilitated.
The storage device configuration information is used for adjusting the corresponding data processing amount according to the current load of the storage device. I.e. a storage device with a smaller load is matched to the data processing request.
Preferably, according to an embodiment of the present invention, if the storage instance list includes a plurality of storage instances, the step of determining the target storage instance according to the storage key value and the storage configuration information further includes:
setting matching priority for the configuration information of the business server, the configuration information of the storage key value and the configuration information of the storage device;
and matching the storage key value with the matching priority of the plurality of information in the storage configuration information in sequence to determine a target storage instance.
Through the setting, the priority is set for the plurality of information in the storage configuration information, the routing scheduling of the access request is further perfected, the load balancing is realized, the system risk is reduced, the data processing efficiency is improved, and the user experience is improved.
Further, according to an embodiment of the present invention, the method further includes: the method comprises the steps of obtaining the current load capacity of a plurality of storage devices, determining a target storage device according to the current load capacity, and sending a data processing request to the target storage device.
Through the arrangement, the target storage device is determined according to the current load capacity of the plurality of storage devices, load balance is achieved, downtime risk is avoided, multiple time-consuming queries are avoided, and data processing efficiency is improved.
Preferably, according to an embodiment of the present invention, the storage apparatus is a distributed cache system cluster, and the method further includes:
and adjusting the distributed cache system corresponding to the data processing request according to the load capacity of a plurality of distributed cache systems in the distributed cache system cluster.
Through the setting, the distributed cache system cluster is adopted, high-concurrency user requests can be responded, and the service capability and the application scene of the data processing method are expanded.
Step S104, if only one storage instance is included in the storage instance list, sending the data processing request to the storage device indicated by the storage instance, so that the storage device executes corresponding data processing.
Specifically, according to the embodiment of the present invention, the step of sending the data processing request to the storage device indicated by the storage instance further includes:
and determining a corresponding storage device according to the service server number corresponding to the data processing request and the storage instance so as to send the data processing request to the storage device.
According to a specific implementation manner of the embodiment of the present invention, the service server is preferentially matched with the storage devices in the same machine room, and a storage instance is generated for the matching relationship and stored in the storage instance list. Through the arrangement, subsequent data management is facilitated.
According to the technical scheme of the embodiment of the invention, the storage instance list is obtained by responding to the data processing request initiated by the service server; the data processing request comprises a data reading request and/or a data writing request; judging whether the storage instance list comprises a plurality of storage instances or not; the storage instance indicates the corresponding relation between the service server and the storage device; if the data processing method is adopted, the storage key value of the to-be-processed data corresponding to the data processing request is determined, the storage configuration information is obtained, the target storage instance is determined according to the storage key value and the storage configuration information, and the data processing request is sent to the storage device indicated by the target storage instance, so that the storage device executes the corresponding data processing technical means.
Fig. 2 is a schematic diagram of a main flow of a data processing method according to a second embodiment of the present invention; as shown in fig. 2, the data processing method provided in the embodiment of the present invention mainly includes:
step S201, acquiring service requirements, and determining a storage key value according to the service requirements and the data type of the data to be processed.
According to the embodiment of the invention, the importance level of the data can be divided according to the service requirement, and the corresponding storage value of the data type corresponding to the data with higher importance level is set as the fixed prefix. Note that, as long as the effect of distinction can be achieved, a fixed suffix or other form of a specific numerical value may be used.
Through the arrangement, different storage key values are set for the data types of the data to be processed according to the service requirements, matching of the target storage devices for the data processing requests according to the storage key values is facilitated, for example, the data with higher importance levels are stored in a certain specific storage device, and the load balance among the storage devices is facilitated to be optimized while the user requirements are met.
Step S202, responding to a data processing request initiated by a service server, and acquiring a storage instance list; the data processing request comprises a data reading request and/or a data writing request.
Specifically, according to the embodiment of the present invention, the corresponding relationship between each service server and the storage device is recorded in the storage instance list, and the corresponding storage device is matched for the user request according to the corresponding relationship, which is beneficial to improving the data processing efficiency and improving the technical effect of user experience.
Step S203, judging the number of storage instances included in the storage instance list; wherein the storage instance indicates a correspondence between the service server and the storage device. If the storage instance list only includes one storage instance, executing step S204; if the storage instance list includes a plurality of storage instances, step S205 is executed.
Specifically, according to the embodiment of the present invention, if the storage instance list only includes one storage instance, that is, for the service server, the storage instance list only corresponds to one storage device, in such a situation, after receiving the data processing request of the service server, only the data processing request needs to be sent to the unique storage device, and no additional routing call operation is required. If the storage instance list includes a plurality of (greater than or equal to two) storage instances, that is, for the service server, it needs to match the target storage device for the data processing request sent by the service server according to the subsequent storage configuration information, so as to complete data processing, thereby achieving the purposes of improving data processing efficiency and improving user experience.
Step S204, according to the service server number corresponding to the data processing request and the storage instance, determining a corresponding storage device, and sending the data processing request to the storage device.
According to a specific implementation manner of the embodiment of the present invention, the service server is preferentially matched with the storage devices in the same machine room, and a storage instance is generated for the matching relationship and stored in the storage instance list. Through the arrangement, subsequent data management is facilitated.
Step S205, if the storage instance list includes multiple storage instances, determine whether the service server configuration information indicates a target storage instance. If yes, that is, the service server configuration information indicates that a target storage instance exists, executing step S206; if not, that is, the service server configuration information does not indicate the target storage instance, step S207 is executed.
Specifically, the configuration information of the service server mainly refers to configuration information of the same machine room, that is, the service server configures a storage device in the same machine room.
Step S206, the data processing request is sent to the storage device corresponding to the target storage instance.
Step S207, determining whether the stored value of the data to be processed is a specific value according to the stored key value configuration information. If yes, i.e. the stored value of the data to be processed is a specific value, go to step S209; if not, that is, the stored value of the data to be processed is not the specific value, go to step S208.
Specifically, the storage key value configuration information refers to setting the grade of the data to be processed according to the storage key value, where the storage key value may be a key value in a specific interval, or may also be set in the form of a specific prefix or a specific suffix to distinguish the data to be processed in different grades, and then matches the target storage device according to the storage key value to optimize load balance among the storage devices, write the data in the same grade into the same storage device, and facilitate subsequent reading and other operations.
In step S208, the data processing request is sent to the storage device corresponding to the specific numerical value.
Step S209, obtaining the current loads of the plurality of storage devices, and determining whether the storage device with the current load being too high is consistent with the storage devices of the same machine room as the service server. If yes, namely the current storage device with the overhigh load is consistent with the storage device of the service server and the storage device of the machine room, executing step S210; if not, that is, the storage device with the current overload is inconsistent with the storage devices of the service server and the machine room, step S211 is executed.
Specifically, the storage device configuration information is to adjust the corresponding data processing amount according to the current load of the storage device. I.e. a storage device with a smaller load is matched to the data processing request.
Preferably, according to an embodiment of the present invention, the service server configuration information, the storage key configuration information, and the storage device configuration information are storage configuration information, and if the storage instance list includes a plurality of storage instances, the method further includes:
setting matching priority for the configuration information of the business server, the configuration information of the storage key value and the configuration information of the storage device;
and matching the storage key value with the matching priority of the plurality of information in the storage configuration information in sequence to determine a target storage instance.
Through the setting, the priority is set for the plurality of information in the storage configuration information, the routing scheduling of the access request is further perfected, the load balancing is realized, the system risk is reduced, the data processing efficiency is improved, and the user experience is improved.
Step S210, sending the data processing request to a storage device in the same machine room as the service server.
Step S211, determining a target storage device according to the current load capacity, and sending the data processing request to the target storage device.
Further, according to an embodiment of the present invention, the method further includes: the method comprises the steps of obtaining the current load capacity of a plurality of storage devices, determining a target storage device according to the current load capacity, and sending a data processing request to the target storage device.
Through the arrangement, the target storage device is determined according to the current load capacity of the plurality of storage devices, so that multiple time-consuming queries are avoided while load balancing is realized, and the data processing efficiency is improved.
In step S212, the storage device is used to perform corresponding data processing.
Preferably, according to an embodiment of the present invention, the storage apparatus is a distributed cache system cluster, and the method further includes:
and adjusting the distributed cache system corresponding to the data processing request according to the load capacity of a plurality of distributed cache systems in the distributed cache system cluster.
Through the setting, the distributed cache system cluster is adopted, high-concurrency user requests can be responded, and the service capability and the application scene of the data processing method are expanded.
According to the technical scheme of the embodiment of the invention, the storage instance list is obtained by responding to the data processing request initiated by the service server; the data processing request comprises a data reading request and/or a data writing request; judging whether the storage instance list comprises a plurality of storage instances or not; the storage instance indicates the corresponding relation between the service server and the storage device; if the data processing method is adopted, the storage key value of the to-be-processed data corresponding to the data processing request is determined, the storage configuration information is obtained, the target storage instance is determined according to the storage key value and the storage configuration information, and the data processing request is sent to the storage device indicated by the target storage instance, so that the storage device executes the corresponding data processing technical means.
FIG. 3 is a schematic diagram of the main blocks of a data processing apparatus according to an embodiment of the present invention; as shown in fig. 3, a data processing apparatus 300 according to an embodiment of the present invention mainly includes:
an obtaining module 301, configured to obtain a storage instance list in response to a data processing request initiated by a service server; the data processing request comprises a data reading request and/or a data writing request.
Specifically, according to the embodiment of the present invention, a large number of users send data reading requests or data writing requests based on the service servers, and in the face of a high-concurrency user request, a storage instance list is obtained, where the storage instance list records the corresponding relationship between each service server and the storage device, so as to match the corresponding storage device for the user request, which is beneficial to improving data processing efficiency and improving the technical effect of user experience.
A determining module 302, configured to determine whether the storage instance list includes multiple storage instances; wherein the storage instance indicates a correspondence between the service server and the storage device.
Specifically, according to the embodiment of the present invention, if the storage instance list only includes one storage instance, that is, for the service server, the storage instance list only corresponds to one storage device, in such a situation, after receiving the data processing request of the service server, only the data processing request needs to be sent to the unique storage device, and no additional routing call operation is required. If the storage instance list includes a plurality of (greater than or equal to two) storage instances, that is, for the service server, it needs to match the target storage device for the data processing request sent by the service server according to the subsequent storage configuration information, so as to complete data processing, thereby achieving the purposes of improving data processing efficiency and improving user experience.
The data processing module 303, if only one storage instance is included in the storage instance list, is configured to send the data processing request to the storage device indicated by the storage instance, so that the storage device executes corresponding data processing; if the storage instance list comprises a plurality of storage instances, the storage instance list is used for determining a storage key value of the to-be-processed data corresponding to the data processing request, acquiring storage configuration information, determining a target storage instance according to the storage key value and the storage configuration information, and sending the data processing request to the storage device indicated by the target storage instance, so that the storage device executes corresponding data processing.
Specifically, according to an embodiment of the present invention, the data processing module 303 is further configured to:
and determining a corresponding storage device according to the service server number corresponding to the data processing request and the storage instance so as to send the data processing request to the storage device.
According to a specific implementation manner of the embodiment of the present invention, the service server is preferentially matched with the storage devices in the same machine room, and a storage instance is generated for the matching relationship and stored in the storage instance list. Through the arrangement, subsequent data management is facilitated.
Further, according to an embodiment of the present invention, the data processing module 303 is further configured to: and acquiring service requirements, and determining a storage key value according to the service requirements and the data type of the data to be processed.
Through the setting, different storage key values are set for the data types of the data to be processed according to the service requirements, the data processing requests can be matched with the target storage devices according to the storage key values subsequently, and the load balance among the storage devices can be optimized.
Preferably, according to an embodiment of the present invention, the storage configuration information includes at least one of the following information: the configuration information of the business server, the configuration information of the storage key value and the configuration information of the storage device.
The configuration information of the service server mainly refers to configuration information of the same machine room, namely, the configuration information of the service server is a storage device in the same machine room.
The storage key value configuration information refers to setting the grade of the data to be processed according to the storage key value, wherein the storage key value can be a key value in a specific interval, and can also be set in the form of a specific prefix and a specific suffix to distinguish the data to be processed in different grades, so that the target storage devices are matched according to the storage key value to optimize load balance among the storage devices, the data in the same grade is written into the same storage device, and subsequent reading and other operations are facilitated.
The storage device configuration information is used for adjusting the corresponding data processing amount according to the current load of the storage device. I.e. a storage device with a smaller load is matched to the data processing request.
Further, according to an embodiment of the present invention, if the storage instance list includes a plurality of storage instances, the data processing module 303 is further configured to:
setting matching priority for the configuration information of the business server, the configuration information of the storage key value and the configuration information of the storage device;
and matching the storage key value with the matching priority of the plurality of information in the storage configuration information in sequence to determine a target storage instance.
Through the setting, the priority is set for the plurality of information in the storage configuration information, the routing scheduling of the access request is further perfected, the load balancing is realized, the system risk is reduced, the data processing efficiency is improved, and the user experience is improved.
Illustratively, according to an embodiment of the present invention, the data processing apparatus 300 further includes an adjusting module, configured to: the method comprises the steps of obtaining the current load capacity of a plurality of storage devices, determining a target storage device according to the current load capacity, and sending a data processing request to the target storage device.
Through the arrangement, the target storage device is determined according to the current load capacity of the plurality of storage devices, load balance is achieved, the risk of downtime of the storage devices is avoided, multiple time-consuming queries are avoided, and data processing efficiency is improved.
Preferably, according to an embodiment of the present invention, the storage apparatus is a distributed cache system cluster, and the adjusting module is further configured to:
and adjusting the distributed cache system corresponding to the data processing request according to the load capacity of a plurality of distributed cache systems in the distributed cache system cluster.
Through the setting, the distributed cache system cluster is adopted, high-concurrency user requests can be responded, and the service capability and the application scene of the data processing method are expanded.
According to the technical scheme of the embodiment of the invention, the storage instance list is obtained by responding to the data processing request initiated by the service server; the data processing request comprises a data reading request and/or a data writing request; judging whether the storage instance list comprises a plurality of storage instances or not; the storage instance indicates the corresponding relation between the service server and the storage device; if the data processing method is adopted, the storage key value of the to-be-processed data corresponding to the data processing request is determined, the storage configuration information is obtained, the target storage instance is determined according to the storage key value and the storage configuration information, and the data processing request is sent to the storage device indicated by the target storage instance, so that the storage device executes the corresponding data processing technical means.
Fig. 4 shows an exemplary system architecture 400 of a data processing method or data processing apparatus to which embodiments of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405 (this architecture is merely an example, and the components included in a particular architecture may be adapted according to application specific circumstances). The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have installed thereon various communication client applications, such as a data processing type application, a shopping type application, a web browser application, a search type application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only).
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server that provides various services, such as a server (for example only) for users to (perform data processing with) the terminal devices 401, 402, 403. The server may analyze and/or otherwise process the received data, such as the data processing request, and feed back the processing result (e.g., the target storage instance — just an example) to the terminal device.
It should be noted that the data processing method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the data processing apparatus is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, a block diagram of a computer system 500 suitable for use with a terminal device or server implementing an embodiment of the invention is shown. The terminal device or the server shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprises an acquisition module, a judgment module and a data processing module. The names of these modules do not in some cases form a limitation on the modules themselves, and for example, the obtaining module may also be described as a "module for obtaining a list of stored instances in response to a data processing request initiated by a service server".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: responding to a data processing request initiated by a service server, and acquiring a storage instance list; the data processing request comprises a data reading request and/or a data writing request; judging whether the storage instance list comprises a plurality of storage instances or not; the storage instance indicates the corresponding relation between the service server and the storage device; if the data processing request is not received, determining a storage key value of the to-be-processed data corresponding to the data processing request, acquiring storage configuration information, determining a target storage instance according to the storage key value and the storage configuration information, and sending the data processing request to a storage device indicated by the target storage instance, so that the storage device executes corresponding data processing.
According to the technical scheme of the embodiment of the invention, the storage instance list is obtained by responding to the data processing request initiated by the service server; the data processing request comprises a data reading request and/or a data writing request; judging whether the storage instance list comprises a plurality of storage instances or not; the storage instance indicates the corresponding relation between the service server and the storage device; if the data processing method is adopted, the storage key value of the to-be-processed data corresponding to the data processing request is determined, the storage configuration information is obtained, the target storage instance is determined according to the storage key value and the storage configuration information, and the data processing request is sent to the storage device indicated by the target storage instance, so that the storage device executes the corresponding data processing technical means.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110153654.8A CN113760179A (en) | 2021-02-04 | 2021-02-04 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110153654.8A CN113760179A (en) | 2021-02-04 | 2021-02-04 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113760179A true CN113760179A (en) | 2021-12-07 |
Family
ID=78786541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110153654.8A Pending CN113760179A (en) | 2021-02-04 | 2021-02-04 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760179A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116303608A (en) * | 2023-03-16 | 2023-06-23 | 中国建设银行股份有限公司 | Data processing method and device for application service |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102289508A (en) * | 2011-08-31 | 2011-12-21 | 上海西本网络科技有限公司 | Distributed cache array and data inquiry method thereof |
| US20160085696A1 (en) * | 2014-09-24 | 2016-03-24 | International Business Machines Corporation | Using access information to determine a storage tier for storing data |
| US9870168B1 (en) * | 2014-12-22 | 2018-01-16 | Emc Corporation | Key-value store with internal key-value storage interface |
| CN109032511A (en) * | 2018-07-09 | 2018-12-18 | 武汉斗鱼网络科技有限公司 | Data storage method, server and storage medium |
| CN109032502A (en) * | 2018-06-13 | 2018-12-18 | 广州市信景技术有限公司 | A kind of distributed data cluster storage system |
| CN109388508A (en) * | 2018-10-10 | 2019-02-26 | 杭州安恒信息技术股份有限公司 | A kind of Embedded cache shared system and method for realizing zero dependence |
| CN109714430A (en) * | 2019-01-16 | 2019-05-03 | 深圳壹账通智能科技有限公司 | Distributed caching method, device, computer system and storage medium |
| CN110545313A (en) * | 2019-08-13 | 2019-12-06 | 北京字节跳动网络技术有限公司 | message push control method and device and electronic equipment |
| US20200319909A1 (en) * | 2019-04-04 | 2020-10-08 | Nutanix, Inc. | System and method for a distributed key-value store |
-
2021
- 2021-02-04 CN CN202110153654.8A patent/CN113760179A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102289508A (en) * | 2011-08-31 | 2011-12-21 | 上海西本网络科技有限公司 | Distributed cache array and data inquiry method thereof |
| US20160085696A1 (en) * | 2014-09-24 | 2016-03-24 | International Business Machines Corporation | Using access information to determine a storage tier for storing data |
| US9870168B1 (en) * | 2014-12-22 | 2018-01-16 | Emc Corporation | Key-value store with internal key-value storage interface |
| CN109032502A (en) * | 2018-06-13 | 2018-12-18 | 广州市信景技术有限公司 | A kind of distributed data cluster storage system |
| CN109032511A (en) * | 2018-07-09 | 2018-12-18 | 武汉斗鱼网络科技有限公司 | Data storage method, server and storage medium |
| CN109388508A (en) * | 2018-10-10 | 2019-02-26 | 杭州安恒信息技术股份有限公司 | A kind of Embedded cache shared system and method for realizing zero dependence |
| CN109714430A (en) * | 2019-01-16 | 2019-05-03 | 深圳壹账通智能科技有限公司 | Distributed caching method, device, computer system and storage medium |
| US20200319909A1 (en) * | 2019-04-04 | 2020-10-08 | Nutanix, Inc. | System and method for a distributed key-value store |
| CN110545313A (en) * | 2019-08-13 | 2019-12-06 | 北京字节跳动网络技术有限公司 | message push control method and device and electronic equipment |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116303608A (en) * | 2023-03-16 | 2023-06-23 | 中国建设银行股份有限公司 | Data processing method and device for application service |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112597126B (en) | Data migration method and device | |
| CN113312553B (en) | User tag determining method and device | |
| CN113742389A (en) | Service processing method and device | |
| CN110162410B (en) | A message processing method and device | |
| CN110321252B (en) | Skill service resource scheduling method and device | |
| CN112398669B (en) | Hadoop deployment method and device | |
| CN110650209A (en) | Method and device for realizing load balance | |
| CN115629853A (en) | Task scheduling method and device | |
| CN113407108A (en) | A data storage method and system | |
| CN112667368A (en) | Task data processing method and device | |
| CN113760179A (en) | Data processing method and device | |
| CN114756173B (en) | Method, system, apparatus and computer readable medium for file merging | |
| CN109213815B (en) | Method, device, server terminal and readable medium for controlling execution times | |
| CN117333170A (en) | Service processing method, device, electronic equipment and storage medium | |
| CN113127416A (en) | Data query method and device | |
| CN113760965B (en) | Data query method and device | |
| CN113746661B (en) | A business processing method and device | |
| CN113448940B (en) | Database capacity expansion method and device | |
| CN112711572B (en) | Online capacity expansion method and device suitable for database and table division | |
| CN113556370B (en) | Service calling method and device | |
| CN114090524A (en) | A kind of Excel file distributed export method and device | |
| CN113110898A (en) | Transaction task execution method and device | |
| CN113722548A (en) | Method and device for processing reference relationship in business system | |
| CN113127504A (en) | Business data display method and device | |
| CN113760177A (en) | Method and device for reporting data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |