CN116303132A - Data caching method, device, equipment and storage medium - Google Patents
Data caching method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116303132A CN116303132A CN202310246672.XA CN202310246672A CN116303132A CN 116303132 A CN116303132 A CN 116303132A CN 202310246672 A CN202310246672 A CN 202310246672A CN 116303132 A CN116303132 A CN 116303132A
- Authority
- CN
- China
- Prior art keywords
- server
- data
- data processing
- disk
- processing task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
- G06F12/0871—Allocation or management of cache space
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data caching method, a data caching device, data caching equipment and a storage medium. The method comprises the following steps: responding to a data processing task, determining a server cluster for executing the data processing task, and controlling a management server in the server cluster to interact with a computing server to execute the data processing task; in the process of executing the data processing task, monitoring the IO rate of reading and writing of the server cluster disk; if the IO rate is detected to exceed the preset rate threshold, adopting different caching strategies to respectively cache the intermediate data generated by the resource manager and the disk data generated by the data processing task. According to the technical scheme, different data can be cached in a targeted manner, the cluster pressure of the server is relieved, and the data processing efficiency is improved.
Description
Technical Field
The present invention relates to the field of big data, and in particular, to a data caching method, apparatus, device, and storage medium.
Background
With the rapid growth of the internet and the wide application of 4G and 5G technologies, various structured and unstructured data are continually being generated. The processing of data by a resource manager has been widely used in different fields, and when the data size is large or the data processing amount is large, a large pressure is caused to the CPU and disk IO of the server cluster.
How to perform targeted cache processing on related data generated in the data processing process of a resource manager, so as to relieve the cluster pressure of a server and improve the data processing efficiency is a problem to be solved urgently at present.
Disclosure of Invention
The invention provides a data caching method, a device, equipment and a storage medium, which are used for carrying out targeted caching processing on different data, relieving the cluster pressure of a server and improving the data processing efficiency.
According to an aspect of the present invention, there is provided a data caching method, performed by a resource manager, comprising:
responding to a data processing task, determining a server cluster for executing the data processing task, and controlling a management server in the server cluster to interact with a computing server to execute the data processing task;
in the process of executing the data processing task, monitoring the IO rate of reading and writing of the server cluster disk;
if the IO rate is detected to exceed the preset rate threshold, adopting different caching strategies to respectively cache the intermediate data generated by the resource manager and the disk data generated by the data processing task.
Optionally, different caching policies are adopted to respectively cache the intermediate data generated by the resource manager and the disk data generated by the data processing task, including:
analyzing the residual memory of the server cluster, and caching intermediate data generated by the resource manager according to the analysis result;
determining a data storage configuration file corresponding to disk data generated by a data processing task, and modifying the data storage configuration file to obtain a storage address of the updated disk data;
and carrying out cache processing on the disk data generated by the data processing task according to the updated storage address of the disk data.
According to the technical scheme, the residual memory of the server cluster is analyzed, so that the intermediate data generated by the resource manager is cached according to an analysis result, one implementation mode for the intermediate data caching strategy is provided, the disk data can be stored in a new storage address by determining the data storage configuration file corresponding to the disk data and modifying the data storage configuration file, one implementation mode for the disk data caching strategy is provided, and the intermediate data and the disk data are cached respectively in different modes, so that the data processing efficiency can be ensured, and the overall performance of the server cluster is improved.
Optionally, analyzing the remaining memory of the server cluster, and caching the intermediate data generated by the resource manager according to the analysis result, including:
determining whether the residual memory of the server cluster is larger than a preset memory threshold;
if yes, based on a preset resource manager computing unit, the remaining memory is allocated, the remaining memory is mounted on a dedicated disk corresponding to a management server, and intermediate data generated by the resource manager are cached to the dedicated disk corresponding to the management server.
According to the technical scheme, the relation between the residual memory of the server cluster and the preset memory threshold is determined, and the residual memory is used as a cache point of the intermediate data generated by the resource manager under the condition that the residual memory is enough, so that the residual memory of the server cluster can be effectively utilized, the processing efficiency of the intermediate data generated by the resource manager is improved, and the overall performance of the server cluster is ensured.
Optionally, before adopting different caching strategies to respectively cache the intermediate data generated by the resource manager and the disk data generated by the data processing task, the method further includes:
the control management server sends an adjustment instruction to each computing server to instruct the computing server to make the data disc into a single disc based on a pre-configured cache card and start the cache function of the cache card;
The control management server sends operation instructions to each computing server to instruct the computing servers to perform disk optimization operation; the disk optimization operation includes at least one of: modifying IO scheduling algorithm, setting disk pre-reading buffer and writing dirty data back to disk.
According to the technical scheme, the management server is controlled to send the adjustment instruction to the calculation server, so that the calculation server can make the data disc into a single disc and start the caching function of the caching card, the disc reading and writing capacity is effectively improved, the management server is controlled to send the operation instruction to the calculation server, the calculation server can perform relevant disc optimization operation, disc performance optimization is performed, and the processing efficiency of subsequent data processing tasks is guaranteed.
Optionally, the control management server sends an operation instruction to each computing server to instruct the computing server to perform disk optimization operation, including:
the control management server sends operation instructions to each computing server, and the proportion of dirty data generated by each computing server process to the system memory and the residence time of the dirty data in the system memory are monitored;
and determining the proportion and the residence time length, and if the proportion is larger than a preset proportion threshold value or the residence time length is larger than a preset time length threshold value, controlling the management server to call a preset process to write dirty data back to the disk.
According to the technical scheme, the management server is controlled to send the operation instruction to each computing server, the proportion of dirty data of each computing server process to the system memory is monitored, the residence time of the dirty data in the system memory is analyzed, when the dirty data reaches the condition of writing back a disk, the operation of writing back the disk is executed, an implementation mode for performing disk optimization operation is provided, the disk of the computing server can be comprehensively and effectively optimized in time, and the performance of the server is improved.
Optionally, in response to a data processing task, determining a server cluster that performs the data processing task includes:
responding to the data processing task, and screening a target server from the candidate servers according to the configuration information of the candidate servers and the task information of the data processing task associated with the resource manager;
and determining a management server and a computing server in the target server, and generating the server cluster.
According to the technical scheme, the configuration information of the candidate servers and the task information of the data processing task are analyzed, a group of servers which are most suitable for executing the data processing task, namely the target servers, can be determined from the candidate servers, the roles of the target servers are further determined, an executable mode for generating the server cluster for executing the data processing task is provided, and the execution efficiency of the task can be improved.
Optionally, controlling the management server in the server cluster to interact with the computing server, and before executing the data processing task, further includes:
and sending a preset function switch instruction to a management server and a computing server in the server cluster to instruct the management server and the computing server to start or close corresponding options in a BIOS system of the management server and the computing server so as to optimize the performance of the server cluster.
According to the technical scheme, before the data processing task is executed, the control management server and the calculation server close options irrelevant to the data processing task or open options capable of improving the overall performance of the cluster, so that the overall performance of the server cluster can be effectively improved, and efficient execution of the subsequent data processing task is ensured.
According to another aspect of the present invention, there is provided a data caching apparatus configured in a resource manager, including:
the execution module is used for responding to the data processing task, determining a server cluster for executing the data processing task, controlling a management server in the server cluster to interact with a calculation server, and executing the data processing task;
the monitoring module is used for monitoring the IO rate of the read-write of the server cluster disk in the process of executing the data processing task;
And the cache module is used for adopting different cache strategies to respectively cache the intermediate data generated by the resource manager and the disk data generated by the data processing task if the IO rate exceeds the preset rate threshold.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data caching method of any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a data caching method according to any embodiment of the present invention.
According to the technical scheme, a server cluster for executing the data processing task is determined in response to the data processing task, a management server in the server cluster is controlled to interact with a computing server to execute the data processing task, the IO rate of the disk reading and writing of the server cluster is monitored in the process of executing the data processing task, and if the IO rate is detected to exceed a preset rate threshold, different caching strategies are adopted to respectively cache intermediate data generated by a resource manager and disk data generated by the data processing task. The IO rate of the read-write of the server cluster disk is monitored, and the cache strategy is adjusted when the IO rate exceeds the rate threshold, so that real-time management of the server cluster disk IO can be realized, effective execution of data processing tasks is ensured, and the intermediate data generated by the resource manager and disk data generated by the data processing tasks are respectively cached by adopting different cache strategies, so that the mutual influence of the two data can be avoided, the server cluster pressure is relieved, and the data processing efficiency is improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1A is a flowchart of a data caching method according to a first embodiment of the present invention;
fig. 1B is a schematic diagram of MapReduce data processing according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a data caching method according to a second embodiment of the present invention;
FIG. 3 is a flowchart of a data caching method according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data buffering device according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device implementing a data caching method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," "target," "candidate," "alternative," and the like in the description and claims of the invention and in the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1A is a flowchart of a data caching method according to a first embodiment of the present invention. Fig. 1B is a schematic diagram of MapReduce data processing according to a first embodiment of the present invention. The embodiment is applicable to the situation that different data are buffered in a targeted manner in the process of executing a data processing task, and the method can be executed by a data buffering device, wherein the data buffering device can be implemented in a form of hardware and/or software, and the data buffering device can be configured in an electronic device and executed by a resource manager, and the resource manager can be a Yarn resource manager, for example. As shown in fig. 1A, the data caching method includes:
s101, responding to the data processing task, determining a server cluster for executing the data processing task, and controlling a management server in the server cluster to interact with a computing server to execute the data processing task.
The data processing task refers to a data processing task that needs to be executed by the resource management server based on a preset distributed framework or a computing model, for example, may be a data processing task executed on the Yarn resource manager based on a MapReduce distributed computing framework. The server cluster refers to a server cluster corresponding to a data processing task, and the server cluster includes at least two servers with different roles, for example, a management server and a computing server. The management server is a server with a Master node (Master node) in a server cluster, and is used for managing data of a corresponding computing server, distributing tasks to the computing server, scheduling tasks and the like. The computing server is a server in a server cluster, which has a Slave node (Slave node) and is used for receiving tasks issued by the management server, storing and computing.
Optionally, the resource manager may call a preset Test tool, such as a Test DFSIO Test tool, to generate a data processing task when receiving a Test instruction of a related person, that is, detect the data processing task; and the data processing task request sent by the relevant client can be directly received to generate a corresponding data processing task.
Optionally, after detecting that the data processing task is generated, the resource manager may screen the managed server cluster in response to the data processing task to determine a server cluster that performs the data processing task, and specifically, in response to the data processing task, determine a server cluster that performs the data processing task, including: responding to the data processing task, and screening a target server from the candidate servers according to the configuration information of the candidate servers and the task information of the data processing task associated with the resource manager; and determining a management server and a computing server in the target server, and generating a server cluster.
The candidate servers refer to all servers managed by the resource manager. The configuration information may include CPU (central processing unit ) model number, memory size, disk size, data disk, network card, and cache card related attributes of each candidate server. The task information of the data processing task may be a data size, copy information, a data block size, and the like of the data to be processed associated with the data processing task.
Alternatively, the configuration information of the candidate servers associated with the resource manager and the task information of the data processing task may be directly input into a pre-trained model, and the optimal combination of the candidate servers executing the data processing task may be output, so as to determine the target server.
Optionally, according to configuration information of each target server, based on a preset screening rule, a management server with better CPU comprehensive performance and a calculation server with better calculation performance can be screened out from the target servers, that is, the management server and the calculation server in the target servers are determined, so that a server cluster is generated.
According to the technical scheme, the configuration information of the candidate servers and the task information of the data processing task are analyzed, a group of servers which are most suitable for executing the data processing task, namely the target servers, can be determined from the candidate servers, the roles of the target servers are further determined, an executable mode for generating the server cluster for executing the data processing task is provided, and the execution efficiency of the task can be improved.
Optionally, after determining the server cluster for executing the data processing task, the data processing task may be sent to a management server in the server cluster, so as to instruct the management server to execute a preset scheduler, distribute the data processing task to each computing server, and perform parallel processing, that is, control the management server in the server cluster to interact with the computing servers, and execute the data processing task.
For example, if the data processing task is executed on the Yarn resource manager based on the MapReduce distributed computing framework, the resource manager may control the management server to allocate resources to the computing server based on the allocation situation of the tasks such as Map, shuffle, reduce and the like associated with the MapReduce data processing task by using the preset Container computing unit format as the minimum allocation unit when allocating computing resources, so as to instruct the computing server to perform concurrent computing processing of the tasks such as Map, shuffle, reduce and the like.
Optionally, the data associated with the data processing task may be analyzed to determine a format of a content computing unit (expressed as content < vcores, vmem >), and specifically, if the data size associated with the data processing task is smaller than a preset size, or the data concurrency corresponding to the data processing task is smaller than a preset concurrency, vcore and vmem may be set to be larger, so as to ensure that the data processing task may be completed, for example content <2,4G >; if the data size associated with the data processing task is larger, or the data concurrency corresponding to the data processing task is larger, vcares and vmem need to be set smaller, for example, content <1,2>, so that the data processing efficiency is improved. Where vcores is the number of CPU cores configured in the Container and vmem is the memory size configured in the Container.
For example, referring to fig. 1B, if the data processing task is performed on the Yarn resource manager based on the MapReduce distributed computing framework, the Yarn resource manager may control the management server to execute the scheduler in response to the data processing task sent by the client 1, the client 2, or the client 3, and interact with the computing server 1, the computing server 2, and the computing server 3 to instruct each computing server to execute the Map task and the Reduce task. The Map task is used for reading and writing each data file, and the Reduce task is used for accumulating statistical information and generating statistical summary.
Optionally, after determining the server cluster, the management server in the server cluster is controlled to interact with the computing server, and before executing the data processing task, the network of the server cluster may be optimized, specifically, a bond may be made through multiple network cards, and a CPU of the multiple NUMA architecture may be made to perform network card interrupt binding, so as to increase the network bandwidth of the server cluster, and network optimization of the server cluster may be performed, for example, mode0 and mode6 may be selected, so as to implement increase of the network card bandwidth, and mode0 and mode6 may be simply understood as aggregate superposition of multiple network card bandwidths, and network bandwidth bottlenecks may be effectively avoided.
Optionally, after determining the server cluster, computing resources of the server cluster may be further optimized, specifically, before controlling interaction between the management server and the computing server in the server cluster, and before executing the data processing task, the method further includes: and sending a preset function switch instruction to a management server and a computing server in the server cluster to instruct the management server and the computing server to start or close corresponding options in a BIOS system of the management server and the computing server so as to optimize the performance of the server cluster.
The BIOS system (Basic Input Output System, basic input/output system) is a basic system configured in the management server and the computing server.
Optionally, the management server may send a preset function switch instruction to the management server and the computing server in the server cluster, so as to instruct the management server and the computing server to open at least one of the following options in the BIOS system thereof: hyper-threading, locking memory frequency to highest, and opening memory prefetching.
Wherein, by turning on the hyper-threading, doubling of CPU processing cores (threads) can be realized, for example, one CPU has 32 physical cores, and the processing cores (threads) available for turning on the hyper-threading become 64. In addition, by opening the options of over-frequency, locking the memory frequency to the highest, opening memory prefetching and the like, the CPU core of the server can work in the maximum performance mode, and the performance optimization of the server cluster is realized.
Optionally, the management server may send a preset function switch instruction to the management server and the computing server in the server cluster, so as to instruct the management server and the computing server to close functional options unrelated to the data processing task in their own BIOS system, for example, BIOS operation functions such as virtualization, SR-IOV, IOMMU and the like are options related to cloud computing, when big data is not used, the big data processing performance is reduced when the management server is opened, so that corresponding functional options can be closed in the BIOS, and performance optimization of the server cluster is achieved.
According to the technical scheme, before the data processing task is executed, the control management server and the calculation server close options irrelevant to the data processing task or open options capable of improving the overall performance of the cluster, and the overall performance of the server cluster can be effectively improved through pre-configuration before the data task is processed, so that the follow-up more efficient completion of the data processing task is facilitated.
S102, monitoring IO rate of reading and writing of the server cluster disk in the process of executing the data processing task.
The IO (input output) rate of disk reading and writing of the server cluster refers to a rate capable of representing the overall disk reading and writing state of the server cluster, and the IO rate of disk reading and writing of the server cluster can be an average IO rate of disk reading and writing of all servers in the server cluster or an IO rate of maximum disk reading and writing of the servers in the server cluster.
Optionally, the resource manager may monitor the read-write status of the disk through a preset IO performance statistics tool, such as nmon, iostat, etc.
It should be noted that, during the task execution process, the management server and the computing server interact to generate intermediate data, where the intermediate data is stored in the yarn. In addition, during the task execution process, there are multiple subtasks queuing to send a request to the CPU to obtain the Container state, where the CPU is in a high utilization state. In addition, sub-tasks associated with the data processing task, such as Map task, shuffle task and Reduce task, also generate a lot of disk data, and generate a lot of read-write operations on the disk, so that the read-write state of the disk needs to be monitored, timely optimization is performed, and efficient execution of the data processing task is ensured.
And S103, if the IO rate is detected to exceed the preset rate threshold, adopting different caching strategies to respectively cache the intermediate data generated by the resource manager and the disk data generated by the data processing task.
The speed threshold value refers to a preset threshold value representing the read-write pressure of the server cluster disk. A typical data disk is a normal SATA disk (hard disk), and the read/write performance is 250MB/s, so the rate threshold may be set to 250MB/s. The caching policy refers to a policy for temporarily storing disk data or intermediate data, and may include, for example, a target storage address of the data. The intermediate data may be, for example, a Yarn intermediate data file cache (file cache intermediate data) and a user cache (user cache intermediate data), belonging to temporary intermediate data. The disk data may be HDFS (Hadoop Distributed File System, distributed file system) data, for example. The disk data may be data to be processed by a data processing task (such as Map Reduce task), or may be data generated after the data processing task processes, and stored in an HDFS file system.
It should be noted that, in the process of executing the data processing task, the intermediate data generated by the resource manager and the disk data generated by the data processing task are generally stored in the same location by default, that is, under the yarn. According to the method, when the data processing task is a high concurrency task, the generated disk IO consumption and the intermediate data generated by the resource manager can react to the disk IO consumption, and when the IO rate is detected to exceed the preset rate threshold, different cache strategies are adopted for different data, so that the data processing efficiency can be effectively improved, and the accurate and effective execution of the data processing task is ensured.
Optionally, if a preset IO performance statistics tool is adopted, it is monitored that the IO rate exceeds a preset rate threshold, it can be determined that the data caching mode needs to be modified, a corresponding caching strategy can be determined according to the type of the data, and targeted caching is performed on the data based on the caching strategy.
Optionally, after the completion of the execution of the data processing task is detected, the CPU utilization condition of the server cluster, the CPU state at the time of disk reading and the CPU state at the time of disk writing in the task processing process may be obtained, so as to analyze and determine whether the time when the disk reading and writing can be greatly performed at the optimal state of the CPU (i.e. the time when the CPU utilization is lower), and if so, indicate that the server cluster better completes the data processing task.
According to the technical scheme, a server cluster for executing the data processing task is determined in response to the data processing task, a management server in the server cluster is controlled to interact with a computing server to execute the data processing task, the IO rate of the disk reading and writing of the server cluster is monitored in the process of executing the data processing task, and if the IO rate is detected to exceed a preset rate threshold, different caching strategies are adopted to respectively cache intermediate data generated by a resource manager and disk data generated by the data processing task. The IO rate of the read-write of the server cluster disk is monitored, and when the IO rate exceeds the rate threshold, the cache strategy is adjusted, so that real-time management of the server cluster disk IO can be realized, effective execution of data processing tasks is ensured, and by adopting different cache strategies, cache processing is respectively carried out on intermediate data generated by a resource manager and disk data generated by the data processing tasks, so that the mutual influence of the two data can be avoided, the server cluster pressure is relieved, the execution efficiency of the data processing tasks is optimized, and the data processing performance is improved.
Optionally, if the IO rate is detected to exceed the preset rate threshold, before buffering the intermediate data and the disk data, starting a buffering function of the computing server, and instructing each computing server to perform a disk optimization operation, and specifically, before buffering the intermediate data generated by the resource manager and the disk data generated by the data processing task respectively by adopting different buffering strategies, further including: the control management server sends an adjustment instruction to each computing server to instruct the computing server to make the data disc into a single disc based on a pre-configured cache card and start the cache function of the cache card; the control management server sends operation instructions to each computing server to instruct the computing servers to perform disk optimization operation; the disk optimization operations include at least one of: modifying IO scheduling algorithm, setting disk pre-reading buffer and writing dirty data back to disk.
Optionally, the management server may be controlled to send an adjustment instruction to each computing server, so as to instruct the computing server to control the interactive interface through the Raid card (i.e. the cache card) installed in the server, or set in the operating system by adopting a software tool corresponding to the Raid card, so as to implement the functions of making the data disc into a single disc and starting the cache of the cache card, and increase the disk reading and writing capability.
By way of example, the instruction instructing the computing server to modify the IO scheduling algorithm may be, for example, "echo bfq >/sys/block/sd$i/queue/schedule", by which the IO scheduling algorithm may be adjusted, for example, readline (deadline scheduler), bfg (budget fair queuing), NOOP (elevator scheduler), etc., sd$i referring to the disk drive of the corresponding computing server, such as sdb, sdc, sdd, etc.
By way of example, the instruction instructing the compute server to set the disk read-ahead cache may be, for example, "echo 8196 >/sys/block/sd$i/queue/read_head_kb", by which the disk read-ahead cache may be set, thereby improving the performance of sequential reads during disk read-ahead.
According to the technical scheme, the management server is controlled to send the adjustment instruction to the calculation server, so that the calculation server can make the data disc into a single disc and start the caching function of the caching card, the disc reading and writing capacity is effectively improved, the management server is controlled to send the operation instruction to the calculation server, the calculation server can perform relevant disc optimization operation, disc performance optimization is performed, and the processing efficiency of subsequent data processing tasks is guaranteed.
Optionally, the control management server sends an operation instruction to each computing server to instruct the computing server to perform disk optimization operation, including: the control management server sends operation instructions to each computing server, and the proportion of dirty data generated by each computing server process to the system memory and the residence time of the dirty data in the system memory are monitored; determining the proportion and the residence time length, and if the proportion is larger than a preset proportion threshold value or the residence time length is larger than a preset time length threshold value, controlling the management server to call a preset process to write dirty data back to the disk. The preset process may be, for example, a pdgroush process. Dirty data (Dirty Read) refers to data that is not within a given range or meaningless in the source system.
Optionally, the management server may be controlled to send an operation instruction "echo 100>/proc/sys/vm/dirty_ratio" to each computing server, monitor the proportion of the dirty data generated by each computing server process to the system memory, and if the proportion is greater than a preset proportion threshold, for example 40%, may control the management server to call the preset process to write the dirty data back to the disk.
Optionally, the management server may be controlled to send an operation instruction "echo 100 >/proc/sys/vm/dirty_expire_centrisecs" to each computing server, monitor a residence time period of dirty data generated by each computing server process in a system memory, and if the residence time period is greater than a preset time period threshold, for example, 3000ms, may control the management server to call the preset process to write the dirty data back to the disk.
According to the technical scheme, the management server is controlled to send the operation instruction to each computing server, the proportion of dirty data of each computing server process to the system memory is monitored, the residence time of the dirty data in the system memory is analyzed, when the dirty data reaches the condition of writing back a disk, the operation of writing back the disk is executed, an implementation mode for performing disk optimization operation is provided, the disk of the computing server can be comprehensively and effectively optimized in time, and the performance of the server is improved.
Example two
FIG. 2 is a flowchart of a data caching method according to a second embodiment of the present invention; the embodiment is optimized and improved based on the technical schemes.
Further, "adopting different caching strategies" respectively carrying out caching treatment on intermediate data generated by the resource manager and disk data generated by the data processing task "is thinned into" carrying out analysis on the residual memory of the server cluster, and carrying out caching treatment on the intermediate data generated by the resource manager according to an analysis result; determining a data storage configuration file corresponding to the disk data generated by the data processing task, and modifying the data storage configuration file to obtain a storage address of the updated disk data; and carrying out buffer processing on the disk data generated by the data processing task according to the updated storage address of the disk data so as to perfect a mode of adopting different buffer strategies to carry out buffer processing on the intermediate data and the disk data.
Further, analyzing the residual memory of the server cluster, and carrying out caching treatment on intermediate data generated by the resource manager according to the analysis result to determine whether the residual memory of the server cluster is larger than a preset memory threshold; if yes, based on a preset resource manager computing unit, the remaining memory is allocated, the remaining memory is mounted on a dedicated disk corresponding to a management server, and intermediate data generated by the resource manager is cached to the dedicated disk corresponding to the management server so as to perfect a caching processing mode of the intermediate data generated by the resource manager.
As shown in fig. 2, the method comprises the following specific steps:
and S201, responding to the data processing task, determining a server cluster for executing the data processing task, and controlling a management server in the server cluster to interact with the computing server to execute the data processing task.
S202, monitoring IO rate of reading and writing of the server cluster disk in the process of executing the data processing task.
And S203, if the IO rate is detected to exceed the preset rate threshold, analyzing the residual memory of the server cluster, and caching the intermediate data generated by the resource manager according to the analysis result.
The remaining memory of the server cluster refers to the total amount of the remaining memory of the server cluster in the whole data processing process. The analysis result may refer to, for example, an analysis result of analyzing whether the remaining memory is greater than a preset memory threshold.
Optionally, if the IO rate is detected to exceed the preset rate threshold, it may be first determined whether the server cluster has enough remaining memory for storing the intermediate data generated by the resource manager, and if not, a cache disk with high read/write and high IOPS (Input/Output Operations Per Second, number of read/write operations per second) may be added, for example, an external storage medium in PCIE (peripheral component interconnect express) form is added, or a high-performance disk is used as a cache medium for caching the intermediate data generated by the resource manager.
Optionally, the analyzing the remaining memory of the server cluster and performing cache processing on the intermediate data generated by the resource manager according to the analysis result includes: determining whether the residual memory of the server cluster is larger than a preset memory threshold; if yes, based on a preset resource manager computing unit, the remaining memory is allocated, the remaining memory is mounted on a dedicated disk corresponding to the management server, and intermediate data generated by the resource manager are cached to the dedicated disk corresponding to the management server.
The preset resource manager calculating unit may be, for example, a calculating unit of a Yarn resource manager, that is, a calculating unit of a content < vcores, vmem >, where content is a calculating unit when Yarn is used, vcores is the number of CPU cores configured in content, and vmem is the memory size configured in content.
The resource manager may allocate the remaining memory by using the Container < vcores, vmem > as a computing unit, that is, make full use of the remaining memory as an intermediate data storage point, specifically, may mount the remaining memory as a TMFS file system, as a cache medium, as a dedicated disk corresponding to the management server, that is, a disk corresponding to the location of the yarn. The TMFS file system is a temporary file system based on a memory.
According to the technical scheme, the relation between the residual memory of the server cluster and the preset memory threshold is determined, and the residual memory is used as a cache point of the intermediate data generated by the resource manager under the condition that the residual memory is enough, so that the residual memory of the server cluster can be effectively utilized, the processing efficiency of the intermediate data generated by the resource manager is improved, and the overall performance of the server cluster is ensured.
S204, determining a data storage configuration file corresponding to the disk data generated by the data processing task, and modifying the data storage configuration file to obtain the storage address of the updated disk data.
The data storage configuration file corresponding to the disk data may be, for example, an hdfs-core.xml configuration file, which is used to configure a storage path of the disk data. The data storage profile of the intermediate data may be, for example, a yarn-core.xml profile for configuring the storage path of the intermediate data.
Optionally, the data storage configuration file corresponding to the disk data may be modified based on a preset rule, a storage path of the disk data is modified from a default yarn.
S205, caching the disk data generated by the data processing task according to the updated storage address of the disk data.
Optionally, in the process of executing the data processing task, the disk data generated for the data processing task can be stored to the dedicated disk under the corresponding computing server based on the storage address of the updated disk data, so that the storage of the disk data to the same position as the intermediate data is avoided, and the interaction possibly caused in the two data generating processes is eliminated.
According to the scheme of the embodiment of the invention, the rest memory of the server cluster is analyzed, so that the intermediate data generated by the resource manager is cached according to an analysis result, one implementation mode aiming at the intermediate data caching strategy is provided, the storage of the disk data can be stored at a new storage address by determining and modifying the data storage configuration file corresponding to the disk data, one implementation mode aiming at the disk data caching strategy is provided, and the efficiency of data processing can be ensured and the integral performance of the server cluster is improved by respectively caching the intermediate data and the disk data in different modes.
Example III
Fig. 3 is a flowchart of a data caching method according to a third embodiment of the present invention. The present embodiment provides a preferred example on the basis of the above-described embodiment.
As shown in fig. 3, the method comprises the following specific steps:
s301, responding to the data processing task, and screening a target server from the candidate servers according to the configuration information of the candidate servers and the task information of the data processing task associated with the resource manager.
S302, determining a management server and a computing server in the target server, and generating a server cluster.
S303, a preset function switch instruction is sent to a management server and a computing server in the server cluster, so that the management server and the computing server can be instructed to start or close corresponding options in a BIOS system of the management server and the computing server, and performance optimization of the server cluster is performed.
S304, the management server in the control server cluster interacts with the computing server to execute the data processing task.
S305, if the IO rate is detected to exceed the preset rate threshold, the control management server sends an adjustment instruction to each computing server to instruct the computing server to make the data disc into a single disc based on a pre-configured cache card and start the cache function of the cache card.
S306, the control management server sends operation instructions to each computing server to instruct the computing servers to perform disk optimization operation.
Wherein the disk optimization operation includes at least one of: modifying IO scheduling algorithm, setting disk pre-reading buffer and writing dirty data back to disk.
Optionally, the control management server sends an operation instruction to each computing server to instruct the computing server to perform disk optimization operation, including: the control management server sends operation instructions to each computing server, and the proportion of dirty data generated by each computing server process to the system memory and the residence time of the dirty data in the system memory are monitored; determining the proportion and the residence time length, and if the proportion is larger than a preset proportion threshold value or the residence time length is larger than a preset time length threshold value, controlling the management server to call a preset process to write dirty data back to the disk.
S307, the rest memory of the server cluster is analyzed, and the intermediate data generated by the resource manager is cached according to the analysis result.
Optionally, the analyzing the remaining memory of the server cluster and performing cache processing on the intermediate data generated by the resource manager according to the analysis result includes: determining whether the residual memory of the server cluster is larger than a preset memory threshold; if yes, based on a preset resource manager computing unit, the remaining memory is allocated, the remaining memory is mounted on a dedicated disk corresponding to a management server, and intermediate data generated by the resource manager are cached to the dedicated disk corresponding to the management server.
S308, determining a data storage configuration file corresponding to the disk data generated by the data processing task, and modifying the data storage configuration file to obtain the storage address of the updated disk data.
S309, caching the disk data generated by the data processing task according to the updated storage address of the disk data.
Example IV
Fig. 4 is a schematic structural diagram of a data buffering device according to a fourth embodiment of the present invention. The data caching device provided by the embodiment of the present invention may be suitable for a situation of performing targeted caching on different data in a process of executing a data processing task, where the data caching device may be implemented in a form of hardware and/or software, and the data caching device may be configured in an electronic device, and executed by a resource manager, for example, a Yarn resource manager, as shown in fig. 4, where the device specifically includes: an execution module 401, a monitoring module 402, and a caching module 403. Wherein,,
An execution module 401, configured to determine a server cluster for executing a data processing task in response to the data processing task, and control a management server in the server cluster to interact with a computing server, so as to execute the data processing task;
the monitoring module 402 is configured to monitor an IO rate of reading and writing of the server cluster disk during the process of executing the data processing task;
and the buffer module 403 is configured to buffer the intermediate data generated by the resource manager and the disk data generated by the data processing task, respectively, by adopting different buffer policies if the IO rate is detected to exceed the preset rate threshold.
According to the technical scheme, a server cluster for executing the data processing task is determined in response to the data processing task, a management server in the server cluster is controlled to interact with a computing server to execute the data processing task, the IO rate of the disk reading and writing of the server cluster is monitored in the process of executing the data processing task, and if the IO rate is detected to exceed a preset rate threshold, different caching strategies are adopted to respectively cache intermediate data generated by a resource manager and disk data generated by the data processing task. The IO rate of the read-write of the server cluster disk is monitored, and the cache strategy is adjusted when the IO rate exceeds the rate threshold, so that real-time management of the server cluster disk IO can be realized, effective execution of data processing tasks is ensured, and the intermediate data generated by the resource manager and disk data generated by the data processing tasks are respectively cached by adopting different cache strategies, so that the mutual influence of the two data can be avoided, the server cluster pressure is relieved, and the data processing efficiency is improved.
Further, the buffer module 403 may include:
the intermediate data processing unit is used for analyzing the residual memory of the server cluster and caching intermediate data generated by the resource manager according to the analysis result;
the address determining unit is used for determining a data storage configuration file corresponding to the disk data generated by the data processing task, and modifying the data storage configuration file to obtain a storage address of the updated disk data;
and the disk data processing unit is used for caching the disk data generated by the data processing task according to the updated storage address of the disk data.
Further, the intermediate data processing unit is specifically configured to:
determining whether the residual memory of the server cluster is larger than a preset memory threshold;
if yes, based on a preset resource manager computing unit, the remaining memory is allocated, the remaining memory is mounted on a dedicated disk corresponding to a management server, and intermediate data generated by the resource manager are cached to the dedicated disk corresponding to the management server.
Further, the device further comprises:
the adjusting module is used for controlling the management server to send adjusting instructions to each computing server so as to instruct the computing server to make the data disc into a single disc based on a pre-configured cache card and start the cache function of the cache card;
The operation module is used for controlling the management server to send operation instructions to each calculation server so as to instruct the calculation server to perform disk optimization operation; the disk optimization operation includes at least one of: modifying an IO scheduling algorithm, setting a disk pre-reading cache and writing dirty data back to the disk;
further, the operation module is specifically configured to:
the control management server sends operation instructions to each computing server, and the proportion of dirty data generated by each computing server process to the system memory and the residence time of the dirty data in the system memory are monitored;
and determining the proportion and the residence time length, and if the proportion is larger than a preset proportion threshold value or the residence time length is larger than a preset time length threshold value, controlling the management server to call a preset process to write dirty data back to the disk.
Further, the execution module 401 is specifically configured to:
responding to the data processing task, and screening a target server from the candidate servers according to the configuration information of the candidate servers and the task information of the data processing task associated with the resource manager;
and determining a management server and a computing server in the target server, and generating the server cluster.
Further, the device is also used for:
And sending a preset function switch instruction to a management server and a computing server in the server cluster to instruct the management server and the computing server to start or close corresponding options in a BIOS system of the management server and the computing server so as to optimize the performance of the server cluster.
Example five
Fig. 5 is a schematic structural diagram of an electronic device implementing a data caching method according to an embodiment of the present invention. Fig. 5 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 5, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the data caching method.
In some embodiments, the data caching method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. One or more of the steps of the data caching method described above may be performed when the computer program is loaded into RAM 13 and executed by processor 11. Alternatively, in other embodiments, the processor 11 may be configured to perform the data caching method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method of data caching, performed by a resource manager, comprising:
responding to a data processing task, determining a server cluster for executing the data processing task, and controlling a management server in the server cluster to interact with a computing server to execute the data processing task;
in the process of executing the data processing task, monitoring the IO rate of reading and writing of the server cluster disk;
If the IO rate is detected to exceed the preset rate threshold, adopting different caching strategies to respectively cache the intermediate data generated by the resource manager and the disk data generated by the data processing task.
2. The method of claim 1, wherein using different caching policies to respectively cache intermediate data generated by a resource manager and disk data generated by the data processing task, comprises:
analyzing the residual memory of the server cluster, and caching intermediate data generated by the resource manager according to the analysis result;
determining a data storage configuration file corresponding to disk data generated by a data processing task, and modifying the data storage configuration file to obtain a storage address of the updated disk data;
and carrying out cache processing on the disk data generated by the data processing task according to the updated storage address of the disk data.
3. The method of claim 2, wherein analyzing the remaining memory of the server cluster and caching the intermediate data generated by the resource manager according to the analysis result comprises:
Determining whether the residual memory of the server cluster is larger than a preset memory threshold;
if yes, based on a preset resource manager computing unit, the remaining memory is allocated, the remaining memory is mounted on a dedicated disk corresponding to a management server, and intermediate data generated by the resource manager are cached to the dedicated disk corresponding to the management server.
4. The method of claim 1, wherein using different caching policies, before caching the intermediate data generated by the resource manager and the disk data generated by the data processing task, respectively, further comprises:
the control management server sends an adjustment instruction to each computing server to instruct the computing server to make the data disc into a single disc based on a pre-configured cache card and start the cache function of the cache card;
the control management server sends operation instructions to each computing server to instruct the computing servers to perform disk optimization operation; the disk optimization operation includes at least one of: modifying IO scheduling algorithm, setting disk pre-reading buffer and writing dirty data back to disk.
5. The method of claim 4, wherein controlling the management server to send operation instructions to each computing server to instruct the computing server to perform disk optimization operations comprises:
The control management server sends operation instructions to each computing server, and the proportion of dirty data generated by each computing server process to the system memory and the residence time of the dirty data in the system memory are monitored;
and determining the proportion and the residence time length, and if the proportion is larger than a preset proportion threshold value or the residence time length is larger than a preset time length threshold value, controlling the management server to call a preset process to write dirty data back to the disk.
6. The method of claim 1, wherein responsive to a data processing task, determining a cluster of servers that perform the data processing task comprises:
responding to the data processing task, and screening a target server from the candidate servers according to the configuration information of the candidate servers and the task information of the data processing task associated with the resource manager;
and determining a management server and a computing server in the target server, and generating the server cluster.
7. The method of claim 1, wherein controlling the management servers in the server cluster to interact with the computing servers, prior to performing the data processing tasks, further comprises:
and sending a preset function switch instruction to a management server and a computing server in the server cluster to instruct the management server and the computing server to start or close corresponding options in a BIOS system of the management server and the computing server so as to optimize the performance of the server cluster.
8. A data caching apparatus, the apparatus being configured in a resource manager, comprising:
the execution module is used for responding to the data processing task, determining a server cluster for executing the data processing task, controlling a management server in the server cluster to interact with a calculation server, and executing the data processing task;
the monitoring module is used for monitoring the IO rate of the read-write of the server cluster disk in the process of executing the data processing task;
and the cache module is used for adopting different cache strategies to respectively cache the intermediate data generated by the resource manager and the disk data generated by the data processing task if the IO rate exceeds the preset rate threshold.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data caching method of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the data caching method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310246672.XA CN116303132A (en) | 2023-03-10 | 2023-03-10 | Data caching method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310246672.XA CN116303132A (en) | 2023-03-10 | 2023-03-10 | Data caching method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116303132A true CN116303132A (en) | 2023-06-23 |
Family
ID=86784670
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310246672.XA Pending CN116303132A (en) | 2023-03-10 | 2023-03-10 | Data caching method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116303132A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117666963A (en) * | 2023-12-13 | 2024-03-08 | 湖南承希科技有限公司 | A data IO acceleration method for CPU cloud computing platform |
-
2023
- 2023-03-10 CN CN202310246672.XA patent/CN116303132A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117666963A (en) * | 2023-12-13 | 2024-03-08 | 湖南承希科技有限公司 | A data IO acceleration method for CPU cloud computing platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9727372B2 (en) | Scheduling computer jobs for execution | |
| CN102156665B (en) | Differential serving method for virtual system competition resources | |
| US8910153B2 (en) | Managing virtualized accelerators using admission control, load balancing and scheduling | |
| US20190258514A1 (en) | I/O Request Scheduling Method and Apparatus | |
| US11100604B2 (en) | Multiple application cooperative frame-based GPU scheduling | |
| US20200159587A1 (en) | Releasable resource based preemptive scheduling | |
| US20120297216A1 (en) | Dynamically selecting active polling or timed waits | |
| CN114968567A (en) | Method, apparatus and medium for allocating computing resources of a compute node | |
| CN115080209A (en) | System resource scheduling method, device, electronic device and storage medium | |
| WO2024041401A1 (en) | Method and apparatus for processing task, and device and storage medium | |
| CN117539598A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN114356547A (en) | Low-priority blocking method and device based on processor virtualization environment | |
| US11388050B2 (en) | Accelerating machine learning and profiling over a network | |
| US20110055831A1 (en) | Program execution with improved power efficiency | |
| CN117472570A (en) | Methods, apparatus, electronic devices and media for scheduling accelerator resources | |
| CN116303132A (en) | Data caching method, device, equipment and storage medium | |
| RU2450330C2 (en) | Hardware-implemented method of executing programs | |
| CN113821174B (en) | Storage processing method, storage processing device, network card equipment and storage medium | |
| CN117093335A (en) | Task scheduling method and device for distributed storage system | |
| CN115756866A (en) | Load balancing method, device and storage medium | |
| CN116893893B (en) | Virtual machine scheduling method and device, electronic equipment and storage medium | |
| CN118760639B (en) | Interrupt processing module, method, bridge chip and multi-core processor system | |
| CN115826857A (en) | NVMe instruction processing method, device, equipment and medium | |
| CN117687792A (en) | Low-delay response method, device, equipment and medium for key process of system | |
| CN120596219A (en) | Interface current limiting method, device, medium and product based on time slice rotation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |