WO2024239569A1 - Cluster service processing method, server, and system - Google Patents

Abstract

The present application provides a cluster service processing method, a server, and a system. The method is applied to a cluster service system, and the cluster service system comprises a first node and a second node. The method comprises: when a heartbeat network of a first node and a heartbeat network of a second node are normal, acquiring a detection result of local input/output (IO) stream detection; when the detection result of the local IO stream detection is anomalous, acquiring a detection result of opposite-end IO stream detection; and determining a fault reason on the basis of the detection result of the opposite-end IO stream detection and the detection result of the local IO stream detection, and processing a cluster service according to the fault reason. The present application solves the problem in the prior art of cluster service systems being unable to switch a service from a node having an RAID card fault to a normal node for operation as a result of heartbeat networks being unable to detect the RAID card fault in the node.

Description

A cluster service processing method, server and system

This application claims the priority of the Chinese patent application filed with the China Patent Office on May 19, 2023, with application number 202310598620.9 and application name “A Cluster Business Processing Method, Server and System”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of database technology, and in particular to a cluster service processing method, server and system.

Background Art

The rapid development of digitalization and informatization has brought a lot of convenience to people's production and life, but it is also accompanied by the generation of a large amount of data. Existing technologies usually use databases to store and analyze this data. In order to ensure business continuity, a high-availability cluster is usually used to build a database. When a high-availability cluster detects that one or more nodes in the cluster have failed, it will switch the business from the failed node to the normal working node, thereby avoiding business interruption.

In the prior art, node switching in a high-availability cluster usually relies on heartbeat network detection. Heartbeat network detection mainly detects whether a node fails by monitoring the heartbeat signals of the nodes in the cluster. When the heartbeat signal of a node in the cluster is not detected within a specified time, the node is determined to be failed, and the business running on the node is switched to a normal node. However, if the node is not failed, if the independent redundant disk array (RAID) card used to control data storage in the node fails, the IO streams of multiple disks managed by the RAID card cannot be read and written normally, thereby affecting the database business. At this time, node switching is also required. Since the node has not failed, its heartbeat signal can still be monitored, so the cluster business system will not switch the business from the node with a failed RAID card to a normal node.

Summary of the invention

The embodiments of the present application provide a cluster service processing method, server and system for solving the problem in the prior art that the heartbeat network detection cannot detect the RAID card failure in the node, resulting in the cluster service system not switching the service from the node with the RAID card failure to the normal node for operation.

In a first aspect, an embodiment of the present application provides a cluster service processing method, which is applied to a cluster service system, and the cluster service system includes: a first node and a second node; the method includes: when the heartbeat network of the first node and the heartbeat network of the second node are normal, obtaining the detection result of the local input and output IO flow detection; when the detection result of the local IO flow detection is abnormal, obtaining the detection result of the peer IO flow detection; based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, determining the cause of the fault, and processing the cluster service according to the cause of the fault; wherein the local input and output IO flow detection is a first local IO flow detection of the IO flow between the first independent redundant disk array RAID card and the first disk in the first node by the first node, and a first local IO flow detection of the IO flow between the second RAID card and the second disk in the second node by the second node Second local IO flow detection; the peer IO flow detection is a first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk by the first node, and/or a second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk by the second node.

In the embodiment of the present application, by introducing the IO flow detection service, each node detects the local IO flow respectively, and in the case of abnormal detection results, each node detects the IO flow of the opposite end. Since the local IO flow detection service is abnormal, it may be caused by the local IO flow detection service abnormality or a soft failure of the local RAID card. Therefore, in the case of abnormal local IO flow detection, by detecting the IO flow of the opposite end, it can be further determined whether the local RAID card has a soft failure, thereby solving the problem of system pseudo-death without switching the master and standby nodes of the cluster service system when the disk cannot be read and written normally, thereby improving the reliability of the cluster service system.

In a specific implementation, obtaining a detection result of a local IO flow detection includes: the first node and the second node respectively initiate a first data read instruction to a disk of the local node; wherein the first data read instruction is used to read first data in the disk of the local node; obtaining first data returned by the disk of the local node based on the first data read instruction; and the detection result of the local IO flow detection is abnormal, including: the first data returned by the disk of the local node is not obtained.

In the above implementation, the first node and the second node respectively send a data read instruction to their respective local disks, and return data based on the data read instruction. If the data in the local disk can be obtained, the local IO flow detection result is normal. If the data in the local disk is not obtained, the local IO flow detection result is abnormal. The above method is used to realize the detection of the local IO flow by each node.

In a specific implementation, obtaining the first data returned by the disk of the local node based on the first read data instruction includes: obtaining the first data returned by the disk of the local node based on the first read data instruction at a preset time interval within a preset first time period; if the detection result of the local IO flow detection is abnormal, it also includes: if within the preset first time period, at the preset time interval, the total number of times the first data returned by the disk of the local node is obtained is less than a first number threshold, then determining that the detection result of the local IO flow detection is abnormal.

In the above implementation, each node reads the data in the disk of the local node at a certain time interval within a preset first time period. If the total number of times the first data returned by the disk of the local node is obtained within the preset first time period is greater than or equal to the first number threshold, it can be determined that the detection result of the local IO flow detection is normal, otherwise it is abnormal. By judging the number of times the first data is successfully obtained within a preset time period, the judgment of the detection result of the IO flow detection is more accurate and reliable.

In a specific implementation, obtaining first data returned by a disk of a local node based on a first read data instruction includes: obtaining the first data returned by the disk of the local node based on the first read data instruction within a preset second time period; if a detection result of a local IO flow detection is abnormal, it also includes: if the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than a first time threshold, determining that the detection result of the local IO flow detection is abnormal; wherein the preset second time is the maximum time for normally obtaining the first data returned by the disk of the local node.

In the above implementation, by judging the timeout time for each node to obtain the first data within the preset second time period, if the timeout time is greater than the first time threshold, the local IO flow detection result is determined to be abnormal, otherwise it is normal. Through the above implementation, the detection result of the local IO flow detection can be made more reliable.

In a specific implementation manner, obtaining the disk of the local node based on the first read data instruction returned by the first A data, including: within a preset first time period, obtaining first data returned by a disk of a local node based on a first data read instruction; the detection result of the local IO flow detection is abnormal, and also including: within the preset first time period, the number of times the difference between the acquisition time of the first data returned by the disk of the local node and a preset second time is greater than a first time threshold is greater than a second number threshold; the preset second time is the maximum time for the first data returned by the disk of the local node to be normally obtained.

In the above implementation, the detection result of the local IO flow is judged by judging the number of timeouts for each node to obtain the first data within the first time period, thereby further improving the reliability and accuracy of the detection.

In a specific implementation, the obtaining of the detection result of the peer IO flow detection includes: the first node or the second node initiating a second read data instruction to the disk of the peer node; wherein the second read data instruction is used to read second data in the disk of the peer node; obtaining second data returned by the disk of the peer node based on the second read data instruction; if the second data returned by the disk of the peer node is obtained, determining that the detection result of the peer IO flow detection is normal; if the second data returned by the disk of the peer node is not obtained, determining that the detection result of the peer IO flow detection is abnormal.

In the above implementation, each node sends a read data instruction to the disk of the peer node. If the second data returned by the peer node can be obtained, the peer IO flow detection result is determined to be normal, otherwise, it is abnormal. By adding the detection of the peer IO flow, the detection result of the local IO flow can be further verified, thereby enhancing the accuracy and reliability of fault judgment.

In a specific implementation, the determining the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection includes: when the detection result of the first local IO flow detection is abnormal and the detection result of the first peer IO flow detection is normal, determining that the RAID state of the first node is abnormal; when the detection result of the second local IO flow detection is abnormal and the detection result of the second peer IO flow detection is normal, determining that the RAID state of the second node is abnormal; when the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is abnormal, and the detection result of the second local IO flow detection is normal, determining that the IO flow detection service of the first node is abnormal; when the detection result of the second local IO flow detection is abnormal, the detection result of the second peer IO flow detection is abnormal, and the detection result of the first local IO flow detection is normal, determining that the IO flow detection service of the second node is abnormal. When the detection result of the second local IO flow detection is abnormal, the detection result of the second peer IO flow detection is abnormal, and the detection result of the first local IO flow detection is normal, determining that the IO flow detection service of the second node is abnormal.

In the above implementation, when the local IO flow detection result is abnormal, the cluster service system fault can be accurately determined by obtaining the detection result of the peer IO flow and combining it with the local IO flow detection result to determine the cause of the fault.

In a specific embodiment, the method also includes: when the heartbeat detection result of the first node is abnormal and the heartbeat detection result of the second node is normal, determining that the heartbeat network of the first node is abnormal; when the heartbeat detection result of the first node is normal and the heartbeat detection result of the second node is abnormal, determining that the heartbeat network of the second node is abnormal; when the heartbeat detection results of the first node and the second node are both abnormal, determining that the heartbeat networks of the first node and the second node are both abnormal.

In the above implementation manner, in addition to the system pseudo-death situation of each node, other faults may also occur in the cluster business system. By detecting the heartbeat network of each node and judging whether the heartbeat network of each node is normal based on the detection results, the fault judgment of the cluster business system is more accurate.

In a specific implementation, the processing of the cluster service according to the fault cause includes: the cluster service system is in a hot standby scenario, the first node is a master node, and the second node is a standby node; When it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is normal, the cluster service is switched from the first node to the second node for operation, and an alarm processing of a soft fault of the RAID card of the first node is performed; when it is determined that the RAID card status of the second node is abnormal and the RAID card status of the first node is normal, an alarm processing of a soft fault of the RAID card of the second node is performed; when it is determined that the RAID card status of the first node is abnormal and the RAID card status of the second node is abnormal, an alarm processing of a soft fault of the RAID cards of the first node and the second node is performed; when it is determined that the IO flow detection service of the first node and/or the second node is abnormal, an alarm processing of an IO flow detection failure of the first node and/or the second node is performed.

In the above implementation, when the system is in a hot standby scenario, the cluster service system is processed accordingly based on each failure cause, thereby improving the reliability of system operation.

In a specific implementation, the processing of the cluster service according to the fault cause includes: the cluster service system is in an active-active scenario, the first node and the second node are standby nodes for each other, the first node runs the first cluster service, and the second node runs the second cluster service; when it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, the first cluster service is switched from the first node to the second node for operation, and an alarm processing of a soft fault of the RAID card of the first node is performed; when it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, the second cluster service is switched from the second node to the first node for operation, and an alarm processing of a soft fault of the RAID card of the second node is performed; when it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is abnormal, an alarm processing of a soft fault of the RAID card of the first node and the second node is performed; when it is determined that the IO flow detection service of the first node and/or the second node is abnormal, an alarm processing of an IO flow detection fault of the first node and/or the second node is performed.

In the above implementation, when the system is in an active-active scenario, the cluster service system is processed accordingly based on each fault cause, thereby improving the reliability of system operation.

In a second aspect, an embodiment of the present application provides a server, comprising: a processor, a memory, and a communication interface; the memory is used to store executable instructions of the processor; wherein the processor is configured to execute the cluster business processing method described in the first aspect by executing the executable instructions.

In a third aspect, an embodiment of the present application provides a cluster service system, comprising: at least one first node and at least one second node, the first node is a master node, and the second node is a backup node; wherein the first node executes the cluster service processing method described in the first aspect.

The embodiment of the present application provides a cluster service processing method, a server and a system, the method is applied to a cluster service system, the cluster service system includes a first node and a second node; the method includes: when the heartbeat network of the first node and the heartbeat network of the second node are normal, obtaining the detection result of the local input and output IO flow detection; when the detection result of the local IO flow detection is abnormal, obtaining the detection result of the opposite end IO flow detection; based on the detection result of the opposite end IO flow detection and the detection result of the local IO flow detection, determining the cause of the fault, and processing the cluster service according to the cause of the fault; wherein the local input and output IO flow detection is a first local IO flow detection of the IO flow between the first independent redundant disk array RAID card in the first node and the first disk, and a second local IO flow detection of the IO flow between the second RAID card in the second node and the second disk; the opposite end IO flow detection is a first opposite end IO flow detection of the IO flow between the second RAID card in the second node and the second disk, and/or a second opposite end IO flow detection of the IO flow between the first RAID card in the first node and the first disk. Compared with the existing technology that relies on heartbeat network detection to realize the service from faulty nodes to normal Node switching, the present application determines the cause of the failure of the business cluster according to the local IO flow detection and the opposite IO flow detection of the first node and the second node, combined with the heartbeat detection results of the first node and the second node, and processes the cluster business according to the cause of the failure, so that the business cluster of the present application can switch the business from the faulty node to the normal node in time when the business needs to be switched due to a soft failure of the RAID card, thereby solving the problem in the prior art that the heartbeat network detection cannot detect the soft failure of the RAID card in the node, resulting in the cluster business system not switching the business from the node with the RAID card failure to the normal node for operation.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

FIG1 is a schematic diagram of the structure of a cluster service system;

FIG2 is a flow chart of a cluster service processing method according to an embodiment of the present application;

FIG3 is a schematic diagram of a cluster service system performing IO flow detection;

FIG4 is a flow chart of a second embodiment of a cluster service processing method provided in an embodiment of the present application;

FIG5 is a flow chart of a third embodiment of a cluster service processing method provided in an embodiment of the present application;

FIG6 is a flow chart of a fourth embodiment of a cluster service processing method provided in an embodiment of the present application;

FIG7 is a flow chart of a fifth embodiment of a cluster service processing method provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of a server embodiment provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of the structure of another server embodiment provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments made by ordinary technicians in this field under the enlightenment of the embodiments belong to the scope of protection of the present application.

The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

First, the terms involved in this application are explained:

RAID card: Redundant Array of Independent Disks (RAID) combines multiple independent disks into a large-capacity disk group. The RAID card manages the multiple disks that make up the disk array. When implementing database services, the operating system needs to read and write data on the disks it manages through the RAID card.

Fig. 1 is a schematic diagram of the structure of a cluster service system. The cluster service system includes a first node 11 and a second node 12. The first node 11 is a master node, and the second node 12 is a standby node. A heartbeat network detection is performed between the first node 11 and the second node 12 via a heartbeat network link 13. The heartbeat network detection mainly detects whether a failure occurs to the node by monitoring the heartbeat signal of the node in the cluster. Specifically, the first node 11 and the second node 12 send heartbeat messages to each other at a fixed frequency via the heartbeat network link 13, and receive the heartbeat message of the opposite node. If the second node 12 does not receive the heartbeat message of the first node 11 within a specified time, it is determined that the first node 11 fails, and the service running on the first node 11 is switched to the second node 12.

As shown in Figure 1, the first node 11 and the second node 12 both include a processor running an operating system, a RAID card, and multiple disks managed by the RAID card. To implement database services, the operating system needs to read and write data on the disks it manages through the RAID card. If a RAID card used to control data reading and writing in a node fails, such as an error in the RAID card program, the IO streams of the multiple disks managed by the RAID card cannot be read and written normally, thereby affecting the database service, and node switching is also required at this time. Since the node has not failed, its heartbeat signal can still be monitored, so the cluster service system will not switch the service from the node where the RAID card fails to a normal node.

Based on the above technical problems, the technical conception process of the present application is as follows: how to detect the RAID card failure of a node to switch the cluster service from the node with the RAID card failure to a normal node.

The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

FIG2 is a flow chart of a cluster service processing method according to an embodiment of the present application. The method is applied to a cluster service system, which includes a first node and a second node. Referring to FIG2 , the cluster service processing method specifically includes the following steps:

Step S201: When the heartbeat network of the first node and the heartbeat network of the second node are normal, the detection result of the local input and output IO flow detection is obtained.

In one example, a cluster service system is in a hot standby scenario, including a first node and a second node, wherein the first node is a primary node and the second node is a standby node. A database service runs on the first node, and when a failure occurs in the first node, the database service switches from the first node to the second node. The cluster service system may also include a management node. The first node and the second node are external storage devices. FIG3 is a schematic diagram of IO flow detection performed by the cluster service system.

Among them, the local IO flow detection is the first local IO flow detection of the IO flow between the first independent redundant disk array RAID card and the first disk in the first node, and/or the second local IO flow detection of the IO flow between the second RAID card and the second disk in the second node by the second node. As shown in Figure 3, the first local IO flow detection is the detection of the IO flow between the first RAID card and the first disk in the first node by the first node; the second local IO flow detection is the detection of the IO flow between the second RAID card and the second disk in the second node by the second node. The local IO flow detection is mainly used to detect whether the local system disk has problems with normal reading and writing.

In the database business cluster, the first node is the main node and the second node is the standby node. Both the first node and the second node include a RAID card and multiple disks managed by it. Among them, the disk can be a system disk. To implement database business, the operating system needs to read and write data to the disk it manages through the RAID card. In the process of reading and writing data, an IO flow is formed between the RAID card and the disk. Therefore, by detecting the IO flow between the RAID card and the disk, it can be determined whether the RAID card has a soft fault such as a program running error. When the detection result of the IO flow between the RAID card and the disk is abnormal, it can be determined that the RAID card has a soft fault such as a program running error.

In this embodiment, when the heartbeat network of the first node and the heartbeat network of the second node are normal, the detection result of the local input and output IO flow detection is obtained. Specifically, the first node detects the IO flow between the first RAID card and the first disk in the first node to obtain the detection result of the first local IO flow detection; the second node detects the IO flow between the second RAID card and the second disk in the second node to obtain the detection result of the second local IO flow detection. The first node, as the master node, obtains the detection results of the first local IO flow detection and the second local IO flow detection, and the management node can also obtain the detection results of the first local IO flow detection and the second local IO flow detection.

Step S202: when the detection result of the local IO flow detection is abnormal, the detection result of the peer IO flow detection is obtained.

Among them, the peer IO flow detection is the first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk, and/or the second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk. As shown in Figure 3, the first peer IO flow detection is the detection of the IO flow between the second RAID card in the second node and the second disk by the first node; the second peer IO flow detection is the detection of the IO flow between the first RAID card in the first node and the first disk by the second node. The peer IO flow detection is used to verify the results, ensure the consistency and reliability of the results, and to verify whether there is an abnormality in the IO flow detection service itself.

In this embodiment, when the detection result of the local IO flow detection is abnormal, the detection result of the peer IO flow detection is obtained. Specifically, the first node detects the IO flow between the second RAID card and the second disk in the second node to obtain the detection result of the first peer IO flow detection; the second node also detects the IO flow between the first RAID card and the first disk in the first node to obtain the detection result of the second peer IO flow detection. The first node, as the master node, obtains the detection results of the first peer IO flow detection and the second peer IO flow detection. The management node can also obtain the detection results of the first peer IO flow detection and the second peer IO flow detection.

In this embodiment, the first node and the second node also perform heartbeat detection, send heartbeat messages to each other at a fixed frequency, and receive heartbeat messages from the opposite node to obtain heartbeat detection results. The first node, as the master node, obtains the heartbeat detection results of the first node and the heartbeat detection results of the second node. The management node can also obtain the heartbeat detection results of the first node and the heartbeat detection results of the second node. When the heartbeat network of the first node and the heartbeat network of the second node are normal, the above steps S201 to S202 are performed.

Step S203: based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, determine the cause of the fault, and process the cluster service according to the cause of the fault.

In this embodiment, the first node or the management node determines the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection.

Exemplarily, under the premise that the heartbeat detection results of the first node and the second node are normal, the detection result of the first local IO flow detection is abnormal, and the detection result of the first peer IO flow detection is normal, it can be preliminarily determined that the RAID card status of the first node is abnormal, and the detection result of the second local IO flow detection is normal, which means that the IO detection service of the second node is normal. When the detection result of the second peer IO flow detection is abnormal, it can be further determined that the RAID status of the first node is abnormal, and the RAID status of the second node is determined to be normal.

The first node is the master node, the second node is the standby node, the RAID status of the first node is abnormal and the RAID status of the second node is normal, then the cluster service needs to be switched, and the cluster service running on the first node is switched to run on the second node.

In an example, the cluster service system is in an active-active scenario, including a first node and a second node, the first node and the second node are standby nodes for each other, database service A runs on the first node, and database service B runs on the second node. When the first node fails, database service A switches from the first node to the second node for operation; when the second node fails, database service B switches from the second node to the first node for operation. The cluster service system may also include a management node.

When the heartbeat network of the first node and the heartbeat network of the second node are normal, the first node, the second node or the management node can obtain the detection result of the local input and output IO flow detection. When the detection result of the local IO flow detection is abnormal, the detection result of the peer IO flow detection is obtained. The first node, the second node or the management node determines the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and processes the cluster service according to the cause of the fault.

In one example, when the database service is started, the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node are obtained. According to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, the RAID card status of the first node and the second node is determined respectively to determine whether to switch the cluster service.

In one example, after the database service is started, the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node can be re-obtained at preset intervals, and the RAID card status of the first node and the second node can be determined based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, respectively, to determine whether to switch the cluster service.

In an example, the detection results of the first local IO flow detection and the first opposite end IO flow detection, the detection results of the second local IO flow detection and the second opposite end IO flow detection, and the heartbeat detection results of the first node and the second node can be obtained multiple times within a preset time. After determining the RAID card status of the first node and the second node according to the multiple detection results of the first local IO flow detection, the first opposite end IO flow detection, the second local IO flow detection and the second opposite end IO flow detection and the heartbeat detection results of the multiple first nodes and the second nodes, the number of abnormal RAID card statuses of the first node and the second node is counted, and whether to switch the cluster service is determined according to the number of abnormal RAID card statuses of the first node and the second node. Exemplarily, the first node is the master node, the second node is the standby node, the number of abnormal RAID card statuses of the first node exceeds the number threshold, and the number of abnormal RAID card statuses of the second node is 0, that is, when the multiple detection results of the RAID card of the second node are all normal, the cluster service is switched.

In this embodiment, the cluster service system includes a first node and a second node; when the heartbeat network of the first node and the heartbeat network of the second node are normal, a detection result of a local input/output IO flow detection is obtained; when the detection result of the local IO flow detection is abnormal, a detection result of a peer IO flow detection is obtained; based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, a fault cause is determined, and the cluster service is processed according to the fault cause; wherein the local IO flow detection is a first local IO flow detection of the IO flow between the first independent redundant disk array RAID card in the first node and the first disk, and/or a second local IO flow detection of the IO flow between the second RAID card in the second node and the second disk; the peer IO flow detection is a first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk, and/or a second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk by the second node. Compared with the prior art that relies on heartbeat network detection to implement service switching from a faulty node to a normal node, the present application uses local IO flow detection of the first node and the second node and peer IO flow detection, combined with the first node The heartbeat detection result of the node and the second node is used to determine the cause of the failure of the business cluster, and the cluster business is processed according to the cause of the failure, so that the business cluster of the present application can switch the business from the faulty node to the normal node in time when the business needs to be switched due to a soft failure of the RAID card, which solves the problem in the prior art that the heartbeat network detection cannot detect the soft failure of the RAID card in the node, resulting in the cluster business system not switching the business from the node with the RAID card failure to the normal node for operation.

Figure 4 is a flow chart of a second embodiment of a cluster business processing method provided by an embodiment of the present application. Based on the embodiment shown in Figure 2 above, the above step S201 may include: the first node and the second node respectively initiate a first read data instruction to the disk of the local node; wherein the first read data instruction is used to read the first data in the disk in the local node; obtain the first data returned by the disk of the local node based on the first read data instruction; the detection result of the local IO flow detection is abnormal, including: the first data returned by the disk of the local node is not obtained.

Specifically, step S201 may include the following steps:

Step S401: a first node initiates a first data read instruction to a first disk in the first node; the first data read instruction is used to read first data in the first disk.

Step S402: the first node obtains the first data returned by the first disk based on the first data read instruction; if the first data returned by the first disk is obtained, it is determined that the detection result of the first local IO stream detection is normal; if the first data returned by the first disk is not obtained, it is determined that the detection result of the first local IO stream detection is abnormal.

In this embodiment, the RAID card manages multiple disks. When implementing database services, the operating system needs to read and write data on the disks it manages through the RAID card. In the process of reading and writing data, an IO flow is formed between the RAID card and the disk. Therefore, it is possible to determine whether a soft fault has occurred in the RAID card by detecting the IO flow between the RAID card and the disk. When a soft fault such as a program running error occurs in the RAID card, the disk will not respond to the read instruction to return data. Therefore, the IO flow between the RAID card and the system disk can be detected by sending a read data instruction to the disk to determine whether a soft fault has occurred in the RAID card. Exemplarily, the underlying IO test tools, such as the disk stress test (Flexible Input Output tester, referred to as FIO), IO test software (Input Output meter, referred to as IOmeter), etc., can be called to perform a read operation on the disk.

Specifically, when the first node performs a first local IO flow detection, it initiates a first read data instruction to the first disk in the first node. The first read data instruction is used to read the first data in the first disk managed by the first RAID card. If the RAID card in the first node is in normal status, the first disk will return the first data to the operating system of the first node based on the first read data instruction; if a soft failure occurs in the RAID card in the first node, the first disk will not return data.

The first node obtains the first data returned by the first disk based on the first data read instruction. If the first data returned by the first disk is successfully obtained, it is determined that the detection result of the first local IO stream detection is normal; if the first data returned by the first disk is not obtained, it is determined that the detection result of the first local IO stream detection is abnormal.

Based on the embodiment shown in FIG. 2 above, step S202 may include: the first node or the second node initiates a second read data instruction to the disk in the opposite node; wherein the second read data instruction is used to read the second data in the disk in the opposite node; obtaining the second data returned by the disk in the opposite node based on the second read data instruction; if the second data returned by the disk in the opposite node is obtained, determining that the detection result of the opposite IO flow detection is normal; if the second data returned by the disk in the opposite node is not obtained, determining that the detection result of the opposite IO flow detection is abnormal.

Specifically, step S202 may include the following steps:

Step S403: The first node initiates a second data read instruction to the second disk in the second node; the second data read instruction The instruction is used to read the second data in the second disk.

Step S404: the first node obtains the second data returned by the second disk based on the second data read instruction; if the second data returned by the second disk is obtained, it is determined that the detection result of the first peer IO stream detection is normal; if the second data returned by the second disk is not obtained, it is determined that the detection result of the first peer IO stream detection is abnormal.

It should be noted that there is no specific limitation on the execution order of steps S402 to S404.

In this embodiment, the first node also performs IO flow detection on the second node at the opposite end, that is, first opposite end IO flow detection. Specifically, a second read data instruction is initiated to the second disk in the second node, and the second read data instruction is used to read the second data in the second disk. If the RAID card in the second node is in normal state, the second disk will return the second data to the operating system of the first node based on the second read data instruction; if a soft failure occurs in the RAID card in the second node, the second disk will not return data.

The first node obtains the second data returned by the second disk based on the second data read instruction; if the second data returned by the second disk is successfully obtained, it is determined that the detection result of the first peer IO flow detection is normal; if the second data returned by the second disk is not obtained, it is determined that the detection result of the first peer IO flow detection is abnormal.

In this embodiment, the attribute parameters of the IO flow detection include: interval time, timeout time, timeout times, etc. Exemplarily, the first node may perform local IO flow detection and peer IO flow detection at preset intervals. For example, the first node obtains the first data returned by the first disk according to the preset time interval nodes within the preset first time period. If the first data returned by the first disk can be obtained at each time interval point within the preset first time period, the detection result of the first local IO flow detection is determined to be normal; or, the number of times the first node obtains the first data returned by the first disk at all time interval points within the preset first time period is greater than or equal to the first number threshold, the detection result of the first local IO flow detection is determined to be normal; if the first node cannot obtain the first data returned by the first disk at all time interval points within the preset first time period, the detection result of the first local IO flow is determined to be abnormal; or the number of times the first node obtains the first data returned by the first disk at all time interval points within the preset first time period is less than the first number threshold, the detection result of the first local IO flow detection is determined to be abnormal; wherein the first number threshold is the minimum number of times the first node can obtain the first data returned by the first disk within the preset time period; the interval time can be 1s, 5s, 1min, 1h, etc., and is not particularly limited here.

In some implementations, if the difference between the time when the first node obtains the first data returned by the first disk and the preset first time is less than the first time threshold, it is determined that the detection result of the first local IO stream detection is normal; if the difference between the time when the first node obtains the first data returned by the first disk and the preset second time is greater than the first time threshold, it is determined that the detection result of the first local IO stream detection is abnormal, wherein the second time is the preset maximum time that the first node can normally obtain the first data returned by the first disk;

In some implementations, when the number of timeouts for the first node to obtain the first data returned by the first disk is greater than a second number threshold within a preset first time period, the detection result of the first local IO stream detection is determined to be abnormal; when the number of timeouts for the first node to obtain the first data returned by the first disk is less than or equal to the second number threshold, the detection result of the first local IO stream detection is determined to be normal. The number of timeouts is the maximum number of times within the time period that the difference between the acquisition time of the first data returned by the first disk obtained by the first node and the preset second time is greater than the first time threshold.

Similarly, the second node performs the second local IO flow detection and the second peer IO flow detection, which can be performed with reference to the above steps S401 to S404.

In this embodiment, the first node performs the first local IO flow detection and the first peer IO flow detection by respectively initiating a read data instruction to the first disk and the second disk in the first node and reading data from the first disk and the second disk. The operation status of the RAID card of the local node and the RAID card of the opposite node are detected respectively. Thus, the IO flow of the local node and the opposite node can be detected without affecting the database business to determine whether a soft failure has occurred in the RAID, which provides the prerequisite for the business cluster to switch the business from the faulty node to the normal node in time when the business needs to be switched due to a soft failure of the RAID card.

FIG5 is a flow chart of a third embodiment of a cluster service processing method provided by an embodiment of the present application. Based on the embodiments shown in FIG2 to FIG4 above, the above step S203 specifically includes the following steps:

Step S501: Determine the RAID card status of the first node and the second node respectively according to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection and the heartbeat detection results of the first node and the second node.

Among them, one of the first node and the second node is a main node, and the other node is a standby node.

In this embodiment, the RAID card status of the first node and the second node is determined according to the detection results of the first local IO flow detection, the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node.

Local IO flow detection is mainly used to detect whether the local system disk has problems with normal reading and writing; peer IO flow detection is used to verify the local IO flow detection results of the peer node to ensure the consistency and reliability of the results, and to verify whether there are any abnormalities in the IO flow detection service itself.

Specifically, when the heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO flow detection is abnormal, and the detection result of the first peer IO flow detection is normal, it is determined that the RAID state of the first node is abnormal. If the heartbeat detection results of the first node and the second node are normal, it means that the network of the first node and the second node is normal; if the detection result of the first local IO flow detection is abnormal, it means that the local disk of the first node may have a problem of not being able to read and write normally (it may also be that the IO flow detection service of the first node is abnormal). In order to further verify whether the local disk of the first node has a problem of not being able to read and write normally, it is necessary to judge the first peer IO flow detection result. If the first peer IO flow detection result is normal, the situation that the IO flow detection service of the first node is abnormal can be ruled out. Therefore, it can be determined that the RAID state of the first node is abnormal.

When the heartbeat detection results of the first node and the second node are both normal and the detection result of the first local IO flow detection is normal, it can be preliminarily determined that the RAID card of the first node and the IO detection service of the first node are normal. In order to further verify the above conclusion, it can be judged by the result of the second peer IO flow detection. If the detection result of the second peer IO flow detection is normal, it is determined that the RAID status of the first node is normal.

Similarly, the heartbeat detection results of the first node and the second node are both normal, and the first local IO flow detection is normal, and the second peer IO flow detection is normal. If the detection result of the first peer IO flow detection is abnormal, it can be preliminarily determined that the RAID card of the second node is abnormal. At this time, if the detection result of the second local IO flow detection is abnormal, it is further determined that the RAID state of the second node is abnormal. The heartbeat detection results of the first node and the second node are normal, which means that the network of the first node and the second node is normal; the first local IO flow detection is normal, and the second peer IO flow detection is normal, which can respectively indicate that the IO flow detection services of the first node and the second node are normal. At this time, if the detection result of the second local IO flow detection is abnormal, it can be preliminarily determined that the local disk of the second node has a problem of not being able to read and write normally. In order to further verify this conclusion, the detection result of the first peer IO flow detection is used for judgment. If the detection result is abnormal, the RAID card state of the second node is abnormal, which further verifies that the local disk of the second node has a problem of not being able to read and write normally. Therefore, it can be further determined that the RAID state of the second node is abnormal.

The heartbeat detection results of the first node and the second node are both normal. If the detection result of the second local IO flow detection is When it is normal, it can be preliminarily determined that the local disk of the second node is reading and writing normally. In order to further verify the above conclusion, verification is performed through the detection result of the first peer IO flow detection. If the above result is normal, it is determined that the RAID status of the second node is normal.

Specifically, when the heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is abnormal, the detection result of the second local IO flow detection is normal, and the detection result of the second peer IO flow detection is normal, it is determined that the IO flow detection service of the first node is abnormal. If the heartbeat detection results of the first node and the second node are normal, it means that the network of the first node and the second node is normal; if the detection result of the first local IO flow detection is abnormal, it means that the local disk of the first node may not be able to read and write normally (it may also be that the IO flow detection service of the first node is abnormal), but the detection result of the second peer IO flow detection is normal, which means that the local system disk of the first node can be read and written normally. At this time, the abnormal state of the RAID card of the first node can be ruled out, and it can be determined that the abnormal detection result of the first local IO flow detection is due to the abnormality of the IO flow detection service of the first node; if the detection result of the second local IO flow detection is normal, it means that the local disk of the second node can be read and written normally, but the detection result of the first peer IO flow detection is abnormal, which further verifies that the IO flow detection of the first node is abnormal.

Similarly, when the heartbeat detection results of the first node and the second node are both normal, the detection result of the first local IO flow detection is normal, the detection result of the first peer IO flow detection is normal, the detection result of the second local IO flow detection is abnormal, and the detection result of the second peer IO flow detection is abnormal, it is determined that the IO flow detection service of the second node is abnormal.

In some implementations, the network conditions of the first node and the second node may also affect the detection result of the peer IO flow detection. When the heartbeat detection result of the first node and/or the second node is abnormal, the detection result of the peer IO flow detection is also abnormal.

Step S502: when it is determined that the RAID card status of the master node is abnormal and the RAID card status of the standby node is normal, the cluster service is switched from the master node to the standby node for operation, and an alarm process for a soft failure of the master node RAID card is performed.

Step S503: When the cluster service system is in a hot standby scenario and it is determined that the RAID card status of the standby node is abnormal and the RAID card status of the master node is normal, an alarm processing of a soft fault of the RAID card of the standby node is performed;

In the cluster service system in the active-active scenario, when it is determined that the RAID card status of the standby node is abnormal and the RAID card status of the active node is normal, the cluster service is switched from the standby node to the active node for operation, and the alarm of the soft fault of the RAID card of the standby node is processed.

Step S504: when it is determined that the RAID card status of the master node is abnormal and the RAID card status of the standby node is abnormal, an alarm process of a soft failure of the RAID cards of the master node and the standby node is performed.

Step S505: when it is determined that the IO flow detection of the primary node and/or the backup node is abnormal, an alarm process of the IO flow detection failure of the primary node and/or the backup node is performed.

Specifically, an IO flow detection anomaly indicates that an IO test tool, such as FIO, IOmeter, etc., has a fault, and an alarm needs to be issued for the IO flow detection fault so that the management personnel can handle the IO flow detection fault.

In one example, a cluster service system is in a hot standby scenario, including a first node and a second node, the first node is a primary node, the second node is a backup node, and a database service runs on the first node. When a failure occurs in the first node, the database service switches from the first node to the second node. The cluster service system may also include a management node. The first node or the management node determines the RAID card status of the first node and the second node respectively based on the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection, and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, and performs a heartbeat detection on the first node and the second node according to the RAID card status of the first node and the second node. The cluster service is processed. The first node is the main node and the second node is the backup node. The specific method is shown in Table 1 below:

Table 1 IO flow detection results and processing solutions of the first node and the second node in the hot standby scenario

In one example, a cluster service system is in a dual-active scenario, including a first node and a second node, and the first node and the second node are standby nodes for each other. The first node runs a first cluster service, and the second node runs a second cluster service. When the first node fails, the first cluster service switches from the first node to the second node; when the second node fails, the second cluster service switches from the second node to the first node. The cluster service system may also include a management node.

The first node, the second node or the management node obtains the detection results of the first local IO flow detection and the first peer IO flow detection, the detection results of the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node. The first node, the second node or the management node determines the RAID card status of the first node and the second node respectively according to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection, and the heartbeat detection results of the first node and the second node, and processes the cluster service according to the RAID card status of the first node and the second node. The specific method is shown in Table 2 below:

Table 2 IO flow detection results and processing solutions of the first node and the second node in the active-active scenario

In the active-active scenario, the RAID status of the first node is abnormal, while the RAID status of the second node is normal. FIG6 is a flow chart of a cluster service processing method embodiment 4 provided in the present application. As shown in FIG6, the first node and the second node are standby nodes for each other. Database service A is running on the first node, and database service A is running on the second node. Carry out database business B. The heartbeat detection results of the first node and the second node are normal, the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is normal, the detection result of the second local IO flow detection is normal, and the detection result of the second peer IO flow detection is abnormal. Therefore, it is determined that the system disk read of the first node is abnormal, there is a system pseudo-death situation, and database business A needs to be switched from the first node to the second node.

In addition, since a soft failure of the RAID card may cause problems in the operation of the operating system, which in turn causes the cluster service to be unable to switch normally, it is necessary to determine whether the cluster service has been successfully switched. If the switch is successful, the switching process ends; if the switch fails, the first node can restart the second node through the out-of-band management channel to switch the cluster service from the second node to the first node. Exemplarily, the out-of-band management channel can be an Intelligent Platform Management Interface (IPMI) channel. Figure 7 is a flow chart of a fifth embodiment of a cluster service processing method provided in an embodiment of the present application.

As shown in FIG7 , the embodiment of the present application performs IO flow detection by calling the underlying IO test tools, such as FIO, IOmeter, etc., to periodically read the system disk. For example, if the result is returned, it is normal; if there is no response after the timeout, it is determined that the system disk read is abnormal, and there is a system pseudo-death situation. The result needs to be synchronized to the node where the cluster service is located to switch the cluster service to the normal node to ensure business continuity. If the node where the cluster service is located cannot be switched under the pseudo-death system at this time, a network channel, such as an IPMI channel, can be established through out-of-band management of normal nodes and faulty nodes to restart the faulty node and thus switch the cluster service.

In this embodiment, the first node determines the RAID card status of the first node and the second node according to the detection results of the first local IO flow detection, the first peer IO flow detection, the second local IO flow detection and the second peer IO flow detection and the heartbeat detection results of the first node and the second node, and processes the cluster service according to the RAID card status of the first node and the second node, and can timely switch the service from the faulty node to the normal node when the service needs to be switched due to the soft failure of the RAID card, and realize the alarm of abnormal situation. This further solves the problem in the prior art that the service cannot be switched from the node with the faulty RAID card to the normal node because the heartbeat network detection cannot detect the soft failure of the RAID card in the node.

The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

FIG8 is a schematic diagram of the structure of a server embodiment provided in an embodiment of the present application; as shown in FIG8 , the server 60 includes: an acquisition module 61 and a processing module 62 . The acquisition module 61 is used to acquire the detection result of the local input and output IO flow detection when the heartbeat network of the first node and the heartbeat network of the second node are normal; the acquisition module 61 is also used to acquire the detection result of the peer IO flow detection when the detection result of the local IO flow detection is abnormal; the processing module 62 is used to determine the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and process the cluster service according to the cause of the fault; wherein the local IO flow detection is a first local IO flow detection of the IO flow between the first independent redundant disk array RAID card in the first node and the first disk, and/or a second local IO flow detection of the IO flow between the second RAID card in the second node and the second disk; the peer IO flow detection is a first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk, and/or a second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk by the second node.

The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

In a possible implementation, the acquisition module 61 is specifically configured to: the first node and the second node to send the local node The first read data instruction is initiated by the disk of the local node; wherein the first read data instruction is used to read the first data in the disk in the local node; the first data returned by the disk of the local node based on the first read data instruction is obtained; the detection result of the local IO flow detection is abnormal, including: the first data returned by the disk of the local node is not obtained.

In a possible implementation scheme, the acquisition module 61 is specifically used to obtain the first data returned by the disk of the local node based on the first read data instruction at a preset time interval within a preset first time period; the detection result of the local IO flow detection is abnormal, and also includes: if within the preset first time period, at the preset time interval, the total number of times the first data returned by the disk of the local node is obtained is less than the first number threshold, then the detection result of the local IO flow detection is determined to be abnormal.

In a possible implementation scheme, the acquisition module 61 is specifically used to acquire, within a preset second time period, the first data returned by the disk of the local node based on the first read data instruction; the detection result of the local IO flow detection is abnormal, and further includes: if the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than the first time threshold, then determining that the detection result of the local IO flow detection is abnormal; wherein the preset second time is the maximum time for normally acquiring the first data returned by the disk of the local node.

In a possible implementation scheme, the acquisition module 61 is specifically used to acquire, within a preset first time period, the first data returned by the disk of the local node based on the first read data instruction; the detection result of the local IO flow detection is abnormal, and also includes: within the preset first time period, the number of times the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than the first time threshold is greater than the second number threshold; the preset second time is the maximum time for normally acquiring the first data returned by the disk of the local node.

The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

In a possible implementation scheme, the acquisition module 61 is specifically used for the first node or the second node to initiate a second read data instruction to the disk in the opposite node; wherein the second read data instruction is used to read the second data in the disk in the opposite node; obtain the second data returned by the disk in the opposite node based on the second read data instruction; if the second data returned by the disk in the opposite node is obtained, it is determined that the detection result of the opposite IO flow detection is normal; if the second data returned by the disk in the opposite node is not obtained, it is determined that the detection result of the opposite IO flow detection is abnormal.

The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

In a possible implementation, the processing module 62 is specifically used to determine that the RAID state of the first node is abnormal when the detection result of the first local IO flow detection is abnormal and the detection result of the first peer IO flow detection is normal; determine that the RAID state of the second node is abnormal when the detection result of the second local IO flow detection is abnormal and the detection result of the second peer IO flow detection is normal; determine that the IO flow detection service of the first node is abnormal when the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is abnormal, and the detection result of the second local IO flow detection is normal; determine that the IO flow detection service of the second node is abnormal when the detection result of the second local IO flow detection is abnormal, the detection result of the second peer IO flow detection is abnormal, and the detection result of the first local IO flow detection is normal.

The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

In a possible implementation scheme, the cluster service system is in a hot standby scenario, the first node is the main node, the second The node is a spare node; the processing module 62 is specifically used to switch the cluster service from the first node to the second node for operation when it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, and perform alarm processing of the soft fault of the RAID card of the first node; when it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, perform alarm processing of the soft fault of the RAID card of the second node; when it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is abnormal, perform alarm processing of the soft fault of the RAID cards of the first node and the second node; when it is determined that the IO flow detection service of the first node and/or the second node is abnormal, perform alarm processing of the IO flow detection failure of the first node and/or the second node.

In a possible implementation scheme, the cluster service system is in a dual-active scenario, the first node and the second node are backup nodes for each other, the first node runs the first cluster service, and the second node runs the second cluster service; the processing module 62 is specifically used to switch the first cluster service from the first node to the second node for operation when it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, and perform alarm processing of the soft fault of the RAID card of the first node; when it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, switch the second cluster service from the second node to the first node for operation, and perform alarm processing of the soft fault of the RAID card of the second node; when it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is abnormal, perform alarm processing of the soft fault of the RAID cards of the first node and the second node; when it is determined that the IO flow detection service of the first node and/or the second node is abnormal, perform alarm processing of the IO flow detection failure of the first node and/or the second node.

The server provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

FIG9 is a schematic diagram of the structure of a server provided in an embodiment of the present application. As shown in FIG9 , the server 70 includes: a processor 71, a memory 72, and a communication interface 73; wherein the memory 72 is used to store executable instructions of the processor 71; the processor 71 is configured to execute the technical solution in any of the aforementioned method embodiments by executing the executable instructions.

Optionally, the memory 72 can be independent or integrated with the processor 71.

Optionally, when the memory 72 is a device independent of the processor 71, the server 70 may further include: a bus 74 for connecting the above devices.

The server is used to execute the technical solution in any of the aforementioned method embodiments, and its implementation principle and technical effect are similar and will not be repeated here.

The embodiment of the present application also provides a cluster service system. The cluster service system includes at least one first node and at least one second node, wherein the first node is a master node and the second node is a backup node; wherein the first node executes the technical solution in any of the above method embodiments.

Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or modify some or all of the technical features thereof. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of this application.

Claims (10)

A cluster service processing method, characterized in that the method is applied to a cluster service system, the cluster service system comprising: a first node and a second node; the method comprises: When the heartbeat network of the first node and the heartbeat network of the second node are normal, obtaining a detection result of a local input and output IO flow detection; When the detection result of the local IO flow detection is abnormal, the detection result of the peer IO flow detection is obtained; Determine the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection, and process the cluster service according to the cause of the fault; The local input/output IO flow detection is a first local IO flow detection of an IO flow between a first independent redundant disk array RAID card in the first node and a first disk by the first node, and a second local IO flow detection of an IO flow between a second RAID card in the second node and a second disk by the second node by the second node; The peer IO flow detection is a first peer IO flow detection of the IO flow between the second RAID card in the second node and the second disk by the first node, and/or a second peer IO flow detection of the IO flow between the first RAID card in the first node and the first disk by the second node. The cluster service processing method according to claim 1, characterized in that obtaining the detection result of the local IO flow detection comprises: The first node and the second node respectively initiate a first data read instruction to the disk of the local node; wherein the first data read instruction is used to read the first data in the disk in the local node; Acquire first data returned by the disk of the local node based on the first data read instruction; The detection result of the local IO flow detection is abnormal, including: the first data returned by the disk of the local node is not obtained. The cluster service processing method according to claim 2, characterized in that the step of obtaining the first data returned by the disk of the local node based on the first read data instruction comprises: Acquire, within a preset first time period, first data returned by the disk of the local node based on the first data read instruction at preset time intervals; The detection result of the local IO flow detection is abnormal, and also includes: if within the preset first time period, according to the preset time interval point, the total number of times the disk of the local node returns the first data is obtained is less than the first number threshold, then the detection result of the local IO flow detection is determined to be abnormal. The cluster service processing method according to claim 2, characterized in that the step of obtaining the first data returned by the disk of the local node based on the first read data instruction comprises: Within a preset second time period, obtaining first data returned by the disk of the local node based on the first data read instruction; The detection result of the local IO flow detection is abnormal, further comprising: if the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than the first time threshold, determining that the detection result of the local IO flow detection is abnormal; The preset second time is the maximum time for the disk of the local node to return the first data normally. The cluster service processing method according to claim 2, characterized in that the obtaining of the local node The first data returned by the disk based on the first data read instruction includes: Within a preset first time period, obtaining first data returned by the disk of the local node based on the first data read instruction; The detection result of the local IO flow detection is abnormal, and also includes: within the preset first time period, the number of times the difference between the acquisition time of the first data returned by the disk of the local node and the preset second time is greater than the first time threshold is greater than the second number threshold; the preset second time is the maximum time for the first data to be returned by the disk of the local node to be normally acquired. The cluster service processing method according to claim 1, wherein the step of obtaining the detection result of the peer IO flow detection comprises: The first node or the second node initiates a second data read instruction to the disk of the opposite node; wherein the second data read instruction is used to read second data in the disk of the opposite node; Obtain the second data returned by the disk of the opposite node based on the second data read instruction; if the second data returned by the disk of the opposite node is obtained, determine that the detection result of the opposite IO flow detection is normal; if the second data returned by the disk of the opposite node is not obtained, determine that the detection result of the opposite IO flow detection is abnormal. The cluster service processing method according to claim 1 is characterized in that: the determining the cause of the fault based on the detection result of the peer IO flow detection and the detection result of the local IO flow detection comprises: When the detection result of the first local IO flow detection is abnormal and the detection result of the first peer IO flow detection is normal, determining that the RAID state of the first node is abnormal; When the detection result of the second local IO flow detection is abnormal and the detection result of the second peer IO flow detection is normal, determining that the RAID state of the second node is abnormal; When the detection result of the first local IO flow detection is abnormal, the detection result of the first peer IO flow detection is abnormal, and the detection result of the second local IO flow detection is normal, it is determined that the IO flow detection service of the first node is abnormal; When the detection result of the second local IO flow detection is abnormal, the detection result of the second peer IO flow detection is abnormal, and the detection result of the first local IO flow detection is normal, it is determined that the IO flow detection service of the second node is abnormal. The cluster service processing method according to claim 7, wherein the processing of the cluster service according to the fault cause comprises: The cluster service system is in a hot standby scenario, the first node is a master node, and the second node is a standby node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, the cluster service is switched from the first node to the second node for operation, and an alarm processing of a soft fault of the RAID card of the first node is performed; When it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, performing alarm processing of a soft fault of the RAID card of the second node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is abnormal, an alarm process of a soft failure of the RAID cards of the first node and the second node is performed; When it is determined that the IO flow detection service of the first node and/or the second node is abnormal, an alarm process of the IO flow detection failure of the first node and/or the second node is performed. The cluster service processing method according to claim 7, wherein the processing of the cluster service according to the fault cause comprises: The cluster service system is in an active-active scenario, the first node and the second node are each other's standby nodes, a first cluster service runs on the first node, and a second cluster service runs on the second node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is normal, the first cluster service is switched from the first node to the second node for operation, and an alarm processing of a soft fault of the RAID card of the first node is performed; When it is determined that the RAID card state of the second node is abnormal and the RAID card state of the first node is normal, switching the second cluster service from the second node to the first node for operation, and performing alarm processing for a soft fault of the RAID card of the second node; When it is determined that the RAID card state of the first node is abnormal and the RAID card state of the second node is abnormal, an alarm process of a soft failure of the RAID cards of the first node and the second node is performed; When it is determined that the IO flow detection service of the first node and/or the second node is abnormal, an alarm process of the IO flow detection failure of the first node and/or the second node is performed. A cluster service system, characterized by comprising: At least one first node and at least one second node, wherein the first node is a primary node and the second node is a backup node; Wherein, the first node executes the cluster service processing method according to any one of claims 1 to 9.