CN108632057A - A kind of fault recovery method of cloud computing server, device and management system - Google Patents

Abstract

The embodiment of the invention discloses a kind of fault recovery method of cloud computing server, device and management system, this method to include：The hardware resource fault message transmitted by IaaS management platforms is obtained, the operating system failure information of cloud computing server is obtained, obtains the application and trouble information of cloud computing server；According to accessed hardware resource fault message, operating system failure information and application and trouble information determine the failure root of the cloud computing server because；According to failure root because determining troubleshooting strategy；Operation indicated by troubleshooting strategy carries out fault recovery.Implement the embodiment of the present invention, high reliability guarantee can be provided for the Legacy System of enterprise in cloud computing platform, advantageously ensure that the reliable even running of Legacy System.

Description

A cloud computing server failure recovery method, device and management system

technical field

The invention relates to the technical field of cloud computing, in particular to a fault recovery method, device and management system for a cloud computing server.

Background technique

Cloud computing (Cloud Computing) is an emerging business computing model, which distributes computing tasks on a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space and various software services as needed. In order to obtain a series of benefits brought by cloud computing, including reducing the complexity of operation and maintenance and saving hardware costs, more and more enterprises choose to migrate traditional IT systems to cloud computing-related resource pools, so that The entire IT system can use cloud computing services to achieve unified operation and maintenance, and the operating environment of these IT systems has undergone tremendous changes. Since the reliability of the cloud computing platform is not as high as that of dedicated servers, the cloud computing platform It must fully consider how to continue to ensure the reliable operation of the system when some computing resources fail. Under the cloud computing platform, computing resources are allocated from resource pools on demand. When computing resources fail, it is necessary to wait for the cloud to reschedule and allocate computing resources, such as triggering through elastic scaling. In the existing technology, in order to adapt to the architecture of cloud computing, if it is to ensure that the traditional IT system can also obtain the guarantee of high reliability (High Availability, HA) after migrating to the resource pool related to cloud computing, it is usually required that the IT system It is a cloud-ready (Cloud-Ready) type system. For a Cloud-Ready type system, first, it should be a distributed system with a high degree of cohesion and transparency; second, it should be redundant, able to handle server failure, and there is no single point of failure .

However, there are often some legacy systems that do not have the above-mentioned characteristics in the internal IT system of the enterprise. These legacy systems are built as chimney-style vertical systems, and the dynamic allocation of resources in the cloud environment and resource failure are not fully considered at the architectural level. "Cloud-Ready" type systems. From the perspective of architecture compatibility, vertical systems and distributed systems do not have coupling, and cloud computing is generally designed for distributed systems, so the current common HA solution for cloud computing platforms cannot be applied to enterprise legacy systems. After the enterprise migrates the entire IT system (including these legacy systems) to cloud computing-related resource pools, for the legacy systems, it simply redeploys the system to the computing resources allocated by the cloud, and cannot obtain cloud computing resources. Computing guarantees its reliability, such as the inability to achieve elastic scaling and allocate resources on demand, so it will face great challenges in terms of reliability.

Contents of the invention

Embodiments of the present invention provide a cloud computing server failure recovery method, device and management system to solve the reliability problem after the legacy system is migrated to the cloud.

In the first aspect, the embodiment of the present invention provides a fault recovery method for a cloud computing server, which is applied to a cloud computing server, including:

The PaaS management platform obtains the hardware resource failure information sent by the infrastructure as a service IaaS management platform, wherein the IaaS management platform is used to manage the hardware resources of the cloud computing server, and is also used to detect hardware resource failures of the hardware resources Information, the IaaS management platform is independent of the cloud computing server; obtain operating system failure information of the cloud computing server, the operating system failure information is used to indicate the failure of the operating system installed on the cloud computing server ; Acquiring application failure information of the cloud computing server, where the application failure information is used to indicate a failure of an application installed in the operating system;

Determine the fault root cause of the cloud computing server according to the acquired hardware resource fault information, the operating system fault information, and the application fault information; determine a fault handling strategy according to the fault root cause; Handle the actions indicated by the policy for failover.

The above first aspect describes a cloud computing server fault recovery method provided by the embodiment of the present invention from the side of the PaaS management platform. By implementing this method, the PaaS management platform can comprehensively detect the hardware resource layer and the operating system layer of the cloud computing server. And the faults that occur in the application layer, and based on the comprehensive analysis of the above faults, determine the root cause of the fault, and adopt the corresponding fault handling strategy for fault recovery. In the embodiment of the present invention, after the enterprise's legacy system is migrated to the cloud computing server, the PaaS management platform provides an HA solution for the legacy system. When the cloud computing server fails, the PaaS management platform can accurately determine whether the failure occurred in the hardware The resource layer and the operating system layer are also the application layer, and corresponding fault recovery is performed corresponding to this layer, so the HA solution provided by the embodiment of the present invention is comprehensive.

With reference to the first aspect, in some possible implementation manners, the operating system further has a first proxy application;

Acquiring operating system failure information of the cloud computing server includes: determining the operating system failure information by detecting heartbeat information of the first proxy application, the heartbeat information being used to indicate whether the operating system fails.

That is to say, the PaaS management platform installs a first agent application (Agent) on the operating systems of all virtual machines on the cloud computing server where applications are deployed (including applications, application systems, enterprise IT systems, etc.). The proxy application performs heartbeat communication with the PaaS management platform. The PaaS management platform detects the heartbeat with the first agent application. When the heartbeat of a first agent application disappears, it indicates that the virtual machine (operating system) has a disconnection failure, and the PaaS management platform obtains the operating system failure information accordingly.

With reference to the first aspect, in some possible implementation manners, the operating system further has a second agent application;

Obtaining the application failure information of the cloud computing server includes: calling the application status detection script through the second proxy application, and determining the application failure information according to a return value of the status detection script.

Wherein, the second proxy application and the first proxy application may be the same proxy application, or different proxy applications.

The second agent application is also deployed in the application layer, and can be used to manage the application in the cloud computing server, and regularly monitor the running status of the application on the virtual machine, for example, the second agent application provides a status detection script through the application (application system) Monitor related operating status. In a specific application scenario, the application (application system) dynamically provides a status detection script during the running process, and the second agent application periodically calls status.sh in the installation directory and obtains the corresponding return value. Understandably, The second proxy application judges the running state of the application (application system) according to the return value of the script, and sends the corresponding running state to the PaaS. If it is determined that the application is faulty, the second proxy application generates application fault information and sends the application fault information to the PaaS.

It can be understood that when the second proxy application and the first proxy application can be the same proxy application, then the PaaS management platform can monitor the running status of the virtual machine (operating system) through the proxy application, and can also monitor the virtual machine (operating system) through the proxy application. The running state of the application, so that the HA solution provided by the embodiment of the present invention can be quickly and conveniently deployed.

With reference to the first aspect, in some possible implementation manners, determining the root cause of the cloud computing server failure according to the acquired hardware resource failure information, the operating system failure information, and the application failure information includes at least :

If both the hardware resource fault information and the operating system fault information are detected within a preset time, it is determined that the root cause of the fault is that the hardware resource is faulty; or the operating system fault information and the operating system fault information are detected within a preset time. If the application fault information is not detected and the hardware resource fault information is not detected, determine that the root cause of the fault is that the operating system is faulty; or if only the application fault information is detected within a preset time, determine that the root cause of the fault is that all The above application fails.

It can be seen that when a cloud computing server fails, the fault is no longer directly processed at this layer, but is converged to the PaaS management platform. Since the PaaS management platform grasps the status information of the cloud computing server at all levels, it has a global view Information view, so the PaaS management platform conducts comprehensive analysis and judgment based on all fault information acquired within a preset time (for example, 3 minutes), so as to accurately determine the root cause of the above-mentioned fault on the cloud computing server. False positives and negative negatives can be prevented, so the HA solution provided by the embodiment of the present invention is accurate.

With reference to the first aspect, in some possible implementation manners, determining the fault handling strategy according to the root cause of the fault includes:

In the case that the root cause of the failure is a failure of the hardware resource, the failure handling strategy includes restarting the virtual machine, rebuilding the virtual machine locally, and migrating the virtual machine; or in the case of a failure of the operating system, the The failure handling strategy at least includes restarting the virtual machine; or in the case that the root cause of the failure is the failure of the application, the failure handling strategy at least includes restarting the virtual machine and restarting the application.

In a specific application scenario, the PaaS management platform determines a specific fault handling strategy based on the type of fault. For example, a fault diagnosis database can be preset in PaaS. The information is assigned different failure levels, such as failure level one, failure level two, failure level three and so on. For example, among the fault handling strategies preset for faults in the hardware resource layer, the fault handling strategy corresponding to fault level 1 is to restart the virtual machine, the fault handling strategy corresponding to fault level 2 is to rebuild the virtual machine locally, and the fault handling strategy corresponding to fault level 3 The corresponding fault handling strategy is to migrate the virtual machine, and so on. After determining the root cause of the failure, PaaS analyzes the failure of the hardware resource layer based on the actual acquisition, determines the failure level corresponding to the failure of the hardware resource layer, and determines the corresponding fault handling strategy based on the failure level.

In another specific application scenario, PaaS assigns different priorities to different fault handling strategies of the same layer in advance, and automatically selects the layer with the highest priority after the fault root is determined to be a fault on a certain layer based on the received fault information for the first time. The fault handling policy is the fault handling policy that needs to be executed. In the case that the fault handling strategy with high priority cannot achieve fault recovery, PaaS reselects the fault handling strategy with lower priority, and repeats the above steps.

For example, when the root cause of the failure is a failure of the hardware resource, the fault handling strategy is to restart the virtual machine; when the restart of the virtual machine cannot realize the recovery of the hardware resource failure, the fault handling strategy is to rebuild locally A virtual machine; in the case that neither restarting the virtual machine nor locally rebuilding the virtual machine can restore the hardware resource failure, the fault handling strategy is to migrate the virtual machine.

For another example, in the case that the root cause of the fault is a fault in the application, the fault handling strategy is to restart the application; in the case that restarting the application fails to recover the application fault, the fault handling strategy is to restart the virtual machine.

It can be seen that the embodiments of the present invention provide multiple fault recovery means for different fault root causes, and when one of the fault recovery means fails to achieve fault recovery, other fault recovery means will continue to be used for corresponding fault recovery. This ensures that the cloud computing server can recover from the fault as soon as possible after a fault occurs, thereby ensuring the high availability of the cloud computing platform.

With reference to the first aspect, in some possible implementation manners, performing the operations indicated by the fault handling policy includes:

In the case that the root cause of the fault is a failure of the hardware resource, executing the operation indicated by the fault handling policy at least includes: calling the interface of the IaaS management platform to perform the operation indicated by the corresponding fault handling policy; When the operating system fails, executing the operation indicated by the fault handling policy includes: calling the IaaS management platform interface to perform the operation indicated by the corresponding fault handling policy; or when the root cause of the failure is the failure of the application, Executing the operation indicated by the fault handling policy includes: invoking the second proxy application to execute the corresponding operation indicated by the fault handling policy.

It can be seen that when fault recovery is required, the PaaS management platform PaaS initiates fault recovery based on the fault handling strategy to ensure the fault recovery capability. The PaaS management platform can call the IaaS management platform or proxy application to perform corresponding fault recovery based on different The root cause of the fault is to adopt different fault recovery means, so the ability and efficiency of fault recovery can be guaranteed by PaaS, that is to say, the HA solution provided by the embodiment of the present invention is not restricted by the HA capability of IaaS, regardless of the HA capability of IaaS. The reliability of the application running on the cloud computing server is guaranteed, so the HA solution provided by the embodiment of the present invention is universal.

With reference to the first aspect, in some possible implementation manners, performing the operations indicated by the fault handling policy further includes:

Generate a fault log based on the fault information, archive the fault log, and report the fault log to a network management system, where the fault information includes the hardware resource fault information, the operating system fault information, and the application fault information. The fault log is used to indicate the time, location, fault type, fault recovery history and other information of the fault.

When all fault handling strategies and corresponding fault recovery of the PaaS management platform fail to achieve fault recovery, the PaaS management platform will send an alarm to the network management system and report the fault log so that the operation and maintenance personnel can find the fault in time through the network management system And manual maintenance to avoid downtime of the cloud computing server due to the inability to achieve fault recovery, and ensure the high availability of the cloud computing platform.

In the second aspect, an embodiment of the present invention provides a device for implementing fault recovery of a cloud computing server, including: a fault detection module, a fault analysis module, a fault strategy module, and a fault recovery module to implement a fault recovery module provided in the first aspect. A method for realizing fault recovery of a cloud computing server, wherein:

The fault detection module is used to obtain the hardware resource fault information sent by the infrastructure as a service IaaS management platform, wherein the IaaS management platform is used to manage the hardware resources of the cloud computing server, and is also used to detect the hardware resources of the hardware resources Resource fault information, the IaaS management platform is independent of the cloud computing server; it is also used to obtain the operating system fault information of the cloud computing server, and the operating system fault information is used to indicate the operation installed on the cloud computing server A fault in the system; it is also used to obtain application fault information of the cloud computing server, and the application fault information is used to indicate a fault in an application installed in the operating system;

The failure analysis module is used to determine the root cause of the failure of the cloud computing server according to the acquired hardware resource failure information, the operating system failure information and the application failure information;

The fault strategy module is used to determine a fault handling strategy according to the root cause of the fault;

The fault recovery module is used to perform fault recovery according to the operations indicated by the fault handling policy.

In a third aspect, the embodiment of the present invention provides yet another device (server) for implementing fault recovery of a cloud computing server, including: a memory, a processor coupled to the memory, a transmitter, and a receiver, wherein: the transmitting The device is used to send instruction data to the outside, the receiver is used to receive the data sent from the outside, the memory is used to store program codes and related data, and the processor is used to execute the program codes stored in the memory to execute a A fault recovery method for a cloud computing server, wherein the method is the method described in the first aspect.

In a fourth aspect, an embodiment of the present invention provides a management system, the management system includes an IaaS management platform, a PaaS management platform, and a SaaS service platform, wherein the PaaS management platform includes a fault detection module, a fault analysis module, a fault strategy module, and a fault As for the recovery module, the SaaS service platform includes agent applications. Different modules of the PaaS management platform are connected to the IaaS management platform through the first IF of the periodic communication interface, and different modules of the PaaS management platform are connected to the SaaS service platform through the second IF. The management system is used to implement the cloud computing server failure recovery method described in the first aspect.

In the fifth aspect, the embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium stores instructions (implementation codes), and when it runs on a computer, it can make the computer execute the above-mentioned The method described in the first aspect.

In a seventh aspect, an embodiment of the present invention provides a computer program product containing instructions, which, when run on a computer, can cause the computer to execute the method described in the first aspect above based on the instructions.

It can be seen that by implementing the embodiment of the present invention, after the enterprise migrates the legacy system to the cloud computing server of the cloud computing platform, the PaaS can monitor the failure of the hardware resource layer through the IaaS, and can monitor the running status and legacy of the operating system through the proxy application. The operating state of the system. When the PaaS obtains the fault information, it continues to obtain other fault information within the preset time (such as 2 minutes). After the preset time is over, it conducts a comprehensive analysis based on the summary of all fault information to determine the root cause of the fault, and Determine the specific fault handling strategy based on the root cause of the fault, and then call the IaaS or proxy application to perform corresponding fault recovery and fault alarm, thereby ensuring the high availability of the legacy system on the cloud computing platform. The HA solution in the embodiment of the present invention has comprehensive Complete features such as reliability, accuracy and versatility.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the following will describe the drawings that need to be used in the embodiments of the present invention or the background technology.

FIG. 1 is a schematic diagram of a cloud computing platform architecture provided by the prior art;

Fig. 2 is a schematic flow chart of a fault recovery method for a cloud computing server provided by an embodiment of the present invention;

FIG. 3 is a schematic flow chart of another method for recovering from a cloud computing server failure provided by an embodiment of the present invention;

4 is a schematic diagram of a PaaS comprehensive detection cloud computing server failure provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of a PaaS judging whether a cloud computing server fails according to an embodiment of the present invention.

FIG. 6 is a schematic flowchart of a priority-based selection of a fault handling strategy by a PaaS provided by an embodiment of the present invention;

FIG. 7 is a schematic diagram of a device for realizing fault recovery of a cloud computing server provided by an embodiment of the present invention;

FIG. 8 is a schematic diagram of another device for realizing fault recovery of a cloud computing server provided by an embodiment of the present invention;

Fig. 9 is a management system provided by an embodiment of the present invention;

Fig. 10 is another management system provided by the embodiment of the present invention.

Detailed ways

Embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.

In today's cloud era with the rapid development of Internet and big data technology, cloud computing has gradually evolved into the mainstream computing generic type of new information systems. Cloud computing is the product of the integration of a series of network technologies and computing technologies such as parallel computing, distributed computing, utility computing, and virtualization. Please refer to FIG. 1. FIG. 1 is a schematic diagram of a cloud computing platform architecture provided by the prior art. Cloud computing platforms can generally be divided into Software as a Service (Software as a Service, SaaS) and Platform as a Service (Platform as a Service) according to the different service levels provided. as a Service, PaaS) and infrastructure as a service (Infra-structure as a Service, IaaS), among which, PaaS and IaaS can directly send to Platform users provide services, and they can also indirectly serve end users as a supporting platform of the SaaS model. in:

For an independent IaaS (layer I) service model, layer I provides virtual machines and operating systems (opera system, OS) on virtual machines, server virtual computing, virtual storage, and virtual network resources. Users usually pay attention to the type of virtual machine and related configurations (CPU, memory, disk, network, etc.), and the middleware, runtime, and applications on the upper layer of the operating system of the virtual machine are deployed by the user. The service provided by IaaS to consumers is the utilization of all facilities, including processing, storage, network and other basic computing resources, users can deploy and run arbitrary software, consumers do not manage or control any cloud computing infrastructure, but can control Choice of operating system, storage space, deployed applications, and control over limited network components may also be gained.

For the independent PaaS (P layer) service model: the service provided by the P layer to users is to deploy the development language and development tools required by users to the cloud computing infrastructure, and provide users with the operating environment of application software, middleware services, lifecycle management and more. Customers do not need to manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, operating systems, middleware, and runtimes, but users can monitor deployed applications (applications, application systems, etc.), and may also Control the configuration of the hosting environment where your application runs. Users often only focus on the development of application software and the deployment of related data and applications in PaaS.

For the independent SaaS (S layer) service model: the S layer provides users with application (application, application system) services running on the cloud computing infrastructure. Users can access through the client interface on various devices, such as browsers, so as to directly obtain the application services provided by the S layer. Users do not need to manage or control any cloud computing infrastructure, including networks, servers, operating systems, storage, development environments, applications, and more.

In the prior art, computing resources in the cloud computing platform are likely to fail. This requires cloud computing to ensure that the running application system has high availability (High Availability, HA). High availability means that a system is highly reliable, that is, there are few failures, or it can be recovered quickly after a failure. That is to say, when the application system in the cloud computing platform has computing resource failure or other failures, there must be a corresponding HA mechanism to ensure the application system can be restored as soon as possible, shortening the downtime caused by planned daily maintenance operations or unplanned system crashes. In this way, business interruption is avoided and the availability of the application system is improved. Cloud management platforms are usually used to provide HA solutions for cloud computing platforms. At present, the most popular cloud management platform is OpenStack, which is an open source project designed to provide software for the construction and management of public and private clouds. Institutions and individuals in the OpenStack community use OpenStack as a common front end to Infrastructure as a Service (IaaS) resources. OpenStack has become the de facto deployment standard for IaaS cloud platforms in industry and academia today. OpenStack is widely used in various industries. The primary task of OpenStack is to simplify the cloud deployment process and bring good scalability to it. From the perspective of OpenStack, in the cloud computing platform, IaaS is the supporting infrastructure of cloud computing. IaaS provides elasticity and scalability. The infrastructure services of the IaaS cloud platform can provide large-scale, on-demand computing services, storage services, and network services to upper-layer applications. The network service of the IaaS cloud platform, as its core service, is the key to the service quality of various cloud applications. Therefore, OpenStack is deployed on the underlying physical computing, storage, and network resources of the cloud platform to achieve unified management of computing, storage, and network resources, and provide unified cloud infrastructure services at the IaaS layer.

OpenStack proposes a virtual machine (Virtual Machine, VM) HA solution, and the OpenStack HA solution is dedicated to solving fault monitoring and fault recovery at the infrastructure layer. OpenStack's HA solution mainly includes: (1) Monitoring (Monitoring), detecting virtualization layer failures, and monitoring computing node failures; (2) Fencing (Fencing), isolating failed computing nodes; (3) Recovery (Recovery), resolving failures The virtual machine recovers.

It can be seen from the above description that the existing OpenStack architecture only guarantees the failure of the IaaS layer from the design, and realizes the HA of the cloud computing platform by guaranteeing the HA of the IaaS layer. For the upper operating system or application layer, OpenStack believes that The application solves it by itself, so the existing OpenStack HA solution cannot detect faults at the upper-layer operating system or application layer. In fact, so far, the OpenStack community does not have a complete virtual machine HA solution.

Moreover, for chimney-style legacy systems, when an enterprise migrates the legacy systems to the cloud computing platform, due to the incompatibility of the legacy system with the cloud computing platform, failures related to the legacy system may occur at the infrastructure layer, OS layer and application layer, while the OpenStack HA solution cannot completely solve the HA problem at the application layer. Moreover, the OpenStack HA solution only obtains fault information at the infrastructure layer, without comprehensive analysis and judgment based on the status of the application, which is prone to misjudgment and thus creates new faults. In addition, the OpenStack HA solution requires IaaS to have the ability to automatically recover from failures. However, the VM HA capabilities of different IaaS management platforms are inconsistent, and some IaaS management platforms even lack the ability to automatically recover from failures. Therefore, the OpenStack HA solution is not in Common to all IaaS management platforms.

In order to solve the shortcomings in the prior art, the embodiment of the present invention provides a cloud computing server fault recovery method, related devices and management system, which establishes a multi-level, all-round fault detection and management system from the IaaS layer, OS layer, and application layer. The processing mechanism solves the problem of how to ensure the reliable operation of the old legacy system of the enterprise after migrating to the cloud computing platform, so as to ensure the reliability of the application (legacy system) to the maximum extent.

Referring to FIG. 9, FIG. 9 is a management system provided by an embodiment of the present invention. The management system consists of a SaaS service platform (hereinafter referred to as SaaS), a PaaS management platform (hereinafter referred to as PaaS) and an IaaS management platform (hereinafter referred to as IaaS) connection is established. The management system can provide corresponding management services for different levels (I layer, P layer and S layer). In a specific implementation, the IaaS management platform, PaaS management platform and SaaS service platform can run on different In the server, the IaaS management platform, the PaaS management platform and the SaaS service platform may also run on the same server.

Specifically, the IaaS management platform can be a cloud computing infrastructure platform for private, public or hybrid IaaS cloud users, which can centrally manage ultra-large-scale servers, storage and network resources, and form a cloud computing resource pool that can be managed and scheduled in a unified manner. Provide users with computing power that can be used on demand and flexibly scheduled. The IaaS management platform can support the integration of multiple virtualization technologies at the same time, and provide unified resource management, scheduling and monitoring for I-layer hardware resources, and realize the full life cycle of virtual machines, storage resources and network resources from creation, detection to destruction Management, providing a series of high-availability guarantees such as rapid creation, elastic expansion, local reconstruction, and dynamic migration for virtual machines on the I layer, and providing operating system support for virtual machines;

Specifically, the PaaS management platform can be built on the IaaS management platform, that is, the IaaS management platform directly manages local hardware resources. When the PaaS management platform needs to obtain or call relevant information of local hardware resources, it can directly request or call the IaaS management platform. In addition, the PaaS management platform also provides end-to-end monitoring and management of applications and related resources, routes request instructions to valid application instances, and relies on components such as proxy applications, cloud controllers, and health managers to monitor operating systems, applications, and related services. The status, operating parameters and other information are managed and monitored.

Specifically, in the embodiment of the present invention, the SaaS service platform can be built on the infrastructure composed of the PaaS management platform and the IaaS management platform. The SaaS service platform only focuses on providing application services to cloud service operators or enterprises. The SaaS service platform The applications and application systems in the system are managed, detected and controlled by the PaaS management platform.

The management system provides an HA solution for the cloud computing platform (cloud computing server), wherein PaaS is at the core of the HA solution, has a global information view, and coordinates fault detection, analysis, strategy and recovery.

On the one hand, after some layers of cloud computing servers fail, the faults are no longer directly processed in this layer, but are converged to PaaS, and PaaS comprehensive analysis and judgment, and its fault detection range covers the hardware resource level of cloud computing servers , operating system level and application level, so the HA solution provided by the embodiment of the present invention is comprehensive;

On the other hand, the faults of the I layer, P layer, and S layer are collectively aggregated to PaaS for processing. PaaS combines hardware resources, operating systems, and application status to conduct root cause analysis to accurately determine the root cause of the fault, thereby preventing false positives and false negatives. , so the HA scheme provided by the embodiment of the present invention is accurate;

On the other hand, PaaS initiates fault recovery based on fault handling strategies, and adopts different fault recovery methods based on different fault root causes, so the ability and efficiency of fault recovery can be guaranteed by PaaS, that is, the HA solution provided by the embodiment of the present invention does not Restricted by the HA capability of the IaaS, regardless of the HA capability of the IaaS, the reliability of applications running on the cloud computing server can be guaranteed, so the HA solution provided by the embodiment of the present invention is universal.

An embodiment of the present invention also provides a cloud computing server failure recovery method, please refer to Figure 2, a cloud computing server failure recovery method, including:

Step S101, acquiring hardware resource fault information, operating system fault information, and application fault information of a cloud computing server.

In the embodiment of the present invention, PaaS is designed as the fault management core of the entire cloud computing platform (cloud computing server), and PaaS is in the middle of IaaS and SaaS. PaaS can be used to collect cloud service business data managed by PaaS itself, and can also be used for Collect data submitted by IaaS and SaaS, wherein the PaaS is independent of the cloud computing server.

In the embodiment of the present invention, the PaaS obtains the fault information of the cloud computing server, and the fault information specifically includes hardware resource fault information, operating system fault information, and application fault information,

Among them, the hardware resource fault information is used to indicate the fault occurred at the hardware resource fault level, such as insufficient storage resources, network abnormality, virtual machine operation fault, etc.; the operating system fault information is used to indicate the fault occurred at the operating system (OS) level , such as operating system login exceptions, system crashes, etc.; application fault information is used to indicate application failures, such as application suspension, application system exceptions, and the like.

Specifically, the PaaS executes the fault information of the cloud computing server, including performing the following steps S201-S203:

Step S201: Obtain hardware resource failure information sent in the infrastructure as a service system IaaS.

In the embodiment of the present invention, IaaS is used to manage hardware resources, including computing resources, storage resources, and network resources, and IaaS is also used to detect faults generated by hardware resources of cloud computing servers, wherein the IaaS is independent of the cloud computing server.

In the embodiment of the present invention, IaaS can monitor local hardware resources in real time, and can dynamically display the running status of computing resources, storage resources, network resources and related virtual machines. Specifically, IaaS can perform resource capacity query, resource usage control, VM running status monitoring, fault alarm, etc., and relevant information reported to PaaS.

In a specific embodiment, when a virtual machine (VM) running on a cloud computing server fails, or a related hardware configuration (CPU, memory, disk, network, etc.) in the cloud computing server fails, the IaaS detects the failure, And generate corresponding hardware resource failure information in real time, and send the hardware resource failure information to the PaaS, and correspondingly, the PaaS acquires the hardware resource failure information.

Step S202: Obtain operating system failure information of the cloud computing server.

PaaS manages the operating environment of application software and middleware services, and provides lifecycle management for applications that need to run. PaaS can obtain information about the operating system on which middleware and applications rely. For the operating system running on the virtual machine, when the operating system has a failure such as system disconnection or system crash, the PaaS can obtain relevant operating system failure information. In a specific implementation manner, the PaaS can set an agent (Agent) in the operating system OS to be detected, and the PaaS communicates with the Agent, and judges the operating status of the OS where the Agent is located by detecting the communication quality.

Step S203: Obtain application failure information of the cloud computing server.

In the embodiment of the present invention, SaaS only focuses on providing application (application software, application, application system, etc.) services, and does not directly manage and monitor the application services, and the role of managing and monitoring the applications is actually played by PaaS, When an application in the SaaS fails, the PaaS detects and acquires application failure information corresponding to the failure in real time.

It should be noted that there is no necessary sequence between step S201, step S202 and step S203. In addition, in a specific embodiment, two steps in step S201, step S202 and step S203 can be performed simultaneously Step S201, step S202 and step S203 may also be performed at the same time, and the description of the above embodiment should not be construed as limiting the present invention.

Step S102: Determine the root cause of the cloud server failure according to the hardware resource failure information, the operating system failure information and the application failure information.

Wherein, after the PaaS acquires the fault information, it determines the source of the fault information, sets a timer for the fault information, and continues to detect whether other fault information can be obtained within a preset time (for example, 3 minutes). When the preset time of the timer ends, the PaaS will conduct a comprehensive analysis based on all the fault information obtained within the preset time to determine the root cause of the cloud server failure, that is, to determine the specific cause of the cloud server failure and its occurrence. The exact location of the fault.

Referring to Table 1, Table 1 shows the correspondence between the fault information obtained by the PaaS within a preset time and the root cause of the fault determined by the PaaS in some specific embodiments of the present invention.

Table 1

Note: √ indicates a healthy state, × indicates a fault is detected, and NA indicates no detection information

It can be seen that the results of fault detection by PaaS can include three situations: healthy status, detected faults, and no detection information reported. The analysis of the root cause of faults includes the following situations:

When PaaS detects hardware resource failure information and operating system failure information within the preset time, then PaaS will determine that the root cause of the failure is the failure of the hardware resource layer of the cloud computing server;

When PaaS detects hardware resource failure information and application failure information within the preset time, then PaaS will determine that the root cause of the failure is the failure of the hardware resource layer of the cloud computing server;

It can be seen that when the PaaS detects operating system failure information and application failure information within the preset time, and no hardware resource failure information is detected within the preset time, then the PaaS will determine that the root cause of the failure is the cloud computing server operating system OS malfunctions; or

It can be seen that when PaaS only detects application failure information within the preset time, but does not detect hardware resource failure information and operating system failure information, then PaaS will determine that the root cause of the failure is the failure of the application layer of the cloud computing server.

It can be seen that, when only operating system failure information is detected by PaaS; or, when only hardware resource failure information is detected, or only hardware resource failure information and application failure information are detected, the PaaS will determine that the The fault information that occurs within a preset time period is a false report of the cloud computing platform (cloud computing server), and in these cases, the PaaS will ignore the above-mentioned relevant fault information.

Step S103: Determine a fault handling strategy according to the root cause of the fault.

After determining the root cause of the fault in step S102, the PaaS determines a corresponding fault handling strategy according to the root cause of the fault.

Referring to Table 2, Table 2 shows some correspondences between fault root causes and fault handling strategies in the embodiments of the present invention.

Table 2

It can be seen that in a specific application scenario, when the root cause of the failure is a hardware resource failure, the failure handling strategy includes at least restarting (reboot) virtual machine, local reconstruction (rebuild) virtual machine and migration (migration) virtual machine.

It can be seen that in a specific application scenario, when the root cause of the failure is the failure of the operating system, the failure handling strategy includes at least restarting the virtual machine. In this case, when the virtual machine is restarted, the virtual machine will load the corresponding The corresponding operating system, and then complete the restart of the operating system; in special cases, if the operating system can be restarted without restarting the virtual machine, then the fault handling strategy is to restart the operating system directly.

It can be seen that in a specific application scenario, when the root cause of the fault is a fault in the application layer, the fault handling strategy includes at least restarting the application and restarting the virtual machine, wherein restarting the application means directly restarting the relevant application. Restart; restarting the virtual machine is to restart the virtual machine where the application is located first, and the virtual machine will load the corresponding operating system accordingly, and then run the application in the operating system; in special cases, restarting can also be achieved without restarting the virtual machine operating system, then the fault handling strategy is directly to restart the operating system, and then run the application in the operating system.

Step S104, performing fault recovery according to the operations indicated by the fault handling policy.

It can be understood that after the fault handling policy is determined, the PaaS can recover related faults based on the operations indicated by the fault handling policy.

In the embodiment of the present invention, IaaS is responsible for the management and control of hardware resources, including adjusting the CPU, memory, and disk expansion of the virtual machine, restarting the virtual machine, rebuilding locally, and dynamically migrating, etc., to ensure the business continuity of the virtual machine to the greatest extent. , so as to reduce or even eliminate the business impact caused by virtual machine failures.

Therefore, when the operation indicated by the fault handling policy is to operate on the hardware resource layer, the PaaS issues an instruction to the IaaS, and the instruction includes the operation indicated by the fault handling policy. Based on the execution of the instruction, the IaaS implements the relevant failure recovery.

For example, when the fault handling policy is to restart the virtual machine, the PaaS invokes the IaaS interface to restart the virtual machine, and then checks the status of the virtual machine to determine whether the fault is recovered.

When the fault handling policy is to rebuild the virtual machine locally, PaaS determines that the system disk where the virtual machine on the cloud computing server is located is a shared disk, then PaaS calls the IaaS interface to rebuild the virtual machine locally on the shared disk, and then checks the virtual machine Judging whether the fault is recovered or not based on the task status.

In the case that the fault handling strategy is to migrate the virtual machine, the PaaS invokes the IaaS interface to migrate the virtual machine on the faulty cloud computing server to other host machines.

When the fault handling strategy is to restart the operating system, PaaS calls the IaaS interface to restart the virtual machine, and loads the corresponding operating system after the virtual machine restarts.

In special cases, if the operating system can be restarted without restarting the virtual machine, then the PaaS invokes the IaaS interface to directly restart the operating system.

It can be seen that by implementing the embodiment of the present invention, after the enterprise migrates the application (such as application program, application system, IT system, legacy system, etc.) Faults, the running status of the operating system and the running status of the application can be monitored through the agent application. When PaaS obtains the fault information, it will continue to obtain other fault information within the preset time. After the preset time is over, it will conduct a comprehensive analysis based on all the fault information summarized to determine the root cause of the fault, and determine the fault based on the root cause of the fault. Specific fault handling strategy, and then call IaaS or proxy application to perform corresponding fault recovery, thereby ensuring the high availability of the application on the cloud computing platform. The HA solution in the embodiment of the present invention is complete in comprehensiveness, accuracy and versatility. feature.

Please refer to Figure 3-Figure 6 comprehensively, Figure 3 is another cloud computing server failure recovery method provided by the embodiment of the present invention, the method includes but is not limited to the following steps:

Step S301: the IaaS detects hardware resource failure information, and sends the resource failure information to the PaaS.

In a specific embodiment, referring to FIG. 4, the IaaS monitors the hardware resources of the cloud computing server to determine whether the I layer of the cloud computing server fails. When the hardware resources fail, the IaaS reports the hardware resource failure information to the PaaS. For example, when an application (application, application system, enterprise IT system, etc.) is deployed on a virtual machine assigned by IaaS, the administrator registers information such as IaaS tenant, user account, virtual machine address, etc. with PaaS as needed, so that PaaS can Provide high availability solutions to IaaS. For example, when the application is a chimney-type legacy system of an enterprise, in order to make the legacy system highly available, the manager registers information related to the legacy system of the enterprise of the IaaS with the PaaS. After the PaaS recognizes the relevant information, it automatically subscribes to the IaaS for fault alarms of the cloud computing server (host machine) where the legacy system is located and the virtual machine. IaaS detects the running status of virtual machines and host machines in real time. When a fault is detected, IaaS generates corresponding hardware resource fault information (such as abnormal VM running status information), and reports the hardware resource fault information to PaaS, so that PaaS Perform follow-up troubleshooting.

Step S302: The PaaS acquires operating system failure information by detecting heartbeat information of the first proxy application, where the heartbeat information is used to indicate the operating system failure information.

Wherein, step S302 uses the first proxy application only to distinguish it from the second proxy application in step S303.

As shown in Figure 4, PaaS installs an agent application (Agentapplication) on all the virtual machines where the application is deployed, that is to say, the first agent application is deployed on the OS layer of the cloud computing server, and the agent application is used to communicate with The PaaS maintains the heartbeat, and the PaaS regularly detects the heartbeat with the first agent application, so as to determine whether the OS layer fails. When the heartbeat of a first agent application disappears, the PaaS sends a heartbeat request. If the first agent application still cannot return a response to the heartbeat request in time, it indicates that the PaaS and the operating system (or virtual machine) where the first agent application is located A disconnection has occurred, so the PaaS will generate a corresponding operating system failure message.

Step S303: The PaaS acquires the application failure information by detecting the running state of the application through the second proxy application.

Specifically, the application is deployed at the application layer of the cloud computing server, and is integrated into an application-level cloud service by SaaS, so as to provide the cloud service to cloud service operators or enterprises.

Wherein, step S303 uses the second proxy application only to distinguish it from the proxy application in step S302. The second agent application is also deployed on all virtual machines, and the application of the virtual machine node is managed through the agent application. In a specific embodiment, the second proxy application and the first proxy application may be the same application or different applications, which is not limited in this embodiment of the present invention.

As shown in Figure 4, the second proxy application is also deployed at the application layer, and can be used to manage the applications in the cloud computing server, and regularly monitor the running status of the applications on the virtual machine, for example, the status provided by the second proxy application through the application The detection script monitors the relevant running status. In a specific application scenario, the application dynamically provides a status detection script during the running process. For example, the status detection script is status.sh, status.sh defines a return value of 1 to indicate that the application is running, and defines a return value of 2 to indicate that the application has stopped Run, define a return value of 3 to indicate that an abnormal situation has occurred in the application. The status.sh is placed in the installation directory of the application (application system), and the second agent application periodically calls status.sh in the installation directory, and obtains the corresponding return value. It is understandable that the second agent application according to the script The return value of determines the running status of the application, and sends the corresponding running status to SaaS. If the returned value is 2 or 3, the second proxy application generates application fault information and sends the application fault information to the PaaS, and correspondingly, the PaaS obtains the application fault information.

Step S304: The PaaS determines the root cause of the cloud server failure according to the hardware resource failure information, the operating system failure information, and the application failure information.

In the embodiment of the present invention, PaaS obtains fault information at all levels, so PaaS can conduct comprehensive analysis and judgment in combination with fault information at the hardware resource layer, OS layer, and application layer to accurately obtain the root cause of the fault.

Referring to Figure 5, PaaS defines the initial working state of the cloud computing server as a normal state. When PaaS receives fault information, PaaS sets a preset time T (for example, 2 minutes) based on the time point of receiving the fault information, and at the preset time T Continue other fault information within the preset time, and PaaS will make a comprehensive judgment based on all the fault information obtained after the preset time.

If the acquired fault information satisfies the preset condition, PaaS defines the working state of the cloud computing server as a fault state, and further determines the root cause of the fault. If the acquired fault information does not meet the preset condition, PaaS continues to define the working state of the cloud computing server as a normal state.

As shown in Figure 5, if after the preset time T is over, if there are both abnormal VM status and heartbeat disconnection in all the fault information obtained by PaaS, then PaaS will define the working state of the cloud computing server as a fault state, and the root cause of the fault is A fault occurs at the hardware resource layer; if after the preset time T is over, if there is only VM state abnormality or heartbeat disconnection based on all the fault information obtained by PaaS, then PaaS will discard the above fault information and define the working state of the cloud computing server as normal status.

Similarly, if PaaS has both application exception and heartbeat disconnection based on all the fault information obtained after the preset time T, then PaaS will define the working state of the cloud computing server as a fault state, and the root cause of the fault is a fault in the OS layer; If after the preset time T is over, PaaS only has application abnormality or heartbeat disconnection based on all the fault information obtained, then PaaS will discard the above fault information and define the working status of the cloud computing server as normal.

In addition, if after the preset time T is over, PaaS only has application exceptions and no other fault information based on all the fault information obtained by PaaS, then PaaS will define the working state of the cloud computing server as a fault state, and the root cause of the fault is the application layer. malfunction.

Step S305: The PaaS determines a fault handling policy according to the root cause of the fault.

In a specific application scenario, PaaS determines a specific fault handling strategy based on the type of fault. For example, a fault diagnosis database can be preset in PaaS. The fault diagnosis database stores a variety of fault information. Different failure levels, such as failure level 1, failure level 2, failure level 3, etc. For example, among the fault handling strategies preset for faults in the hardware resource layer, the fault handling strategy corresponding to fault level 1 is to restart the virtual machine, the fault handling strategy corresponding to fault level 2 is to rebuild the virtual machine locally, and the fault handling strategy corresponding to fault level 3 The corresponding fault handling strategy is to migrate the virtual machine, and so on. After determining the root cause of the failure, PaaS analyzes the failure of the hardware resource layer based on the actual acquisition, determines the failure level corresponding to the failure of the hardware resource layer, and determines the corresponding fault handling strategy based on the failure level.

In another specific application scenario, PaaS assigns different priorities to different fault handling strategies of the same layer in advance, and automatically selects the layer with the highest priority after the fault root is determined to be a fault on a certain layer based on the received fault information for the first time. The fault handling policy is the fault handling policy that needs to be executed. In the case that the fault handling strategy with high priority cannot achieve fault recovery, PaaS reselects the fault handling strategy with lower priority, and repeats the above steps.

For example, in a specific application scenario, referring to Figure 6, the fault handling strategies corresponding to the hardware resource layer are, in descending order of priority, respectively: restarting the virtual machine, rebuilding the virtual machine locally, migrating the virtual machine, and reporting to the network management system. After determining that the root cause of the fault is a hardware resource failure, PaaS chooses to restart the virtual machine (the highest priority in this layer) as a fault handling strategy. In the case that subsequent restart of the virtual machine fails to achieve fault recovery, the PaaS re-selects the fault management strategy to rebuild the virtual machine locally. In the case that subsequent local rebuilding of virtual machines fails to achieve fault recovery, PaaS reselects the fault management strategy as migrating virtual machines. In the case that the fault recovery cannot be achieved after the subsequent migration of the virtual machine, the PaaS re-selects the fault management strategy as reporting to the network management system, and ends the above fault recovery process after performing the operation of reporting to the network management system.

Wherein, reporting to the network management system specifically includes: PaaS generates a fault log based on the fault information, and archives the fault log, and the fault log is used to indicate information such as the time, location, fault type, and fault recovery history of the fault. The PaaS reports the fault log to the network management system, so that the operation and maintenance personnel can find the fault in time through the network management system and perform manual maintenance.

As another example, in another specific application scenario, the fault handling policy corresponding to the operating system at least includes restarting the virtual machine and reporting to the network management system. In the case that the root cause of the failure is the failure of the operating system, PaaS chooses to restart the virtual machine (the highest priority in this layer) as a failure handling strategy. In the case that the subsequent execution of restarting the virtual machine fails to achieve fault recovery, PaaS reselects the fault management strategy as reporting to the network management system, and ends the above fault recovery process after performing the operation of reporting to the network management system.

As another example, in another specific application scenario, the fault handling strategies corresponding to the application are, in descending order of priority, respectively: restarting the application, restarting the virtual machine, and reporting to the network management system. When the root cause of the failure is a failure at the application layer, PaaS chooses to restart the application (with the highest priority at this layer) as the fault handling strategy. In the case that the subsequent restart of the application fails to achieve fault recovery, the PaaS re-selects the fault management policy as restarting the virtual machine. In the case that the subsequent execution of restarting the virtual machine fails to achieve fault recovery, PaaS reselects the fault management strategy as reporting to the network management system, and ends the above fault recovery process after performing the operation of reporting to the network management system.

Step S306, PaaS sends fault recovery instructions to IaaS respectively, and the fault recovery instructions include the determined fault handling strategy. The second proxy application sends a fault recovery instruction, the fault recovery command includes the determined fault handling strategy, and correspondingly, the second proxy application executes the operation indicated by the fault handling strategy to perform fault recovery on the S layer;

Specifically, the PaaS sends a fault recovery command to the IaaS, and the fault recovery command includes the determined fault handling policy of the I layer. Rebuild the virtual machine and execute the migration of the virtual machine, as shown in Figure 6, after the IaaS performs the above operations, if the PaaS determines that the fault has been recovered, then the above operation process will end. If PaaS judges that the fault has not been recovered after IaaS performs the above operations, then PaaS will perform the operation of reporting to the network management system.

Specifically, the PaaS sends a fault recovery instruction to the second agent application, the fault recovery instruction includes the determined fault handling policy of the S layer, and the IaaS executes the operation indicated by the fault processing policy to perform the fault recovery of the S layer including: executing restarting the application, After the second proxy application executes the above operations, if the PaaS determines that the fault has been recovered, the above operation process will be ended. If IaaS executes the above operations, PaaS judges that the fault has not recovered, then PaaS instructs the second proxy application to restart the application after restarting the virtual machine, and if the fault has not recovered, PaaS will perform the operation of reporting to the network management system.

It can be seen that by implementing the embodiment of the present invention, after the enterprise migrates the application to the cloud computing server of the cloud computing platform, the PaaS can monitor the failure of the hardware resource layer through the IaaS, and can monitor the running status of the operating system and the legacy system through the agent application. operating status. When PaaS obtains the fault information, it will continue to obtain other fault information within the preset time. After the preset time is over, it will conduct a comprehensive analysis based on all the fault information summarized to determine the root cause of the fault, and determine the fault based on the root cause of the fault. Specific fault handling strategies, and then invoke IaaS or proxy applications to perform corresponding fault recovery. If all fault processing strategies cannot achieve fault recovery, PaaS will send fault alarms to the network management system for further fault maintenance, thus ensuring that legacy systems are deployed in the cloud. Due to the high availability of the computing platform, the HA scheme of the embodiment of the present invention has complete features such as comprehensiveness, accuracy and versatility.

Based on the same inventive concept, an embodiment of the present invention provides an apparatus 70 for realizing fault recovery of a cloud computing server. Referring to FIG. device 701. The transmitter 703, the receiver 704, the memory 702, and the processor 701 may be connected via a bus or in other ways (in FIG. 7, connection via a bus is taken as an example). in:

The processor 701 may be one or more central processing units (Central Processing Units, CPUs). In FIG. 7, a processor is taken as an example. When the processor 701 is a CPU, the CPU may be a single-core CPU. It can also be a multi-core CPU.

Memory 702, including but not limited to random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), or Portable read-only memory (Compact Disc Read-Only Memory, CD-ROM), the memory 702 is used for relevant instructions and data, and is also used for storing program codes, and the program codes are specifically used to realize the embodiment in Fig. 5 or Fig. 8 the function of said control node;

Transmitter 703 is used to send instruction data to the outside;

The receiver 704 is used to receive data from the outside;

Specifically, the processor 701 is used to call the program code stored in the memory 702, and perform the following steps:

Use the receiver 704 to acquire the hardware resource failure information sent by the infrastructure as a service IaaS management platform, wherein the IaaS management platform is used to manage the hardware resources of the cloud computing server, and is also used to detect the hardware resources of the hardware resources Fault information, the IaaS management platform is independent of the cloud computing server;

Using the receiver 704 to obtain operating system failure information of the cloud computing server, the operating system failure information is used to indicate a failure of the operating system installed on the cloud computing server;

Obtaining application failure information of the cloud computing server by using the receiver 704, where the application failure information is used to indicate a failure of an application installed in the operating system;

The processor 701 determines the root cause of the failure of the cloud computing server according to the acquired hardware resource failure information, the operating system failure information and the application failure information;

The processor 701 determines a fault handling strategy according to the root cause of the fault;

Use the transmitter 703 to perform fault recovery according to the operations indicated by the fault handling strategy.

Specifically, the operating system also has a first agent application;

Using the receiver 704 to obtain the operating system failure information of the cloud computing server includes: using the receiver 704 to determine the operating system failure information by detecting the heartbeat information of the first proxy application, the heartbeat information being used to indicate the If the above operating system malfunctions.

Specifically, the operating system also has a second agent application;

Obtaining the application fault information of the cloud computing server by using the receiver 704 includes: using the receiver 704 to call the state detection script of the application through the second proxy application, and determine the application according to the return value of the state detection script accident details.

The processor 701 determines the root cause of the failure of the cloud computing server according to the acquired hardware resource failure information, the operating system failure information and the application failure information, at least including: all detected within a preset time In the case of the hardware resource fault information and the operating system fault information, the processor 701 determines that the root cause of the fault is that the hardware resource is faulty; or detects the operating system fault information and the application fault information within a preset time, And if the hardware resource fault information is not detected, the processor 701 determines that the root cause of the fault is that the operating system is faulty; The above application fails.

The processor 701 determines the fault handling strategy according to the root cause of the fault, including:

In the case that the root cause of the failure is a failure of the hardware resource, the failure handling strategy includes restarting the virtual machine, rebuilding the virtual machine locally, and migrating the virtual machine; or in the case of a failure of the operating system, the The failure handling strategy at least includes restarting the virtual machine; or in the case that the root cause of the failure is the failure of the application, the failure handling strategy at least includes restarting the virtual machine and restarting the application.

Specifically, when the root cause of the failure is a failure of the hardware resource, the failure handling strategy includes restarting the virtual machine, locally rebuilding the virtual machine, and migrating the virtual machine, specifically: when the root cause of the failure is the failure of the hardware resource In this case, the fault handling strategy is to restart the virtual machine; in the case that the hardware resource failure recovery cannot be realized by restarting the virtual machine, the fault handling strategy is to rebuild the virtual machine locally; In the case that the hardware resource fault recovery cannot be realized, the fault handling strategy is to migrate the virtual machine.

Specifically, in the case that the root cause of the fault is a fault in the application, the fault handling strategy includes at least restarting the virtual machine and restarting the application, specifically: in the case that the root cause of the fault is faulty in the application, the fault handling The strategy is to restart the application; in the case that restarting the virtual machine fails to recover the application failure, the fault handling strategy is to restart the virtual machine.

Specifically, executing the operations indicated by the fault handling strategy includes: in the case that the root cause of the fault is the failure of the hardware resource, executing the operations indicated by the fault handling strategy at least includes: calling the IaaS management platform interface to execute the corresponding fault Processing the operation indicated by the policy; or in the case that the root cause of the fault is that the operating system fails, executing the operation indicated by the fault handling policy includes: calling the IaaS management platform interface to perform the operation indicated by the corresponding fault handling policy; Or in the case that the root cause of the fault is a fault in the application, executing the operation indicated by the fault handling policy includes: invoking the second proxy application to execute the corresponding operation indicated by the fault handling policy.

The processor 701 executes the operation indicated by the fault handling strategy, which also includes:

The processor 701 generates a fault log based on the fault information, archives the fault log, and uses the transmitter 703 to report the fault log to the network management system, the fault information includes the hardware resource fault information, the operating system fault information and The application failure information.

It should be noted that those skilled in the art can clearly know the implementation method of each functional unit contained in the device 70 through the detailed description of the embodiment in FIGS.

Based on the same inventive concept, an embodiment of the present invention provides an apparatus 80 for realizing fault recovery of a cloud computing server, please refer to FIG. 8 , the apparatus 80 includes a plurality of functional modules, and the detailed description of each functional module is as follows.

The fault detection module 801 is configured to obtain hardware resource fault information sent by the infrastructure as a service IaaS management platform, wherein the IaaS management platform is used to manage the hardware resources of the cloud computing server, and is also used to detect the hardware resources hardware resource fault information, the IaaS management platform is independent of the cloud computing server; it is also used to obtain the operating system fault information of the cloud computing server, and the operating system fault information is used to indicate that it is installed on the cloud computing server the failure of the operating system; and also used to obtain the application failure information of the cloud computing server, the application failure information is used to indicate the failure of the application installed in the operating system;

A failure analysis module 802, configured to determine the root cause of the failure of the cloud computing server according to the acquired hardware resource failure information, the operating system failure information and the application failure information;

A fault strategy module 803, configured to determine a fault handling strategy according to the root cause of the fault;

The fault recovery module 804 is configured to perform fault recovery according to the operations indicated by the fault handling policy.

In a specific embodiment, the operating system has a first agent application; the fault detection module 801 is also used to acquire the operating system fault information of the cloud computing server, including: the fault detection module 801 is also used to detect the The heartbeat information of the first agent application is used to determine the operating system failure information, and the heartbeat information is used to indicate whether the operating system fails.

In a specific embodiment, a second agent application is installed in the operating system; the fault detection module 801 is also used to obtain the application fault information of the cloud computing server includes: the fault detection module 801 is also used to pass The second proxy application invokes a state detection script of the application, and determines the application failure information according to a return value of the state detection script.

In a specific embodiment, the failure analysis module 802 is configured to determine the root cause of the failure of the cloud computing server according to the acquired hardware resource failure information, the operating system failure information and the application failure information, at least including:

The failure analysis module 802 is configured to determine that the root cause of the failure is that the hardware resource fails when both the failure information of the hardware resource and the failure information of the operating system are detected within a preset time; or the failure analysis module 802 It is used to detect the operating system fault information and the application fault information within a preset time, and if the hardware resource fault information is not detected, determine that the root cause of the fault is that the operating system is faulty; or the fault The analysis module 802 is configured to determine that the root cause of the fault is that the application is faulty when only application fault information is detected within a preset time.

In a specific embodiment, the fault strategy module 803 is configured to determine a fault handling strategy according to the root cause of the fault, including:

In the case that the root cause of the failure is a failure of the hardware resource, the failure handling strategy includes restarting the virtual machine, rebuilding the virtual machine locally, and migrating the virtual machine; or in the case of a failure of the operating system, the The failure handling strategy at least includes restarting the virtual machine; or in the case that the root cause of the failure is the failure of the application, the failure handling strategy at least includes restarting the virtual machine and restarting the application.

Wherein, in the case that the root cause of the failure is a hardware resource failure, the failure handling strategy includes restarting the virtual machine, locally rebuilding the virtual machine and migrating the virtual machine, specifically:

In the case where the root cause of the failure is a failure of the hardware resource, the failure handling strategy is to restart the virtual machine; when the hardware resource failure recovery cannot be realized by restarting the virtual machine, the failure handling strategy is to rebuild the virtual machine locally; In the case that neither restarting the virtual machine nor rebuilding the virtual machine locally can recover the hardware resource failure, the failure handling strategy is to migrate the virtual machine.

Wherein, in the case that the root cause of the failure is the failure of the application, the failure handling strategy includes at least restarting the virtual machine and restarting the application, specifically:

In the case that the root cause of the failure is the failure of the application, the failure handling strategy is to restart the application; in the case that restarting the virtual machine cannot realize application failure recovery, the failure handling strategy is to restart the virtual machine.

In a specific embodiment, the fault recovery module 804 is configured to perform fault recovery according to the operations indicated by the fault handling policy, including:

In the case that the root cause of the fault is a failure of the hardware resource, executing the operation indicated by the fault handling policy at least includes: calling the interface of the IaaS management platform to perform the operation indicated by the corresponding fault handling policy; When the operating system fails, executing the operation indicated by the fault handling policy includes: calling the IaaS management platform interface to perform the operation indicated by the corresponding fault handling policy; or when the root cause of the failure is the failure of the application, Executing the operation indicated by the fault handling policy includes: invoking the second proxy application to execute the corresponding operation indicated by the fault handling policy.

In a specific embodiment, the device 80 further includes a fault alarm module 805, the fault alarm module is configured to generate a fault log based on fault information, archive the fault log, and report the fault log to the network management system, the The fault information includes the hardware resource fault information, the operating system fault information and the application fault information.

It should be noted that those skilled in the art can clearly know the implementation method of each functional unit included in the device 80 through the detailed description of the embodiment in FIGS.

Based on the same inventive concept, the embodiment of the present invention also provides another management system. Referring to FIG. A module 801 , a failure analysis module 802 , a failure strategy module 803 and a failure recovery module 804 , and the SaaS service platform 903 includes an agent application 806 . Different modules of the PaaS management platform 902 are connected to the IaaS management platform 901 through the periodic communication interface IF, and different modules of the PaaS management platform 902 are also connected to the SaaS service platform 903 through the IF. The different interfaces are described as follows:

interface name Interface connection relationship IF1 Connect fault detection module 801 and fault analysis module 802 IF2 Connect fault strategy module 803 and fault analysis module 802 IF3 Connect fault recovery module 804 and fault analysis module 802 IF4 Connect fault detection module 801 and IaaS management platform 901 IF5 Connection fault detection module 801 and agent application 806 IF6 Connect fault recovery module 804 and IaaS management platform 901 IF7 Connection failure recovery module 804 and agent application 806

It should be noted that the functions of each management platform, module, and each interface in the management system have been embodied in the above embodiments, for details, please refer to the relevant descriptions in Fig. 2-Fig. 9 , and will not repeat them here.

In the above embodiments, all or part may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention will be generated. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a network site, computer, server, or data center Transmission to another network site, computer, server, or data center via wired (eg, coaxial cable, optical fiber, DSL) or wireless (eg, infrared, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer, and may also be a data storage device such as a server or a data center integrated with one or more available media. The usable medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (such as a DVD, etc.), or a semiconductor medium (such as a solid-state hard disk), and the like.

In the foregoing embodiments, the descriptions of each embodiment have different emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

Claims (19)

1. a kind of fault recovery method of cloud computing server, which is characterized in that be applied to cloud computing server, the method packet It includes：

It is the hardware resource fault message serviced transmitted by IaaS management platforms to obtain infrastructure, and the IaaS management platforms are used In the hardware resource fault message for detecting the hardware resource；

The operating system failure information of the cloud computing server is obtained, the operating system failure information, which is used to indicate, to be installed on The failure that the operating system of the cloud computing server occurs；

The application and trouble information of the cloud computing server is obtained, the application and trouble information, which is used to indicate, is installed on the operation The failure that systematic difference occurs；

According to the accessed hardware resource fault message, the operating system failure information and the application and trouble information Determine the failure root of the cloud computing server because；

According to the failure root because determining troubleshooting strategy；

Operation indicated by the troubleshooting strategy carries out fault recovery.

2. according to the method described in claim 1, it is characterized in that, the operating system also has first agent's application；

The operating system failure information of the cloud computing server is obtained, including：

The operating system failure information, the heartbeat message are determined by detecting the heartbeat message of first agent's application It is used to indicate whether the operating system breaks down.

3. method according to claim 1 or 2, which is characterized in that also there is second agent's application in the operating system；

The application and trouble information of the cloud computing server is obtained, including：

By the state-detection script applied described in second agent's application call, according to the return of the state-detection script Value determines the application and trouble information.

4. method according to any one of claims 1 to 3, which is characterized in that according to the accessed hardware resource Fault message, the operating system failure information and the application and trouble information determine the failure root of the cloud computing server Cause includes at least：

It all detects under the hardware resource fault message and the operating system failure information state, determines in preset time Failure root is because the hardware resource breaks down；Or

The operating system failure information and the application and trouble information are detected in preset time, and are not detected described In the case of hardware resource fault message, failure root is determined because the operating system breaks down；Or

It is only detected under application and trouble information state in preset time, determines failure root because the application is broken down.

5. according to claim 4 any one of them method, which is characterized in that according to the failure root because determining troubleshooting plan Slightly include：

Failure root because the hardware resource break down in the case of, the troubleshooting strategy include restart virtual machine, It is local to rebuild virtual machine and migrate one or more in virtual machine；Or

In failure root because in the case that the operating system breaks down, the troubleshooting strategy includes restarting virtual machine； Or in failure root because in the case that the application is broken down, the troubleshooting strategy includes restarting application and restarting virtual One or both of machine.

6. according to claim 5 any one of them method, which is characterized in that in failure root because event occurs in the hardware resource In the case of barrier, the troubleshooting strategy includes restarting virtual machine, local one kind rebuild in virtual machine and migration virtual machine Or it is a variety of, specially：

In failure root because in the case that the hardware resource breaks down, the troubleshooting strategy is to restart virtual machine；

In the case where execution restarts virtual machine and can not achieve hardware resource fault recovery, the troubleshooting strategy is local weight Build virtual machine；

In the case where virtual machine is restarted in execution and local reconstruction virtual machine all can not achieve hardware resource fault recovery, the event Barrier processing strategy is migration virtual machine.

7. according to claim 5 any one of them method, which is characterized in that break down because of the application in failure root In the case of, the troubleshooting strategy includes one or both of restarting application and restarting virtual machine, specially：

In failure root because in the case that the application is broken down, the troubleshooting strategy is to restart application；

In the case where execution is restarted using can not achieve application and trouble recovery, the troubleshooting strategy is to restart virtual machine.

8. according to the method described in claim 5, it is characterized in that, execute troubleshooting strategy indicated by operation, including：

In failure root because in the case that the hardware resource breaks down, the operation indicated by execution troubleshooting strategy is at least Including：The IaaS management platforms interface is called to execute the operation indicated by corresponding troubleshooting strategy；Or

In failure root because in the case that the operating system breaks down, the operation packet indicated by troubleshooting strategy is executed It includes：The IaaS management platforms interface is called to execute the operation indicated by corresponding troubleshooting strategy；Or

In failure root because in the case that the application is broken down, executing the operation indicated by troubleshooting strategy includes：It adjusts With the operation indicated by the corresponding troubleshooting strategy of second agent's application execution.

9. a kind of device of fault recovery that realizing cloud computing server, which is characterized in that including：

Fault detection module services hardware resource fault message transmitted by IaaS management platforms for obtaining infrastructure, Wherein, the IaaS management platforms are used to detect the hardware resource fault message of the hardware resource；It is additionally operable to obtain the cloud The operating system failure information of calculation server, the operating system failure information, which is used to indicate, is installed on the cloud computing service The failure that the operating system of device occurs；It is additionally operable to obtain the application and trouble information of the cloud computing server, the application event Barrier information, which is used to indicate, is installed on the failure that the application of the operating system occurs；

Failure analysis module, for according to the accessed hardware resource fault message, the operating system failure information With the application and trouble information determine the failure root of the cloud computing server because；

Failure strategy module is used for according to the failure root because determining troubleshooting strategy；

Failure Recovery Module carries out fault recovery for the operation indicated by the troubleshooting strategy.

10. device according to claim 9, which is characterized in that the operating system is applied with first agent；

Fault detection module is additionally operable to obtain the operating system failure information of the cloud computing server, including：

The fault detection module is additionally operable to determine the operation system by detecting the heartbeat message of first agent's application System fault message, the heartbeat message are used to indicate whether the operating system breaks down.

11. device according to claim 9 or 10, which is characterized in that be equipped with second agent in the operating system Using；

The application and trouble information that the fault detection module is additionally operable to obtain the cloud computing server includes：

The fault detection module is additionally operable to the state-detection script by being applied described in second agent's application call, according to The return value of the state-detection script determines the application and trouble information.

12. according to claim 9 to 11 any one of them device, which is characterized in that failure analysis module 802 is used for according to institute The hardware resource fault message, the operating system failure information and the application and trouble information got determines the cloud The failure root of calculation server is because including at least：

The failure analysis module in preset time for all detecting the hardware resource fault message and operation system It unites in the case of fault message, determines failure root because the hardware resource breaks down；Or

The failure analysis module in preset time for detecting the operating system failure information and the application and trouble Information, and in the case of not detecting the hardware resource fault message, determine failure root because the operating system occurs therefore Barrier；Or

The failure analysis module for only being detected under application and trouble information state in preset time, determine failure root because The application is broken down.

13. according to claim 12 any one of them device, which is characterized in that failure strategy module 803 is used for according to Failure root because determine troubleshooting strategy include：

Failure root because the hardware resource break down in the case of, the troubleshooting strategy include restart virtual machine, It is local to rebuild one or more of virtual machine and migration virtual machine；Or

In failure root because in the case that the operating system breaks down, the troubleshooting strategy, which includes at least, restarts virtually Machine；Or

Failure root because the application break down in the case of, the troubleshooting strategy include at least restart application and again Open one or two of virtual machine.

14. according to claim 13 any one of them device, which is characterized in that in failure root because the hardware resource occurs In the case of failure, the troubleshooting strategy includes restarting virtual machine, local one rebuild in virtual machine and migration virtual machine It is a or multiple, specially：

In failure root because in the case that the hardware resource breaks down, the troubleshooting strategy is to restart virtual machine；

In the case where execution restarts virtual machine and can not achieve hardware resource fault recovery, the troubleshooting strategy is local weight Build virtual machine；

In the case where virtual machine is restarted in execution and local reconstruction virtual machine all can not achieve hardware resource fault recovery, the event Barrier processing strategy is migration virtual machine.

15. according to claim 13 or 14 any one of them devices, which is characterized in that in failure root because the application occurs In the case of failure, the troubleshooting strategy, which includes at least, one or two of restarts application and restarts virtual machine, specifically For：

In failure root because in the case that the application is broken down, the troubleshooting strategy is to restart application；

In the case where execution is restarted using can not achieve application and trouble recovery, the troubleshooting strategy is to restart virtual machine.

16. device according to claim 15, which is characterized in that Failure Recovery Module 804 is used at according to the failure Operation indicated by reason strategy carries out fault recovery, including：

In failure root because in the case that the hardware resource breaks down, the operation indicated by execution troubleshooting strategy is at least Including：The IaaS management platforms interface is called to execute the operation indicated by corresponding troubleshooting strategy；Or

In failure root because in the case that the operating system breaks down, the operation packet indicated by troubleshooting strategy is executed It includes：The IaaS management platforms interface is called to execute the operation indicated by corresponding troubleshooting strategy；Or

In failure root because in the case that the application is broken down, executing the operation indicated by troubleshooting strategy includes：It adjusts With the operation indicated by the corresponding troubleshooting strategy of second agent's application execution.

17. a kind of device of fault recovery that realizing cloud computing server, which is characterized in that including：Memory and with it is described Processor, transmitter and the receiver of memory coupling, wherein：The transmitter is used to send director data, institute with to outside Data of the receiver for receiving external transmission are stated, the memory is for storing program code and related data, the place Reason device is for executing the program code stored in the memory, to execute a kind of fault recovery method of cloud computing server, Wherein, the method is such as claim 1 to 8 any one of them method.

18. a kind of management system, including IaaS management platforms, PaaS management platforms and SaaS service platforms, wherein PaaS is managed Platform includes fault detection module, failure analysis module, failure strategy module and Failure Recovery Module, and SaaS service platforms include Agent application, PaaS management platforms are connect with IaaS management platforms and SaaS service platforms by periodic communication interface.It is described Management system is for realizing such as claim 1-8 any one of them method.

19. a kind of computer readable storage medium, which is characterized in that including instruction, when run on a computer so that meter Calculation machine executes such as claim 1-8 any one of them methods.