CN110138634A - Key data monitoring method and terminal - Google Patents

Abstract

The invention discloses a method and a terminal for monitoring key data, which collect all data requests through a real-time streaming platform, send return data corresponding to all the data requests to a message middleware, acquire corresponding key data from the message middleware according to key data request information acquired from the real-time streaming platform, and judge whether the key data is normal, and are different from the prior art which can only monitor related data of the same service process.

Description

A monitoring method and terminal for key data

technical field

The invention relates to the technical field of data monitoring, in particular to a monitoring method and terminal for key data.

Background technique

Most modern Internet products are used through the client, and each client has its own server. Usually, each server runs a separate project, and some may also aggregate multiple servers or microservices into one project, and there are various situations.

During the normal operation of the system, there may be abnormal business data. In order to detect the abnormal situation in operation in time, many systems have monitoring systems. In these monitoring systems, especially for monitoring in micro-service scenarios, generally only monitor their own corresponding server-related service information, for example, only monitor the CPU and memory resources of its own server, as well as its own thread pool and thread total. Information such as quantity, utilization rate, etc., at most, there may be some business data monitoring for its own services, such as data format, type, and data standard format. The way of monitoring is not perfect.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a monitoring method and terminal for key data, which can monitor key business data in the whole process and improve efficiency.

In order to solve the above-mentioned technical problems, a kind of technical scheme adopted in the present invention is:

A monitoring method for key data, including steps:

S1. Collect all data requests through the real-time streaming platform, and send the data returned according to the data requests to the message middleware;

S2. Acquire preset key data request information from the real-time streaming platform, acquire corresponding key data from the message middleware according to the key data request information, and determine whether the key data is normal.

In order to solve the above-mentioned technical problems, another technical scheme adopted by the present invention is:

A monitoring terminal for key data, comprising a memory, a processor and a computer program stored on the memory and running on the processor, the processor implements the following steps when executing the computer program:

S1. Collect all data requests through the real-time streaming platform, and send the data returned according to the data requests to the message middleware;

S2. Acquire preset key data request information from the real-time streaming platform, acquire corresponding key data from the message middleware according to the key data request information, and determine whether the key data is normal.

The beneficial effect of the present invention is that: all data requests are collected through the real-time streaming platform, and the return data corresponding to all data requests are sent to the message middleware, and according to the key data request information obtained from the real-time streaming platform The middleware obtains the corresponding key data, and judges whether the key data is normal. Different from the prior art that can only monitor the related data of the same service process, the present invention can monitor the key business data of the whole process, without affecting the original In the case of business logic, the collection of global data requests and returned data can be realized, which can timely determine whether the key data is normal, avoid failures caused by abnormal data, and improve the monitoring efficiency of key data.

Description of drawings

1 is a flow chart of steps of a method for monitoring key data according to an embodiment of the present invention;

2 is a schematic structural diagram of a monitoring terminal for key data according to an embodiment of the present invention;

3 is a relationship diagram of each module in a method for monitoring key data according to an embodiment of the present invention;

Label description:

1. A monitoring terminal for key data; 2. Memory; 3. Processor.

Detailed ways

In order to describe in detail the technical content, achieved objects and effects of the present invention, the following descriptions are given with reference to the embodiments and the accompanying drawings.

The most critical concept of the present invention is: the real-time streaming platform collects all data requests, the message middleware collects the return data corresponding to all data requests, and obtains from the message middleware according to the key data request information obtained from the real-time streaming platform Corresponding key data, it is judged whether the key data is normal.

Please refer to Fig. 1, a kind of monitoring method of key data, including steps:

S1. Collect all data requests through the real-time streaming platform, and send the data returned according to the data requests to the message middleware;

S2. Acquire preset key data request information from the real-time streaming platform, acquire corresponding key data from the message middleware according to the key data request information, and determine whether the key data is normal.

As can be seen from the above description, the beneficial effect of the present invention is: collect all data requests through the real-time streaming platform, and send the returned data corresponding to all data requests to the message middleware, and request the key data obtained from the real-time streaming platform according to the The information obtains the corresponding key data from the message middleware, and judges whether the key data is normal. Different from the prior art that can only monitor the related data of the same service process, the present invention can monitor the key business data of the whole process. It realizes global data request and collection of returned data without affecting the original business logic. It can timely judge whether the key data is normal, avoid failures caused by abnormal data, and improve the monitoring efficiency of key data.

Further, in the step S1, sending the data returned according to the data request to the message middleware includes:

Forward the data returned according to the data request to the gateway through the real-time streaming platform;

The data returned according to the data request is sent to the message middleware through the gateway.

It can be seen from the above description that the real-time streaming platform forwards the returned data to the gateway, and the gateway is used to realize the sending of the returned data from the real-time streaming platform to the message middleware, which can realize data The fast forwarding can improve the efficiency; at the same time, the data request and the returned data can be stored separately in different modules, which is convenient for subsequent monitoring of multiple key data.

Further, the data returned according to the data request includes the unique identifier of the corresponding interface;

Before the step S1, it includes:

Generate the unique identifier of the corresponding interface of each data request through the gateway;

In the step S2, obtaining the preset key data request information from the real-time streaming platform includes:

Obtain the corresponding key data request from the real-time streaming platform according to the unique identifier;

determining the data item of the key data request;

In the step S2, obtaining the corresponding key data from the message middleware according to the information requested by the key data includes:

Acquire key data corresponding to the key data request from the message middleware according to the unique identifier and data item of the key data request.

As can be seen from the above description, by setting the unique identifier, it is convenient to obtain the returned data corresponding to the key data request, that is, the key data, which is convenient for subsequent collection and judgment of the key data, so as to effectively monitor whether the key data is normal. When obtaining the key data request according to the unique identifier, it is also necessary to determine the data item of the key data request, and obtain the corresponding key data from the message middleware according to the unique identifier and the data item, ensuring that Accuracy in obtaining said key data to avoid omissions.

Further, in the step S2, judging whether the key data is normal includes:

It is judged whether the key data is in the normal range according to the preset key data calculation rule, and if not, an alarm message is issued.

It can be seen from the above description that the key data is judged according to the preset key data rules, which can be applied to a variety of business scenarios and has wide applicability. At the same time, it is convenient to detect abnormalities of key data in time and improve the security and accuracy of the business. .

Further, before judging whether the key data is within the normal range according to the preset key data calculation rule further includes:

Verify whether the key data obtained from the message middleware is the key data corresponding to the key interface, and if so, judge whether the key data is in the normal range according to the preset key data calculation rule; if not, re-acquire the key data Key data corresponding to key interfaces.

As can be seen from the above description, before judging whether the key data is abnormal according to the preset key data calculation rules, the key data needs to be verified, and only when it is determined that the key data is the key data corresponding to the key interface. , the abnormality judgment is performed, which further improves the accuracy of the monitoring of the key data.

Please refer to FIG. 2 , a monitoring terminal for key data, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the following steps when executing the computer program:

S1. Collect all data requests through the real-time streaming platform, and send the data returned according to the data requests to the message middleware;

S2. Acquire preset key data request information from the real-time streaming platform, acquire corresponding key data from the message middleware according to the key data request information, and determine whether the key data is normal.

As can be seen from the above description, the beneficial effect of the present invention is: collect all data requests through the real-time streaming platform, and send the returned data corresponding to all data requests to the message middleware, and request the key data obtained from the real-time streaming platform according to the The information obtains the corresponding key data from the message middleware, and judges whether the key data is normal. Different from the prior art that can only monitor the related data of the same service process, the present invention can monitor the key business data of the whole process. It realizes global data request and collection of returned data without affecting the original business logic. It can timely judge whether the key data is normal, avoid failures caused by abnormal data, and improve the monitoring efficiency of key data.

Further, in the step S1, sending the data returned according to the data request to the message middleware includes:

Forward the data returned according to the data request to the gateway through the real-time streaming platform;

The data returned according to the data request is sent to the message middleware through the gateway.

It can be seen from the above description that the real-time streaming platform forwards the returned data to the gateway, and the gateway is used to realize the sending of the returned data from the real-time streaming platform to the message middleware, which can realize data The fast forwarding can improve the efficiency; at the same time, the data request and the returned data can be stored separately in different modules, which is convenient for subsequent monitoring of multiple key data.

Further, the data returned according to the data request includes the unique identifier of the corresponding interface;

Before the step S1, it includes:

Generate the unique identifier of the corresponding interface of each data request through the gateway;

In the step S2, obtaining the preset key data request information from the real-time streaming platform includes:

Obtain the corresponding key data request from the real-time streaming platform according to the unique identifier;

determining the data item of the key data request;

In the step S2, obtaining the corresponding key data from the message middleware according to the information requested by the key data includes:

Acquire key data corresponding to the key data request from the message middleware according to the unique identifier and data item of the key data request.

As can be seen from the above description, by setting the unique identifier, it is convenient to obtain the returned data corresponding to the key data request, that is, the key data, which is convenient for subsequent collection and judgment of the key data, so as to effectively monitor whether the key data is normal. When obtaining the key data request according to the unique identifier, it is also necessary to determine the data item of the key data request, and obtain the corresponding key data from the message middleware according to the unique identifier and the data item, ensuring that Accuracy in obtaining said key data to avoid omissions.

Further, in the step S2, judging whether the key data is normal includes:

It is judged whether the key data is in the normal range according to the preset key data calculation rule, and if not, an alarm message is issued.

It can be seen from the above description that the key data is judged according to the preset key data rules, which can be applied to a variety of business scenarios and has wide applicability. At the same time, it is convenient to detect abnormalities of key data in time and improve the security and accuracy of the business. .

Further, before judging whether the key data is within the normal range according to the preset key data calculation rule further includes:

Verify whether the key data obtained from the message middleware is the key data corresponding to the key interface, and if so, judge whether the key data is in the normal range according to the preset key data calculation rule; if not, re-acquire the key data Key data corresponding to key interfaces.

As can be seen from the above description, before judging whether the key data is abnormal according to the preset key data calculation rules, the key data needs to be verified, and only when it is determined that the key data is the key data corresponding to the key interface. , the abnormality judgment is performed, which further improves the accuracy of the monitoring of the key data.

Example 1

Please refer to Figure 1 and Figure 3, a method for monitoring key data, comprising the steps:

After the client requests the unified gateway interface, the business logic is executed through the gateway to generate the unique identifier of the interface corresponding to each data request;

S1. Collect all data requests through the real-time streaming platform, and send the data returned according to the data requests to the message middleware;

Wherein, the data returned according to the data request includes the unique identifier of the corresponding interface;

Through the unique identifier, the returned data corresponding to the data request of the same interface can be easily identified;

Specifically, all data requests, that is, requests to access microservices, pass through the real-time streaming platform in advance, and collect initial data requests through the real-time streaming platform;

As shown in FIG. 3 , after the microservice to be accessed by the data request returns the corresponding data, the real-time streaming platform obtains the data returned according to the data request;

The real-time streaming platform forwards the data returned according to the data request to the gateway;

The gateway records all the returned data, and sends the data returned according to the data request to the MQ message middleware;

The gateway sends the returned data to the MQ message middleware in JSON data format, and the specific fields include the interface type called by the client, the interface type of the calling microservice, the data information returned by the microservice interface, and the unique interface corresponding to the interface. identification, etc.;

After the gateway sends the returned data to the MQ message middleware, the returned data is returned to the client;

S2. Acquire preset key data request information from the real-time streaming platform, acquire corresponding key data from the message middleware according to the key data request information, and determine whether the key data is normal.

Among them, the key data request is the data request corresponding to the key interface, the key interface is set by itself according to the specific business, which interfaces can be configured as key interfaces in the background, marked according to the URL address of each interface, and monitored in the monitoring of key data. In the process, the key interface is determined according to the mark, and the unique identifier corresponding to the key interface is obtained;

Specifically, the data monitoring platform obtains the key data request parameter information of the corresponding interface from the real-time streaming platform according to the unique identifier, and determines the data item of the key data request;

The data monitoring platform subscribes the key data corresponding to the key data request from the message middleware in real time according to the unique identifier of the key data request and the data item, and the key data is all return fields of the key data request;

When the data monitoring platform receives the subscribed key data, it determines whether the key data is in the normal range according to the preset key data calculation rules, and if not, sends an alarm message to the notification platform, and the notification platform notifies the relevant personnel. There is currently information about abnormal data that needs to be processed;

The preset key data calculation rules are determined according to specific business rules, and the normal range of the key data also needs to be set according to the specific business rules, such as determining whether the sum of the key data within a preset time is greater than a preset value, so as to determine Whether the key data is abnormal;

For example, in a financial business system, the transaction amount of different types of accounts in a fixed period of time needs to be within a certain range, and then business rules need to be monitored. The account of small user type is between 8:00-16:00 every day, and the total amount of each expenditure and income is required to be between 0-50000. Only between 0-50000 is the data considered to be normal. The range is considered to be abnormal data;

By monitoring each transaction interface, there can be multiple batches of operations within each transaction, which can be spent or received, and the data request of the request interface is collected through the real-time streaming platform, and the message middleware records the data. Request the corresponding return data;

The data monitoring platform subscribes in real time from the message middleware according to the unique identifier and the data item the returned data of all data requests for each small user type account corresponding to the transaction interface between 8:00 and 16:00. The data is the key data at this time;

After receiving the subscribed key data, the data monitoring platform adds the key data corresponding to each small user type account to realize the calculation of the total transaction amount of each small user type account, and judges the total transaction amount of each small user type account. Whether it is between 0 and 50000, if so, the transaction amount complies with the business rules and the data is normal; if not, it means that the data is abnormal, the account may be stolen or the system has BUG and other abnormal situations, and an alarm message will be sent to the notification platform, and the The notification platform notifies the relevant personnel that there is currently abnormal data that needs to be processed.

Embodiment 2

The difference between this embodiment and the first embodiment is that:

Before judging whether the key data is within the normal range according to the preset key data calculation rule, further includes:

Verify whether the key data obtained from the message middleware is the key data corresponding to the key interface, and if so, judge whether the key data is in the normal range according to the preset key data calculation rule; if not, re-acquire the key data Key data corresponding to key interfaces.

Specifically, after receiving the subscribed key data, the data monitoring platform pre-verifies whether the received key data is the key data corresponding to the key interface, that is, whether the received key data is the return of the key data request corresponding to the key interface If yes, determine whether the key data is in the normal range according to the preset key data calculation rule; if not, re-acquire key data corresponding to the key interface, that is, perform step S2 again.

Embodiment 3

Please refer to FIG. 2 , a monitoring terminal 1 for key data includes a memory 2, a processor 3 and a computer program stored in the memory 2 and running on the processor 3, and the processor 3 executes the computer Each step in the first embodiment is implemented during the program.

Embodiment 4

Please refer to FIG. 2 , a monitoring terminal 1 for key data includes a memory 2, a processor 3 and a computer program stored in the memory 2 and running on the processor 3, and the processor 3 executes the computer Each step in the second embodiment is implemented during the program.

To sum up, a monitoring method and terminal for key data provided by the present invention collect all data requests through a real-time streaming platform, and send the return data corresponding to all data requests to the message middleware through the gateway. Identify the data item for which the corresponding key data request is determined from the real-time streaming platform, obtain the corresponding key data from the message middleware according to the unique identifier and the data item, determine whether the key data is normal, and determine whether the key data is normal. Timely alarm when the data is abnormal, different from the prior art that can only monitor the related data of the same service process, the present invention can monitor the key business data of the whole process, and realize the global data request and return without affecting the original business logic The collection of data can realize the fast forwarding of data, ensure the accuracy of obtaining the key data, avoid omission, and can timely judge whether the key data is normal, avoid failures caused by abnormal data, and improve the monitoring efficiency of key data. wide.

The above descriptions are only examples of the present invention, and are not intended to limit the scope of the present invention. Any equivalent transformations made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in related technical fields, are similarly included in the within the scope of patent protection of the present invention.

Claims (10)

1. a monitoring method of key data, is characterized in that, comprises the steps: S1. Collect all data requests through the real-time streaming platform, and send the data returned according to the data requests to the message middleware; S2. Acquire preset key data request information from the real-time streaming platform, acquire corresponding key data from the message middleware according to the key data request information, and determine whether the key data is normal. 2. The method for monitoring key data according to claim 1, wherein in the step S1, sending the data returned according to the data request to the message middleware comprises: Forward the data returned according to the data request to the gateway through the real-time streaming platform; The data returned according to the data request is sent to the message middleware through the gateway. 3. the monitoring method of a kind of key data according to claim 1, it is characterised in that the data returned according to the data request includes the unique identifier of the corresponding interface; Before the step S1, it includes: Generate the unique identifier of the corresponding interface of each data request through the gateway; In the step S2, obtaining the preset key data request information from the real-time streaming platform includes: Obtain the corresponding key data request from the real-time streaming platform according to the unique identifier; determining the data item of the key data request; In the step S2, obtaining the corresponding key data from the message middleware according to the information requested by the key data includes: Acquire key data corresponding to the key data request from the message middleware according to the unique identifier and data item of the key data request. 4. the monitoring method of a kind of key data according to claim 1, is characterized in that, in described step S2, judging whether described key data is normal comprises: It is judged whether the key data is in the normal range according to the preset key data calculation rule, and if not, an alarm message is issued. 5. The method for monitoring key data according to claim 4, wherein the step of judging whether the key data is in a normal range according to a preset key data calculation rule further comprises: Verify whether the key data obtained from the message middleware is the key data corresponding to the key interface, and if so, judge whether the key data is in the normal range according to the preset key data calculation rule; if not, re-acquire the key data Key data corresponding to key interfaces. 6. a monitoring terminal for key data, comprising a memory, a processor and a computer program that is stored on the memory and can run on the processor, wherein the processor implements the following steps when executing the computer program : S1. Collect all data requests through the real-time streaming platform, and send the data returned according to the data requests to the message middleware; S2. Acquire preset key data request information from the real-time streaming platform, acquire corresponding key data from the message middleware according to the key data request information, and determine whether the key data is normal. 7. The monitoring terminal for key data according to claim 6, characterized in that, in the step S1, sending the data returned according to the data request to the message middleware comprises: Forward the data returned according to the data request to the gateway through the real-time streaming platform; The data returned according to the data request is sent to the message middleware through the gateway. 8. The monitoring terminal for key data according to claim 6, wherein the data returned according to the data request includes the unique identifier of the corresponding interface; Before the step S1, it includes: Generate the unique identifier of the corresponding interface of each data request through the gateway; In the step S2, obtaining the preset key data request information from the real-time streaming platform includes: Obtain the corresponding key data request from the real-time streaming platform according to the unique identifier; determining the data item of the key data request; In the step S2, obtaining the corresponding key data from the message middleware according to the information requested by the key data includes: Acquire key data corresponding to the key data request from the message middleware according to the unique identifier and data item of the key data request. 9. The monitoring terminal for key data according to claim 6, wherein, in the step S2, judging whether the key data is normal comprises: It is judged whether the key data is in the normal range according to the preset key data calculation rule, and if not, an alarm message is issued. 10. The monitoring terminal for key data according to claim 9, wherein the step of judging whether the key data is in a normal range according to a preset key data calculation rule further comprises: Verify whether the key data obtained from the message middleware is the key data corresponding to the key interface, and if so, judge whether the key data is in the normal range according to the preset key data calculation rule; if not, re-acquire the key data Key data corresponding to key interfaces.