CN112039726A - Data monitoring method and system for content delivery network CDN device - Google Patents
Data monitoring method and system for content delivery network CDN device Download PDFInfo
- Publication number
- CN112039726A CN112039726A CN202010860803.XA CN202010860803A CN112039726A CN 112039726 A CN112039726 A CN 112039726A CN 202010860803 A CN202010860803 A CN 202010860803A CN 112039726 A CN112039726 A CN 112039726A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring
- module
- monitoring data
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 169
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000005540 biological transmission Effects 0.000 claims abstract description 36
- 230000002776 aggregation Effects 0.000 claims abstract description 12
- 238000004220 aggregation Methods 0.000 claims abstract description 12
- 238000005070 sampling Methods 0.000 claims description 12
- 230000002159 abnormal effect Effects 0.000 claims description 7
- 238000013515 script Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 238000012800 visualization Methods 0.000 claims description 4
- 230000004931 aggregating effect Effects 0.000 claims description 3
- 238000004140 cleaning Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 abstract description 2
- 238000012546 transfer Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 4
- 238000005192 partition Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the invention provides a data monitoring method and a system of CDN equipment, wherein the method comprises the following steps: the method comprises the following steps that a data acquisition module acquires monitoring data of CDN equipment to be monitored and reports the monitoring data to a transmission module; the transmission module writes the monitoring data into a message queue module; the collecting and storing module reads the monitoring data from the message queue module and writes the monitoring data into a time sequence database; and the time sequence data display module configures a data source into the time sequence database and sends out an alarm according to the monitoring data in the time sequence database. Compared with the prior art, the technical scheme of the embodiment of the invention has simple and light logic and high speed, adopts a mode of directly warehousing the message queue, does not need to use functions of hadoop data cleaning and the like, and has higher aggregation speed than the prior technical scheme.
Description
Technical Field
The invention relates to data monitoring of a Content Delivery Network (CDN) in cloud computing, in particular to a data monitoring method and a data monitoring system of CDN equipment.
Background
One set of technologies related to hadoop big data analysis in the prior art is complex and difficult to maintain. In the prior art, monitoring is reported to a center for analysis through logs, and the consumed resources are more. The prior art has various components and is difficult to maintain, and a whole set of scheme of hadoop big data needs to be known.
Disclosure of Invention
The embodiment of the invention provides a data monitoring method and a data monitoring system for CDN (content delivery network) equipment, which are used for solving the problems that in the prior art, monitoring is performed through log reporting to a center for analysis, resources are consumed, components are various and maintenance is difficult.
In a first aspect, an embodiment of the present invention provides a data monitoring method for a content delivery network CDN device, where the method includes:
the method comprises the following steps that a data acquisition module acquires monitoring data of CDN equipment to be monitored and reports the monitoring data to a transmission module;
the transmission module writes the monitoring data into a message queue module;
the collecting and storing module reads the monitoring data from the message queue module and writes the monitoring data into a time sequence database;
and the time sequence data display module configures a data source into the time sequence database and sends out an alarm according to the monitoring data in the time sequence database.
In a second aspect, an embodiment of the present invention provides a data monitoring system for a CDN device in a content delivery network, where the system includes:
the data acquisition module is used for acquiring monitoring data of the CDN equipment to be monitored and reporting the monitoring data to the transmission module;
the transmission module is used for writing the monitoring data into a message queue module;
the collecting and storing module is used for reading the monitoring data from the message queue module and writing the monitoring data into a time sequence database;
and the time sequence data display module is used for configuring a data source into the time sequence database and sending out an alarm according to the monitoring data in the time sequence database.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the method for monitoring data of a CDN device in a content delivery network is implemented.
The technical scheme has the following beneficial effects:
the embodiment of the invention collects the monitoring data of the CDN equipment to be monitored through the data collection module and reports the monitoring data to the transmission module; the transmission module writes the monitoring data into the message queue module; the collection storage module reads out the monitoring data from the message queue module and writes the monitoring data into the time sequence database, so that brand-new technical consideration and brand-new scheme design are adopted, the logic is simpler, and the mode of storing the monitoring data into the database is changed into the mode of infiluxdb and other time sequence databases, thereby overcoming the problems that in the prior art, monitoring is reported to a center through logs, the resource consumption is more, and the components are various and are difficult to maintain.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a data monitoring method for a CDN device in a content delivery network according to an embodiment of the present invention;
FIG. 2 is a functional block diagram of a data monitoring system of a CDN device of a content delivery network according to an embodiment of the present invention;
FIG. 3 is an exemplary architecture diagram of an embodiment of the present invention;
fig. 4 is a graph illustrating an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a flowchart of a data monitoring method for a CDN device in a content delivery network according to an embodiment of the present invention. As shown in fig. 1, the method comprises the steps of:
s110: the data acquisition module acquires monitoring data of the CDN equipment to be monitored and reports the monitoring data to the transmission module.
S120: and the transmission module writes the monitoring data into the message queue module.
S130: the collecting and storing module reads out the monitoring data from the message queue module and writes the monitoring data into a time sequence database;
s140: the time sequence data display module configures a data source into a time sequence database and gives an alarm according to the monitoring data in the time sequence database.
In some embodiments, the step S110 of acquiring, by the data acquisition module, monitoring data of the CDN device to be monitored, and reporting the monitoring data to the transmission module may specifically include:
the data acquisition module acquires a monitoring item through a configuration file, acquires corresponding monitoring data through the monitoring item at regular time, and then sends the monitoring data to the transmission module through an http request; or,
the data acquisition module acquires monitoring data reported by other programs or scripts at regular time through the monitoring port of the data acquisition module, and then sends a request carrying the monitoring data to the transmission module.
In some embodiments, the writing of the monitoring data into the message queue module by the transmission module in step S120 may specifically include:
when the transmission module is started, an IP address and a logical address topic of a message queue module, such as a kafka cluster of a distributed message system, are obtained through a configuration file;
and the transmission module writes the monitoring data into the kafka cluster of the distributed message system according to the message queue module, such as the IP address and the logical address topic of the kafka cluster. In this embodiment, the message queue needs to obtain the target address during writing and reading, and the target address is composed of the IP address of the kafka program and the logical address topic in the kafka program. Topic, also called a subject, represents a logical address in kafka, which can also be considered as a category, with the unique target address being determined by the ip address and Topic.
In some embodiments, the monitoring data includes the following fields: monitoring item names, monitoring item marks, monitoring item acquisition periods, unique marks of monitored CDN equipment, monitoring item acquisition values and timestamps corresponding to monitoring data and representing acquisition time; the time sequence data display module sends out an alarm according to the monitoring data in the time sequence database, and the time sequence data display module specifically comprises the following steps:
the time sequence data display module executes a Structured Query Language (SQL) aggregation statement, aggregates monitoring item acquisition values meeting conditions every second preset time length according to a specified monitoring item name metric or a specified monitoring item name metric and a monitoring item tag within a first preset time length to obtain aggregate data of the current sampling time, and draws a curve graph according to the aggregate data of all sampling time within the first preset time length; the first preset time length is longer than the second preset time length;
for each drawn curve graph, when the aggregate data of a certain sampling moment is greater than or equal to a threshold value preset for the curve graph, alarming is carried out, and the alarming mode comprises the following steps: and marking an abnormal area on the graph, and sending a mail to a preset mail server for receiving alarm information or sending the alarm information to a specified social media account.
The following are exemplified as follows:
in one example, the fields in which the monitoring data is reported to the database may include these:
endPoint represents a machine, namely a CDN device to be monitored;
tags, representing a monitoring item tag; as a refinement metric (refinement indicator), may not write, tags may be empty;
timestamp, representing a Timestamp;
metric, representing a monitoring item name;
value, which represents the collection Value of the monitoring item;
step, Step is used to determine how often to report, in this example 1 minute.
As an example, one possible aggregated scenario is similar to the following:
each server is data reported separately, and when the number of requests of 10 machines in a cluster needs to be counted, the embodiment of the present invention needs to aggregate the data of the 10 machines.
The aggregation method is to find the metric (monitoring item) and tags (may not be present) of the 10 machines (endpoints), and aggregate according to time.
For example:
13:00 machine A mteric: qps value:10
13:00 machine B mteric: qps value:30
13:01 machine A mteric: qps value:20
13:01 machine B mteric: qps value:50
After polymerization, the following are obtained:
13:00 Cluster 1 commercial qps value 40
13:01 Cluster 1 metric: qps value:70
Wherein qps is the number of requests as a monitoring item, which is just an example. In one case, tags and metric determine the only monitoring items. In another case, tags may not be present, i.e. in a scenario where no subdivision of the metric is required, then the metric is used as a field to determine the unique monitoring entry.
In some embodiments, the timing database may include: infiluxdb or open-tsdb; the message queue module may include: kafka clusters, rabbitMQ, ActiveMQ, ZeroMQ, RocktetMQ, or redis; the collection and storage module may include: a stash cluster; the time series data presentation module comprises a visualization tool grafana.
Fig. 2 is a functional block diagram of a data monitoring system of a CDN device in a content delivery network according to an embodiment of the present invention. As shown in fig. 2, the system includes:
the data acquisition module 10 is configured to acquire monitoring data of the CDN device to be monitored, and report the monitoring data to the transmission module 20;
a transmission module 20, configured to write the monitoring data into the message queue module 30;
a collection and storage module 40 for reading out the monitoring data from the message queue module 30 and writing the monitoring data into the time-series database 50;
and a time-series data display module 60, configured to configure the data source as a time-series database, and issue an alarm according to the monitoring data in the time-series database 50.
In some embodiments, the data acquisition module 10 may be specifically configured to: acquiring a monitoring item through a configuration file, acquiring corresponding monitoring data through the monitoring item at regular time, and sending the monitoring data to the transmission module 20 through an http request; or, the monitoring data reported by other programs or scripts at regular time is acquired through the monitoring port, and then a request carrying the monitoring data is sent to the transmission module 20.
In some embodiments, the transmission module 20 may be specifically configured to: acquiring the IP address and the logical address topic of the kafka cluster through a configuration file during starting; the monitoring data is written to the distributed messaging system kafka cluster 30 based on the IP address of the kafka cluster and the logical address topic.
In some embodiments, the monitoring data may include the following fields: monitoring item names, monitoring item marks, monitoring item acquisition periods, unique marks of monitored CDN equipment, monitoring item acquisition values and timestamps corresponding to monitoring data and representing acquisition time; the time series data display module 60 may be specifically configured to:
executing a Structured Query Language (SQL) aggregation statement, aggregating the monitoring item acquisition values meeting the conditions every second preset time within a first preset time according to a specified monitoring item name metric or a specified monitoring item name metric and a monitoring item tag, obtaining the aggregation data of the current sampling time, and drawing a curve graph according to the aggregation data of all sampling times within the first preset time; the first preset time length is longer than the second preset time length;
for each drawn curve graph, when the aggregate data of a certain sampling moment is greater than or equal to a threshold value preset for the curve graph, alarming is carried out, and the alarming mode comprises the following steps: and marking an abnormal area on the graph, and sending a mail to a preset mail server for receiving alarm information or sending the alarm information to a specified social media account.
In some embodiments, the timing database 50 may include: infiluxdb or open-tsdb; the message queue module includes: kafka clusters, rabbitMQ, ActiveMQ, ZeroMQ, RocktetMQ, or redis; this collect and deposit module includes: a stash cluster; the time series data presentation module comprises a visualization tool grafana.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the data monitoring method for a CDN device in a content delivery network as described above.
The above technical solution of the embodiment of the present invention is described in more detail below:
the embodiment of the invention adopts brand-new technical consideration and brand-new scheme design, so that the logic is simpler, and the mode of storing the data into the database is changed into the mode of time sequence databases such as influxdb and the like.
Fig. 3 is an exemplary architecture diagram of an embodiment of the present invention. As shown in fig. 3, as an example, the present technical solution may include: data acquisition, data reporting, data warehousing and alarm display.
(1) Data acquisition
The client agent is responsible for collecting cdn application data (e.g. number of logs, request number, connection number, number of various status codes, etc. of the device, and reporting the application data to an interface of a data transmission component transfer through an http post request, where the transfer is a transmission component that receives the agent request and converts the http request into a kafka message, and the collection and reporting can be performed in the following manner, for example. The full name of HTTP is Hyper Text Transfer Protocol, hypertext Transfer Protocol. The HTTP protocol is a transfer protocol for transferring hypertext data from a network to a local browser, which ensures efficient and accurate transfer of hypertext documents.
Acquisition mode 1: the agent obtains an acquisition item through a manual definition configuration file, wherein the acquisition item comprises: acquiring an index metric, marking Tags, acquiring a period step, and uniquely marking device and time by equipment, acquiring corresponding data at regular time through an acquisition item, and then sending the data to a transfer through an http request. This approach is primarily used in scenarios where data can be obtained from outside the captured application.
The collection mode 2: other programs or scripts (e.g., nginx, or scripts written by other technicians) on the cdn device send the collected data to the agent listening port through an http post request by regularly reporting, and the agent forwards the request for reporting data to the transfer. The method is mainly used for sending http requests to agents between the interiors of other programs, for example, a method of embedding a code into an internal program.
The data collected were as follows:
nginx _ status, which represents the name of the monitoring item;
tags:5xx, monitor entry markers; used as a subdivision metric, may not be written, tags may be empty;
step 60, representing the acquisition period of the monitoring items;
device 192.168.1.1, representing a monitoring Device unique signature;
value 99, representing the collection Value of the monitoring item;
time 1593748649, unix timestamp indicating the acquisition Time.
(2) Data reporting
Firstly, a topic is created in a distributed message system kafka, when a transfer is started, an IP address and a port of a kafka cluster and a theme topic are obtained through a configuration file, a tcp (Transmission Control Protocol) long connection is established according to the address and the theme of the kafka, monitoring data are written into the corresponding topic of the kafka cluster of the distributed message system through the tcp connection, messages are stored according to partitions by the kafka, when a plurality of consumers exist, the messages are averagely distributed to different partitions of each consumer to ensure load balance of the consumers, when one of the consumers is down or a network is abnormal, the partition corresponding to the consumer is redistributed after a certain timeout time, and the process is called a repance, so that high availability of the kafka cluster is ensured. The Kafka cluster is a distributed, partition-supported (partition), multi-copy (replenica), zookeeper-based coordinated distributed messaging system. Topic: a topic, which represents a category of messages. A publisher publishes a message that must specify topic, and a subscriber can consume the message by subscribing to topic. Topic represents a category of data and a Topic may be considered a category of messages. In some alternative embodiments, Kafka may be replaced by a Message Queue (MQ), REDIS, or the like.
When there are multiple transfers, it is necessary to configure the ip address port and topic of the same kafka, and it is guaranteed that the same topic is written, and kafka can be replaced by other message queues, such as rabbitMQ, ActiveMQ, ZeroMQ, rocktmq, redis, etc.
(3) Data warehousing
Reading data from kafka by a stack (the stack is a component that consumes kafka data and defines the data to be put in a warehouse according to the rules of different time-series databases), writing the data into an infixtb or open-tsdb database, and ensuring that a group of transfer consumes kafka data is 1 share by using the same group id by a plurality of stacks. The group is to distinguish multiple consumers consuming the same topic without duplication, and if the same group is used, the data is one copy, and if multiple groups are used, the data is multiple copies.
(4) Alarm display
Configuring the data source as influxdb or open-tsdb through the grafana. InfluxDB is an open source time sequence database developed by InfluxData. It is written by Go and focuses on querying and storing time-ordered data with high performance. The grafana is an open source application written by adopting a go language, is mainly used for visualizing and displaying large-scale index data, and is a time sequence data display tool in network architecture and application analysis. OpenTSDB is a distributed, scalable time-series database based on Hbase, suitable for data storage for monitoring classes. OpenTSDB is mainly suitable for storing data with time characteristics, such as monitoring data, temperature change data, and the like, according to data with time characteristics and requirements, and provides a specific tool for querying and the like.
Technicians map the grafana pages according to the standard and tags according to the requirement, each graph can aggregate data according to the sql statement, and set a threshold alarm for each graph, wherein the alarm and the mapping use the same data, and the alarm mode includes but is not limited to abnormal area marking, mail sending, WeChat sending and the like. Here, the data source is exemplified as the influxdb.
Example aggregation statements:
configuring influxdb as a data source in grafana, and configuring to execute the following statements can obtain data aggregated by metric _ nginx _ status and tags _ 5xx per minute in the last three hours, and a graph can be drawn directly after configuration is completed:
SELECT sum("value")FROM"statsh_kafka"WHERE("metric"＝'nginx_status'AND"tags"＝’5xx’)AND time>＝now()-3h GROUP BY time(1m)fill(null)。
note: statsh _ kafka is a table name for infiluxdb, and stash writes data to the table and grafana reads data from the table.
Examples of alarms are: the alert module of the grafana supports simple policy alarming, for example, when the value is greater than or less than a certain value, alarming is performed, a red abnormal area is marked on a graph corresponding to the index, if an email needs to be sent, the email server and a corresponding email address of an alarm person need to be connected by the email server, and the alarming modes are various and are not described herein.
Fig. 4 is a graph illustrating an embodiment of the present invention. In fig. 4, the axis of abscissa indicates time, the axis of ordinate indicates a monitoring item, here, bandwidth, and the unit is bit, the axis of ordinate may also be other monitoring items, and the units of different monitoring items are not consistent. In fig. 4, the uppermost curve is obtained by aggregating the three curves below it (corresponding to the three devices to be monitored, respectively) in the direction of increasing along the ordinate axis. As can be seen from fig. 4, the same monitoring items of a plurality of devices are grouped into a line, i.e., the uppermost curve in fig. 4.
The embodiment of the invention has the advantages that:
compared with the prior art, the technical scheme of the embodiment of the invention has simple and light logic and high speed, adopts a mode of directly warehousing the message queue, does not need to use functions of hadoop data cleaning and the like, and has higher aggregation speed than the prior technical scheme. The whole system has no core single point, is easy to operate and maintain, is easy to deploy and can be horizontally expanded.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device, the electronic device and the readable storage medium embodiments, since they are substantially similar to the method embodiments, the description is simple, and the relevant points can be referred to the partial description of the method embodiments.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (11)
1. A data monitoring method for CDN (content delivery network) equipment is characterized by comprising the following steps:
the method comprises the following steps that a data acquisition module acquires monitoring data of CDN equipment to be monitored and reports the monitoring data to a transmission module;
the transmission module writes the monitoring data into a message queue module;
the collecting and storing module reads the monitoring data from the message queue module and writes the monitoring data into a time sequence database;
and the time sequence data display module configures a data source into the time sequence database and sends out an alarm according to the monitoring data in the time sequence database.
2. The method according to claim 1, wherein the data acquisition module acquires monitoring data of the CDN device to be monitored, and reports the monitoring data to the transmission module, specifically including:
the data acquisition module acquires a monitoring item through a configuration file, acquires corresponding monitoring data through the monitoring item at regular time, and then sends the monitoring data to the transmission module through an http request; or,
the data acquisition module acquires monitoring data reported by other programs or scripts at regular time through a monitoring port of the data acquisition module, and then sends a request carrying the monitoring data to the transmission module.
3. The method according to claim 1, wherein the transmitting module writes the monitoring data into a message queue module, specifically comprising:
when the transmission module is started, the IP address and the logical address topic of the message queue module are obtained through a configuration file;
and the transmission module writes the monitoring data into the message queue module according to the IP address and the logical address topic of the message queue module.
4. The method of claim 1, wherein the monitoring data comprises the following fields: monitoring item name metric, monitoring item marks tag, monitoring item acquisition period, unique mark of monitored CDN equipment, monitoring item acquisition value and a timestamp which is corresponding to monitoring data and represents acquisition time;
the time sequence data display module sends out an alarm according to the monitoring data in the time sequence database, and the time sequence data display module specifically comprises the following steps:
the time sequence data display module executes a Structured Query Language (SQL) aggregation statement, aggregates monitoring item acquisition values meeting conditions every second preset time length according to a specified monitoring item name metric or a specified monitoring item name metric and a monitoring item tag within a first preset time length to obtain aggregate data of the current sampling time, and draws a curve graph according to the aggregate data of all sampling time within the first preset time length; the first preset time length is longer than the second preset time length;
for each drawn curve graph, when the aggregate data of a certain sampling moment is greater than or equal to a threshold value preset for the curve graph, alarming is carried out, and the alarming mode comprises the following steps: and marking an abnormal area on the graph, and sending a mail to a preset mail server for receiving alarm information or sending the alarm information to a specified social media account.
5. The method of any of claims 2-4, wherein the time series database comprises: infiluxdb or open-tsdb; the message queue module includes: kafka clusters, rabbitMQ, ActiveMQ, ZeroMQ, RocktetMQ, or redis; the collection and storage module includes: a stash cluster; the time series data display module comprises a visualization tool grafana.
6. A data monitoring system for a content delivery network CDN device, the system comprising:
the data acquisition module is used for acquiring monitoring data of the CDN equipment to be monitored and reporting the monitoring data to the transmission module;
the transmission module is used for writing the monitoring data into a message queue module;
the collecting and storing module is used for reading the monitoring data from the message queue module and writing the monitoring data into a time sequence database;
and the time sequence data display module is used for configuring a data source into the time sequence database and sending out an alarm according to the monitoring data in the time sequence database.
7. The system of claim 6, wherein the data acquisition module is specifically configured to:
acquiring a monitoring item through a configuration file, acquiring corresponding monitoring data through the monitoring item at regular time, and sending the monitoring data to a transmission module through an http request; or,
and acquiring monitoring data reported by other programs or scripts at regular time through a monitoring port of the monitoring module, and sending a request carrying the monitoring data to the transmission module.
8. The system of claim 6, wherein the transmission module is specifically configured to:
the method comprises the steps that when the system is started, the IP address and the logical address topic of a message queue module are obtained through a configuration file;
and writing the monitoring data into a message queue module of the distributed message system according to the IP address and the logical address topic of the message queue module.
9. The system of claim 6, wherein the monitoring data comprises the following fields: monitoring item names, monitoring item marks, monitoring item acquisition periods, unique marks of monitored CDN equipment, monitoring item acquisition values and timestamps corresponding to monitoring data and representing acquisition time; the time sequence data display module is specifically configured to:
executing a Structured Query Language (SQL) aggregation statement, aggregating the monitoring item acquisition values meeting the conditions every second preset time within a first preset time according to a specified monitoring item name metric or a specified monitoring item name metric and a monitoring item tag, obtaining the aggregation data of the current sampling time, and drawing a curve graph according to the aggregation data of all sampling times within the first preset time; the first preset time length is longer than the second preset time length;
for each drawn curve graph, when the aggregate data of a certain sampling moment is greater than or equal to a threshold value preset for the curve graph, alarming is carried out, and the alarming mode comprises the following steps: and marking an abnormal area on the graph, and sending a mail to a preset mail server for receiving alarm information or sending the alarm information to a specified social media account.
10. The system according to any one of claims 7-9, wherein the time series database comprises: infiluxdb or open-tsdb; the message queue module includes: kafka clusters, rabbitMQ, ActiveMQ, ZeroMQ, RocktetMQ, or redis; the collection and storage module includes: a stash cluster; the time series data display module comprises a visualization tool grafana.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method for data monitoring of a CDN device for a content delivery network as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010860803.XA CN112039726A (en) | 2020-08-25 | 2020-08-25 | Data monitoring method and system for content delivery network CDN device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010860803.XA CN112039726A (en) | 2020-08-25 | 2020-08-25 | Data monitoring method and system for content delivery network CDN device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112039726A true CN112039726A (en) | 2020-12-04 |
Family
ID=73581144
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010860803.XA Pending CN112039726A (en) | 2020-08-25 | 2020-08-25 | Data monitoring method and system for content delivery network CDN device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112039726A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112817820A (en) * | 2021-01-29 | 2021-05-18 | 北京思特奇信息技术股份有限公司 | JAVA component-oriented source code deep monitoring method and system |
| CN113064789A (en) * | 2021-03-15 | 2021-07-02 | 上海浦东发展银行股份有限公司 | Index data acquisition system and method based on configuration center and storage medium |
| CN113672476A (en) * | 2021-10-25 | 2021-11-19 | 煜象科技(杭州)有限公司 | A method for counting the running time of multiple terminal software |
| CN113810463A (en) * | 2021-08-06 | 2021-12-17 | 新浪网技术(中国)有限公司 | Configuration method and system for issuing CDN server |
| CN114095384A (en) * | 2021-10-29 | 2022-02-25 | 新浪网技术(中国)有限公司 | CDN fused service quality real-time display method and system |
| CN114690019A (en) * | 2020-12-29 | 2022-07-01 | 北京达佳互联信息技术有限公司 | An information monitoring method, device and system of an integrated circuit |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103024060A (en) * | 2012-12-20 | 2013-04-03 | 中国科学院深圳先进技术研究院 | Open type cloud computing monitoring system for large scale cluster and method thereof |
| WO2016044413A1 (en) * | 2014-09-16 | 2016-03-24 | CloudGenix, Inc. | Methods and systems for business intent driven policy based network traffic characterization, monitoring and control |
| CN109634519A (en) * | 2018-11-28 | 2019-04-16 | 平安科技（深圳）有限公司 | The method and storage medium of electronic device, monitoring data caching |
| CN109800129A (en) * | 2019-01-17 | 2019-05-24 | 青岛特锐德电气股份有限公司 | A kind of real-time stream calculation monitoring system and method for processing monitoring big data |
| CN110430260A (en) * | 2019-08-02 | 2019-11-08 | 哈工大机器人（合肥）国际创新研究院 | Robot cloud platform based on big data cloud computing support and working method |
| CN111200637A (en) * | 2019-12-20 | 2020-05-26 | 新浪网技术(中国)有限公司 | Cache processing method and device |
-
2020
- 2020-08-25 CN CN202010860803.XA patent/CN112039726A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103024060A (en) * | 2012-12-20 | 2013-04-03 | 中国科学院深圳先进技术研究院 | Open type cloud computing monitoring system for large scale cluster and method thereof |
| WO2016044413A1 (en) * | 2014-09-16 | 2016-03-24 | CloudGenix, Inc. | Methods and systems for business intent driven policy based network traffic characterization, monitoring and control |
| CN109634519A (en) * | 2018-11-28 | 2019-04-16 | 平安科技（深圳）有限公司 | The method and storage medium of electronic device, monitoring data caching |
| CN109800129A (en) * | 2019-01-17 | 2019-05-24 | 青岛特锐德电气股份有限公司 | A kind of real-time stream calculation monitoring system and method for processing monitoring big data |
| CN110430260A (en) * | 2019-08-02 | 2019-11-08 | 哈工大机器人（合肥）国际创新研究院 | Robot cloud platform based on big data cloud computing support and working method |
| CN111200637A (en) * | 2019-12-20 | 2020-05-26 | 新浪网技术(中国)有限公司 | Cache processing method and device |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114690019A (en) * | 2020-12-29 | 2022-07-01 | 北京达佳互联信息技术有限公司 | An information monitoring method, device and system of an integrated circuit |
| CN112817820A (en) * | 2021-01-29 | 2021-05-18 | 北京思特奇信息技术股份有限公司 | JAVA component-oriented source code deep monitoring method and system |
| CN113064789A (en) * | 2021-03-15 | 2021-07-02 | 上海浦东发展银行股份有限公司 | Index data acquisition system and method based on configuration center and storage medium |
| CN113810463A (en) * | 2021-08-06 | 2021-12-17 | 新浪网技术(中国)有限公司 | Configuration method and system for issuing CDN server |
| CN113810463B (en) * | 2021-08-06 | 2024-03-05 | 新浪技术（中国）有限公司 | Method and system for configuring delivery CDN server |
| CN113672476A (en) * | 2021-10-25 | 2021-11-19 | 煜象科技(杭州)有限公司 | A method for counting the running time of multiple terminal software |
| CN113672476B (en) * | 2021-10-25 | 2022-04-29 | 煜象科技(杭州)有限公司 | A method for counting the running time of multiple terminal software |
| CN114095384A (en) * | 2021-10-29 | 2022-02-25 | 新浪网技术(中国)有限公司 | CDN fused service quality real-time display method and system |
| CN114095384B (en) * | 2021-10-29 | 2023-03-24 | 新浪网技术(中国)有限公司 | CDN fused service quality real-time display method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112039726A (en) | Data monitoring method and system for content delivery network CDN device | |
| CN108874640B (en) | Cluster performance evaluation method and device | |
| US11921791B2 (en) | Cardinality of time series | |
| CN111984499B (en) | Fault detection method and device for big data cluster | |
| CN112131073B (en) | Monitoring method and system of server | |
| US10360193B2 (en) | Method and apparatus for smart archiving and analytics | |
| CN112507029B (en) | Data processing system and data real-time processing method | |
| CN110784419A (en) | Data visualization method and system for railway electric affairs | |
| CN109726074A (en) | Log processing method, device, computer equipment and storage medium | |
| CN111585840B (en) | Service resource monitoring method, device and equipment | |
| CN110232010A (en) | A kind of alarm method, alarm server and monitoring server | |
| CN101707632A (en) | Method for dynamically monitoring performance of server cluster and alarming real-timely | |
| CN105718351A (en) | Hadoop cluster-oriented distributed monitoring and management system | |
| CN112199394A (en) | Alarm information push method, system, intelligent terminal and storage medium | |
| US20110320971A1 (en) | Cross-domain business service management | |
| CN110598051A (en) | A monitoring system, method and device for electric power industry | |
| CN115314452B (en) | Monitoring indicator collection and storage method, system and storage medium based on message queue | |
| CN114510708A (en) | Real-time data warehouse construction, abnormal detection method, device, equipment and product | |
| US20210156401A1 (en) | Vehicle with a Boom Comprising a Hydraulic Control Circuit with a Load Control Valve | |
| CN117768362A (en) | A network diagnosis and early warning method, device and medium based on eBPF | |
| CN114546610B (en) | A massive data distributed desensitization device | |
| CN113138896B (en) | A method, device and equipment for monitoring application operation status | |
| CN113051222B (en) | Log storage method, device, computer readable storage medium and electronic equipment | |
| CN114816907B (en) | Delay information acquisition method and device, computing device and readable storage medium | |
| CN116166711A (en) | Statistical method and device for device liveness, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201204 |
|
| RJ01 | Rejection of invention patent application after publication |