CN107634848A - A kind of system and method for collection analysis network equipment information - Google Patents
A kind of system and method for collection analysis network equipment information Download PDFInfo
- Publication number
- CN107634848A CN107634848A CN201710666927.2A CN201710666927A CN107634848A CN 107634848 A CN107634848 A CN 107634848A CN 201710666927 A CN201710666927 A CN 201710666927A CN 107634848 A CN107634848 A CN 107634848A
- Authority
- CN
- China
- Prior art keywords
- dimension
- index
- data
- network performance
- polymerization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000004458 analytical method Methods 0.000 title claims abstract description 30
- 238000006116 polymerization reaction Methods 0.000 claims abstract description 45
- 238000012806 monitoring device Methods 0.000 claims abstract description 11
- 201000003373 familial cold autoinflammatory syndrome 3 Diseases 0.000 claims abstract description 9
- 238000004422 calculation algorithm Methods 0.000 claims description 61
- 230000004931 aggregating effect Effects 0.000 claims description 27
- 230000006835 compression Effects 0.000 claims description 16
- 238000007906 compression Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 16
- 235000019580 granularity Nutrition 0.000 claims description 14
- 238000012544 monitoring process Methods 0.000 claims description 11
- 239000003638 chemical reducing agent Substances 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 5
- 230000002776 aggregation Effects 0.000 claims description 4
- 238000004220 aggregation Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 4
- 230000003139 buffering effect Effects 0.000 claims description 3
- 239000010902 straw Substances 0.000 claims description 2
- 238000012545 processing Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002224 dissection Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a kind of system and method for collection analysis network equipment information, collector is by the message data unserializing received into key assignments plaid matching formula data；Key-value pair formatted data is converted to network performance unified metric data by analyzer；Are entered into row index dimension by network performance unified metric data for polymerizer and time dimension polymerize；Network performance unified metric data output after polymerization to network performance monitoring device, or write-in disk are filed initial data by follower.The present invention solves the problems, such as that incomplete problem and packet loss problem are analyzed in prior art blended data source, agreement specification, and message data can be achieved, and supports backtracking statistics.
Description
Technical field
The present invention relates to performance monitoring field, in particular it relates to a kind of system of collection analysis network equipment information and side
Method.
Background technology
With the development of social science and technology, network has turned into the indispensable core composition portion of enterprise, especially financial company
Point.
The purpose safeguarded for the network operation, general bank, stock trader, Virtual network operator, large-scale Internet firm all can
On-premise network performance monitoring platform, i.e. NPM (Network Performance Manager).These platforms are built upon can
Obtain on the basis with network flow calculation data.Gathering these data can be entered by the way of Direct Acquisition network packet
OK.But under some network environments, consider for information security, user is not intended to monitor supervision platform energy Direct Acquisition network data
Bag, which prevent the progress of this mode.Fortunately, some network equipments can export the network flow by its own at present
Measure statistical information.These information would generally be exported with the message of some agreements such as IPFIX, NETFLOW and SFLOW.So
In such a case, network monitoring platform can realize data acquisition by this indirect mode, avoid information security and ask
Topic.
IPFIX (IP Flow Information Export) is a kind of protocol specification, and it, which is produced, is derived to various IP
The network equipment such as router, interchanger etc. obtain the demand of its IP flow statistic.How IPFIX standards define IP stream informations
It is formatted, and how is sent to information collector.NetFlow is a kind of protocol specification based on IPFIX, and is done
Some improvement.SFlow is also a kind of similar protocol specification in way.Compared with IPFIX and NetFlow statistics IP stream informations,
SFlow is more focused on the sampling of statistics network packet.
Under real network environment, device category, model are often different, easily cause this kind of protocol massages data source simultaneously
The situation deposited.Existing acquisition analysis system can not support all these agreements simultaneously, though or support can not be by this all association
The data for discussing type carry out merger by index implication, unified to present.Which increase the complexity of network monitoring implementation, it is necessary to put into
More manpowers into monitor different types of network equipment respectively originally with software/hardware.
Existing acquisition analysis system is relatively simple to these protocol massages statistical methods.The index of the offer of these agreements
Species is very abundant, and existing analysis method can not cover all these indexs, often in order to which a certain analysis result only counts
Certain several index.
These agreements are transmitted based on UDP.UDP transmission has quickly, but the characteristics of unreliable, easy packet loss.
Some receiving terminal systems usually can cause statistical information to be lost because of packet loss, cause statistical result and actual conditions difference huge,
Final supervisor can do the judgement and measure to make mistake accordingly.
Found according to retrieval result：
The patent of invention of Application No. 201610183366.6 disclose a kind of speed of calculating network flow method and
Device, according to the home record time of the received netflow flow record and cut-off record time, it is determined that described
The measurement period that the record duration of netflow flow record is covered, the measurement period include period 1 and/or second week
Phase；According to default rule, the NetFlow that the byte number in the netflow flow record is counted on to the determination flows
In the measurement period that the record duration of record is covered；According to what is counted on respectively in the period 1 and second round
Total bytes in netflow flow record calculate the network flow in the period 1 and in second round with the cycle duration
The speed of amount.Using the method for the embodiment of the present application, network traffics are calculated based on netflow flow record exactly in realization
Speed.The invention function is single, can only network flow calculation, in addition can not count other network indexes.
The patent of invention of Application No. 201210091099.1 discloses a kind of network traffics analysis system and method, according to
The dependence and data volume situation in each word domain, build multidimensional structure, according to device IP, TOS, agreement in netflow records
Type, destination address are high-order, the order foundation of inflow equipment end slogan, destination interface, destination address, source port, source address is more
Structure is tieed up, the discharge record in multidimensional structure is traveled through according to information, made in ergodic process for the more node of child node
With hash algorithm, other nodes use array, find the information to match and are overlapped operation, when reaching one minute, write-in
File, caching is emptied, restarts recording data information.The invention is only limitted to NetFlow packet parsing, it is impossible to dissection process
In addition other type of messages.
The content of the invention
For in the prior art the defects of, can receive, analyze and escape information it is an object of the invention to provide one kind, and
It is delivered to the system and method for the collection analysis network equipment information of network performance monitoring device.
According to a kind of system of collection analysis network equipment information provided by the invention, including：
Collector：By the message data unserializing received into key assignments plaid matching formula data；
Analyzer：Key-value pair formatted data is converted into network performance unified metric data；
Polymerizer：Network performance unified metric data are entered into row index dimension and time dimension polymerization；
Follower：By the network performance unified metric data output after polymerization to network performance monitoring device, or write-in disk
File initial data.
Preferably, the collector includes：
Socket receiving terminals：The network port is monitored, reads the message data that the network equipment sends over, and write internal memory and delay
Rush pond；
Memory buffer pond：The speed buffering of message data is provided；
Unserializing device：Message data is read from memory buffer pond, the model agreement id field in message data
Inquiry obtains corresponding model agreement from model agreement storehouse, further according to model agreement by message data unserializing into key-value pair
Formatted data, and it is sent to the analyzer；
Model agreement storehouse：The addition and inquiry service of model agreement are provided, and are responsible for the storage to model agreement data.
Preferably, the analyzer includes：
IP flow objects establish module：The key-value pair formatted data from the collector is read, is established according to five-tuple
IP flow objects, it is stored in cache table；
Escape algorithm queries module：According to the field in IP flow objects, to the inquiry acquisition pair of index escape Algorithm mapping table
The escape algorithm of field is answered, escape algorithm is uniquely determined by protocol type and field name；
IP flow object meaning transferring modules：The field in IP flow objects is traveled through, field is converted to by net according to field escape algorithm
Network performance unified metric data；
IP flow object output modules：IP flow objects after index escape are sent to the polymerizer.
Preferably, the polymerizer includes：
Index dimension statistic unit requestor builds module：The configuration of index dimension is read, builds index dimension statistic unit
Requestor, corresponding to index dimension statistic unit requestor is positioned or created by reading the dimension index in IP flow objects
Index dimension statistic unit；
IP flow object distribute modules：According to the dimension of configuration, IP flow objects are created and distributed by service index dimension requestor
To corresponding index dimension statistic unit；
Aggregating algorithm acquisition module：Network performance unified metric data in IP flow objects, it polymerize to index dimension
Inquiry obtains the aggregating algorithm of corresponding index in Algorithm mapping table, and the aggregating algorithm of corresponding index is unique true by the configuration of index dimension
It is fixed；
Index dimension aggregation module：The network performance unified metric data in IP flow objects are traveled through, use corresponding index
Aggregating algorithm enters the polymerization of row index dimension；
Time dimension statistic unit requestor builds module：Read more time granularity configurations, structure time dimension statistics
Unit requestor；
Time dimension aggregation module：According to more time granularities of configuration, usage time dimension statistic unit requestor will
Index dimension statistic unit is assigned to multiple time dimension aggregate statistics units, using with index dimension identical aggregating algorithm,
Network performance unified metric data after polymerizeing to index dimension carry out time dimension polymerization again；
Output module after polymerization：Network performance unified metric data after the polymerization of extraction time dimension, are sent to described defeated
Go out device.
Preferably, the follower includes：
Serialize device：Achievement data sequence chemical conversion byte stream after time dimension is polymerize；
Compressor reducer：Byte stream is subjected to streaming compression；
Network follower：Byte stream after compression is sent to network performance monitoring platform；
File follower：Byte stream after compression is write into filing data file；
Index：The filing data document creation information of the file follower is indexed, creates and updates index
File.
According to a kind of method of collection analysis network equipment information provided by the invention, including：
Collection step：By the message data unserializing received into key assignments plaid matching formula data；
Analytical procedure：Key-value pair formatted data is converted into network performance unified metric data；
Polymerization procedure：Network performance unified metric data are entered into row index dimension and time dimension polymerization；
Export step：By the network performance unified metric data output after polymerization to network performance monitoring device, or write-in magnetic
Disk files initial data.
Preferably, the collection step includes：
Memory buffer step：The network port is monitored, reads the message data that the network equipment sends over, and write internal memory and delay
Rush pond；
Unserializing step：Message data is read from memory buffer pond, the model agreement ID words in message data
Section inquiry from model agreement storehouse obtains corresponding model agreement, further according to model agreement by message data unserializing into key assignments
To formatted data.
Preferably, the analytical procedure specifically includes：
IP flow object establishment steps：The key-value pair formatted data from the collector is read, is established according to five-tuple
IP flow objects, it is stored in cache table；
Escape algorithm queries step：According to the field in IP flow objects, to the inquiry acquisition pair of index escape Algorithm mapping table
The escape algorithm of field is answered, escape algorithm is uniquely determined by protocol type and field name；
IP flow object escape steps：The field in IP flow objects is traveled through, field is converted to by net according to field escape algorithm
Network performance unified metric data；
IP flow objects export step：IP flow objects after index escape are sent to the polymerization procedure.
Preferably, the polymerization procedure specifically includes：
Index dimension statistic unit requestor construction step：The configuration of index dimension is read, builds index dimension statistic unit
Requestor, corresponding to index dimension statistic unit requestor is positioned or created by reading the dimension index in IP flow objects
Index dimension statistic unit；
IP flow object allocation steps：According to the dimension of configuration, IP flow objects are created and distributed by service index dimension requestor
To corresponding index dimension statistic unit；
Aggregating algorithm obtaining step：Network performance unified metric data in IP flow objects, it polymerize to index dimension
Inquiry obtains the aggregating algorithm of corresponding index in Algorithm mapping table, and the aggregating algorithm of corresponding index is unique true by the configuration of index dimension
It is fixed；
Index dimension polymerization procedure：The network performance unified metric data in IP flow objects are traveled through, use corresponding index
Aggregating algorithm enters the polymerization of row index dimension；
Time dimension statistic unit requestor construction step：Read more time granularity configurations, structure time dimension statistics
Unit requestor；
Time dimension polymerization procedure：According to more time granularities of configuration, usage time dimension statistic unit requestor will
Index dimension statistic unit is assigned to multiple time dimension aggregate statistics units, using with index dimension identical aggregating algorithm,
Network performance unified metric data after polymerizeing to index dimension carry out time dimension polymerization again；
Step is exported after polymerization：Network performance unified metric data after the polymerization of extraction time dimension, it is sent to output step
Suddenly.
Preferably, the output step includes：
Serialisation step：Network performance unified metric data sequence chemical conversion byte stream after time dimension is polymerize；
Compression step：Byte stream is subjected to streaming compression；
Network exports step：Byte stream after compression is sent to network performance monitoring platform；
File exports step：Byte stream after compression is write into filing data file；
Straw line step：Filing data document creation information is indexed, creates and updates index file.
Compared with prior art, the present invention has following beneficial effect：
1st, solves the problems, such as blended data source：System and method supports tri- kinds of IPFIX, NetFlow and sFlow simultaneously
Data source, and more data sources can be flexibly added according to client agreement, the quantity for receiving network equipment end is expansible；
2nd, solve agreement specification and analyze incomplete problem：System and method is carried out to all indexs of these agreements
Analysis, by escape, the method for merger is converted into a unified index set, allows all indexs of these agreements can be to network
O＆M plays value；
3rd, solves the problems, such as packet loss：High-performance packet capturing pattern is received and is combined by system and method with udp protocol, fully
Overcome the packet loss phenomenon caused by network interface card inadequate buffer space using the advantage of large space internal memory, there is provided a kind of reliable net
Network message sink, it ensure that the accuracy of statistical indicator analysis；
4th, message data can be achieved, supports backtracking statistics：When user pay close attention to a certain historical events it is disconnected, it is necessary to
During a kind of customized statistical method statistical history data, the system can with being run with batch mode and load archive data, and
Result of calculation is sent to network performance monitoring device and presented.
Brief description of the drawings
The detailed description made by reading with reference to the following drawings to non-limiting example, further feature of the invention,
Objects and advantages will become more apparent upon：
Fig. 1 is the module composition figure of the system of collection analysis network equipment information provided by the invention；
Fig. 2 is the internal structure schematic diagram of receiver provided by the invention；
Fig. 3 is the workflow diagram of analyzer provided by the invention；
Fig. 4 is the workflow diagram of polymerizer provided by the invention；
Fig. 5 is the internal structure schematic diagram of follower of the present invention.
Embodiment
With reference to specific embodiment, the present invention is described in detail.Following examples will be helpful to the technology of this area
Personnel further understand the present invention, but the invention is not limited in any way.It should be pointed out that the ordinary skill to this area
For personnel, without departing from the inventive concept of the premise, some changes and improvements can also be made.These belong to the present invention
Protection domain.
As shown in figure 1, according to a kind of system of collection analysis network equipment information provided by the invention, including：
Collector：By the message data unserializing received into key assignments plaid matching formula data；
Analyzer：Key-value pair formatted data is converted into network performance unified metric data；
Polymerizer：Network performance unified metric data are entered into row index dimension and time dimension polymerization；
Follower：By the network performance unified metric data output after polymerization to network performance monitoring device, or write-in disk
File initial data.
Shown in Fig. 2 is the internal structure schematic diagram of the receiver of the system, including：
1) Socket receiving terminals：The network port is monitored, reads the message that the network equipment sends over, and write memory buffer
Pond；
2) memory buffer pond：The speed buffering of message data is provided, solves Socket receiving velocities and unserializing device
Unserializing speed is unequal, causes network interface card cache overflow and the problem of packet loss.The big I of buffer pool can use money according to server
Source and CPU processing speed reasonable distributions；
3) unserializing device：Message is read from memory buffer pond, the model agreement id field in message is from agreement
Inquiry obtains corresponding model agreement in ATL, further according to model agreement by message unserializing into key assignments plaid matching formula data,
And it is sent to subsequent module (analyzer) processing；
4) model agreement storehouse：Management agreement template.The addition and inquiry service of model agreement are provided, and are responsible for agreement
The storage of template data, for the loading after system reboot to model agreement.For the addition of model agreement, there are two kinds of sources：
A) the model agreement data read from serializing device.If unserializing device runs into model agreement data,
The addition interface of invocation protocol ATL is then subjected to template addition；
B) derived model agreement data will be imported from the network equipment in advance from attendant.If system
Just come into operation, model agreement storehouse does not have template, and unserializing device can be caused can not to inquire about to obtain agreement mould corresponding to message
Plate and unserializing can not be carried out, this can cause loss of data.And this source imported in advance is with regard to that can solve the problem.
Shown in Fig. 3 is the analyzer of the system, by each field escape in message object data into network performance monitoring
The unified metric that device can identify.The system have studied index all in IPFIX, NetFlow and sFlow, and and internetworking
Field it can be contrasted needed for monitor, the one escape algorithm that has been every kind of index definition forms an escape algorithms library, built-in
In analyzer.Analyzer includes volume of data processing step：
Step 1：The key-value pair formatted data from receiver is read, extracts five-tuple.IP streams are established according to five-tuple
Object, it is stored in cache table.Five-tuple includes source IP address, purpose IP address, source port address, destination interface address and IP agreement
Species.Then IP flow objects are subjected to follow-up index calculating；
Step 2：According to the field in IP flow objects, the escape for obtaining corresponding field is inquired about to index escape Algorithm mapping table
Algorithm.Escape algorithm is uniquely determined by protocol type and field name；
Step 3：The field in IP flow object data is traveled through, field is converted to by network performance system according to field escape algorithm
One achievement data；
Step 4：IP flow objects after index escape are sent to polymerizer.
Shown in Fig. 4 is the polymerizer of the system, and the network performance unified metric data in IP flow objects are tieed up by index
Degree carries out aggregate statistics.Index dimension is configured as needed by network performance monitoring platform.On the basis of the polymerization of index dimension,
It can be polymerize according further to time dimension, so as to export the indicator-specific statistics value under different time granularity simultaneously.It is poly-
Clutch includes some column data processing steps：
Step 1：The configuration of index dimension is read, builds index dimension statistic unit requestor；Index dimension statistic unit is looked into
Index dimension statistics corresponding to asking device and positioning or creates by reading five-tuple in IP flow objects or other dimension indexs
Unit；
Step 2：According to the dimension of configuration, IP flow objects are created and are assigned to corresponding index by service index dimension requestor
Dimension statistic unit；
Step 3：Network performance unified metric data in IP flow objects, into index dimension aggregating algorithm mapping table
Inquiry obtains the aggregating algorithm of corresponding index.The aggregating algorithm of corresponding index is uniquely determined by the configuration of index dimension；
Step 4：The network performance unified metric data in IP flow objects are traveled through, are carried out using the aggregating algorithm of corresponding index
Index dimension polymerize；
Step 5；More time granularity configurations are read, build time dimension statistic unit requestor；
Step 6：According to more time granularities of configuration, usage time dimension statistic unit requestor counts index dimension
Unit is assigned to multiple time dimension aggregate statistics units, using with index dimension identical aggregating algorithm, index dimension is gathered
Network performance unified metric data after conjunction carry out time dimension polymerization again；
Step 7；Network performance unified metric data after the polymerization of extraction time dimension, are sent to follower.
Shown in Fig. 5 is the follower of the system, including：
Serialize device：Achievement data sequence is melted into byte stream, and is conveyed to compressor reducer；
Compressor reducer：Byte stream is subjected to streaming compression, and the byte stream after compression is conveyed to network follower and file
Follower；
Network follower：Byte stream is sent to network performance monitoring platform, the latter is presented achievement data in real time；
File follower：Byte stream is write into filing data file, during establishment file, file needs to use index to carry out
Index；
Index：The filing data document creation information of the file follower is indexed, creates and updates index
File, to accelerate file search speed during land parcel change trace statistics.
It is as follows that the present invention provides a specific implementation：
First, collector
1) Socket receiving terminals require high-performance, avoid the packet loss caused by receiving slowly.So independent operating program.Due to
Processing logic is simple and requires high-performance, is realized so being developed using the low-level language of such as C language.Set for each network
Standby data source, one Socket receiving terminal of independent operating；
2) memory buffer pond is realized using memory sharing technology, realizes that the data of receiving terminal and processing end heterogeneous platform are handed over
Change；It is independent to distribute a memory buffer pond for each Socket receiving terminal；
3) unserializing device is realized respectively according to different agreement, and standard is that the Internet Engineering Task group of corresponding agreement is solicited
Opinion original text, i.e. Request For Comments (RFC) document.For example, IPFIX consensus standards refer to https://
Tools.ietf.org/rfc/rfc5102.txt, NetFlow consensus standard refer to https://www.ietf.org/rfc/
rfc3954.txt.During system operation, for each memory buffer pond, one unserializing device of independent operating, unserializing device
Set by actual message protocol type.
4) model agreement storehouse due to data volume it is smaller, so using SQLite database realizings.In order to accelerate template query
Speed, each query SQL ite databases are avoided, set up memory cache, only retain a number of template number inquired about recently
According to.
2nd, analyzer
1) for one section of new message, the IP flow objects corresponding to establishment in IP flow object cache tables.IP flow object cache tables
Realized using Hash table, the key of Hash table is calculated using five-tuple field to be obtained, and is replied with representing in an IP.One IP stream pair
Image data includes the total amount for the multistage message data for belonging to an IP reply in intervals.Time interval is by application definition
The report cycle determine.Once reaching the report cycle, IP flow objects just are conveyed into follow-up polymerizer is handled；
2) escape algorithms library is by analyzing protocol fields and network performance monitoring device index implication, by establishing mapping relations
Set.Such as IPFIX agreements, exist such as the mapping of table 1：
| IPFIX indexs | Network performance monitoring device index |
| monitoringIntervalStartMilliSeconds | Timestamp |
| vlanId | Vlan |
| protocolIdentifier | IP agreement |
| sourceIPv4Address | Source IP address |
| sourceTransportPort | Source port |
| destinationIPv4Address | Purpose IP address |
| destinationTransportPort | Destination interface |
| packetDeltaCount | Packet sum |
| octetDeltaCount | Total length of data packets |
| transactionCountDelta | Message sum |
The IPFIX index escape mapping tables (part) of table 1
3rd, polymerizer
1) configuration of index dimension includes IP to, IP ports to dimensions such as, IP, IP port, Apply Names, Vlan and MPLS；
2) index and dimension aggregating algorithm mapping table are determined by the configuration of index dimension.For example, it is being separately provided various indexs
In the case of dimension, there are the mapping relations such as table 2 in part index number：
2 indexs of table-aggregating algorithm mapping table (part)
In actual use, different index dimensions can be overlapped, and after superposition, change can be sent for indicator polymerization algorithm
Change.For example, after IP is superimposed Vlan dimensions to dimension, the aggregating algorithm of Vlan indexs can be turned by " the different Vlan quantity of statistics "
It is changed into " record Vlan values "；
3) time dimension can be arranged as required to the time granularity such as 1 minute, 15 minutes or 1 hour；When data source
Between stamp reach the integral point of corresponding time granularity, just by all statistic unit report outputs of time granularity caching to exporting
Device, while empty and carry out next round time cycle statistics.
4th, follower
1) serialization format is formulated by the system and network performance monitoring device joint consultation；
2) compressor reducer uses such as Snappy etc rapid data compress technique, and can accomplish that streaming is compressed；
3) network follower sends achievement data using ZeroMQ network transmission technologies, can realize high-property transmission, again
The management for transmission connection can be simplified；
4) file of file follower switching write-in per minute, while file designation is to be accurate to the timestamp of minute.One
All Files in hours period is deposited in be accurate to the file of hours stamp name.So, only by only passing through text
Part path is with regard to that can search out desired archive file.
5) index is except that can index the time, moreover it is possible to which some key indexs are indexed.For example, IP address.If it is desired to
Certain is recalled to the statistical information to be communicated between IP address, just can be by indexing the archive file involved by fast positioning, and carry out
Batch processing counts.
Above-mentioned technical proposal is employed, the present invention can simplify the deployment of data acquisition unit under hybrid network facility environment,
Integration networkses Network Performance Monitor data source, and packet loss is avoided, data reliability is high.
One skilled in the art will appreciate that except realizing system provided by the invention in a manner of pure computer readable program code
And its beyond each device, module, unit, device, completely can be by the way that method and step be carried out into programming in logic come the present invention
The system of offer and its each device, module, unit, device are with gate, switch, application specific integrated circuit, Programmable logical controller
The form of device and embedded microcontroller etc. realizes identical function.So system provided by the invention and its every device,
Module, unit, device are considered a kind of hardware component, and the device, the mould that are used to realize various functions to including in it
Block, unit, device can also be considered as the structure in hardware component；Can also be by device, module, the list for realizing various functions
Member, device, which are considered as, not only can be the software module of implementation method but also can be the structure in hardware component.
The specific embodiment of the present invention is described above.It is to be appreciated that the invention is not limited in above-mentioned
Particular implementation, those skilled in the art can make a variety of changes or change within the scope of the claims, this not shadow
Ring the substantive content of the present invention.In the case where not conflicting, the feature in embodiments herein and embodiment can any phase
Mutually combination.
Claims (10)
- A kind of 1. system of collection analysis network equipment information, it is characterised in that including：Collector：By the message data unserializing received into key assignments plaid matching formula data；Analyzer：Key-value pair formatted data is converted into network performance unified metric data；Polymerizer：Network performance unified metric data are entered into row index dimension and time dimension polymerization；Follower：By the network performance unified metric data output after polymerization to network performance monitoring device, or write-in disk filing Initial data.
- 2. the system of collection analysis network equipment information according to claim 1, it is characterised in that the collector bag Include：Socket receiving terminals：The network port is monitored, reads the message data that the network equipment sends over, and write memory buffer Pond；Memory buffer pond：The speed buffering of message data is provided；Unserializing device：Message data is read from memory buffer pond, the model agreement id field in message data is from association Inquiry obtains corresponding model agreement in view ATL, further according to model agreement by message data unserializing into key assignments plaid matching formula Data, and it is sent to the analyzer；Model agreement storehouse：The addition and inquiry service of model agreement are provided, and are responsible for the storage to model agreement data.
- 3. the system of collection analysis network equipment information according to claim 1, it is characterised in that the analyzer bag Include：IP flow objects establish module：The key-value pair formatted data from the collector is read, IP streams are established according to five-tuple Object, it is stored in cache table；Escape algorithm queries module：According to the field in IP flow objects, inquired about to index escape Algorithm mapping table and obtain corresponding word The escape algorithm of section, escape algorithm are uniquely determined by protocol type and field name；IP flow object meaning transferring modules：The field in IP flow objects is traveled through, field is converted to by internetworking according to field escape algorithm Can unified metric data；IP flow object output modules：IP flow objects after index escape are sent to the polymerizer.
- 4. the system of collection analysis network equipment information according to claim 3, it is characterised in that the polymerizer bag Include：Index dimension statistic unit requestor builds module：Read the configuration of index dimension, the statistic unit inquiry of structure index dimension Device, index dimension statistic unit requestor position or create by reading the dimension index in IP flow objects corresponding to index Dimension statistic unit；IP flow object distribute modules：According to the dimension of configuration, IP flow objects are created and are assigned to pair by service index dimension requestor The index dimension statistic unit answered；Aggregating algorithm acquisition module：Network performance unified metric data in IP flow objects, to index dimension aggregating algorithm Inquiry obtains the aggregating algorithm of corresponding index in mapping table, and the aggregating algorithm of corresponding index is uniquely determined by the configuration of index dimension；Index dimension aggregation module：The network performance unified metric data in IP flow objects are traveled through, use the polymerization of corresponding index Algorithm enters the polymerization of row index dimension；Time dimension statistic unit requestor builds module：More time granularity configurations are read, build time dimension statistic unit Requestor；Time dimension aggregation module：According to more time granularities of configuration, usage time dimension statistic unit requestor is by index Dimension statistic unit is assigned to multiple time dimension aggregate statistics units, using with index dimension identical aggregating algorithm, to refer to Network performance unified metric data after mark dimension polymerization carry out time dimension polymerization again；Output module after polymerization：Network performance unified metric data after the polymerization of extraction time dimension, are sent to the follower.
- 5. the system of collection analysis network equipment information according to claim 1, it is characterised in that the follower bag Include：Serialize device：Achievement data sequence chemical conversion byte stream after time dimension is polymerize；Compressor reducer：Byte stream is subjected to streaming compression；Network follower：Byte stream after compression is sent to network performance monitoring platform；File follower：Byte stream after compression is write into filing data file；Index：The filing data document creation information of the file follower is indexed, creates and updates index file.
- A kind of 6. method of collection analysis network equipment information, it is characterised in that including：Collection step：By the message data unserializing received into key assignments plaid matching formula data；Analytical procedure：Key-value pair formatted data is converted into network performance unified metric data；Polymerization procedure：Network performance unified metric data are entered into row index dimension and time dimension polymerization；Export step：Network performance unified metric data output after polymerization to network performance monitoring device, or write-in disk are returned Shelves initial data.
- 7. the method for collection analysis network equipment information according to claim 6, it is characterised in that the collection step bag Include：Memory buffer step：The network port is monitored, reads the message data that the network equipment sends over, and write memory buffer Pond；Unserializing step：Read message data from memory buffer pond, the model agreement id field in message data from Inquiry obtains corresponding model agreement in model agreement storehouse, further according to model agreement by message data unserializing into key assignments plaid matching Formula data.
- 8. the method for collection analysis network equipment information according to claim 6, it is characterised in that the analytical procedure tool Body includes：IP flow object establishment steps：The key-value pair formatted data from the collector is read, IP streams are established according to five-tuple Object, it is stored in cache table；Escape algorithm queries step：According to the field in IP flow objects, inquired about to index escape Algorithm mapping table and obtain corresponding word The escape algorithm of section, escape algorithm are uniquely determined by protocol type and field name；IP flow object escape steps：The field in IP flow objects is traveled through, field is converted to by internetworking according to field escape algorithm Can unified metric data；IP flow objects export step：IP flow objects after index escape are sent to the polymerization procedure.
- 9. the method for collection analysis network equipment information according to claim 8, it is characterised in that the polymerization procedure tool Body includes：Index dimension statistic unit requestor construction step：Read the configuration of index dimension, the statistic unit inquiry of structure index dimension Device, index dimension statistic unit requestor position or create by reading the dimension index in IP flow objects corresponding to index Dimension statistic unit；IP flow object allocation steps：According to the dimension of configuration, IP flow objects are created and are assigned to pair by service index dimension requestor The index dimension statistic unit answered；Aggregating algorithm obtaining step：Network performance unified metric data in IP flow objects, to index dimension aggregating algorithm Inquiry obtains the aggregating algorithm of corresponding index in mapping table, and the aggregating algorithm of corresponding index is uniquely determined by the configuration of index dimension；Index dimension polymerization procedure：The network performance unified metric data in IP flow objects are traveled through, use the polymerization of corresponding index Algorithm enters the polymerization of row index dimension；Time dimension statistic unit requestor construction step：More time granularity configurations are read, build time dimension statistic unit Requestor；Time dimension polymerization procedure：According to more time granularities of configuration, usage time dimension statistic unit requestor is by index Dimension statistic unit is assigned to multiple time dimension aggregate statistics units, using with index dimension identical aggregating algorithm, to refer to Network performance unified metric data after mark dimension polymerization carry out time dimension polymerization again；Step is exported after polymerization：Network performance unified metric data after the polymerization of extraction time dimension, are sent to output step.
- 10. the method for collection analysis network equipment information according to claim 6, it is characterised in that the output step Including：Serialisation step：Network performance unified metric data sequence chemical conversion byte stream after time dimension is polymerize；Compression step：Byte stream is subjected to streaming compression；Network exports step：Byte stream after compression is sent to network performance monitoring platform；File exports step：Byte stream after compression is write into filing data file；Straw line step：Filing data document creation information is indexed, creates and updates index file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710666927.2A CN107634848B (en) | 2017-08-07 | 2017-08-07 | System and method for collecting and analyzing network equipment information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710666927.2A CN107634848B (en) | 2017-08-07 | 2017-08-07 | System and method for collecting and analyzing network equipment information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107634848A true CN107634848A (en) | 2018-01-26 |
| CN107634848B CN107634848B (en) | 2020-10-27 |
Family
ID=61099343
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710666927.2A Active CN107634848B (en) | 2017-08-07 | 2017-08-07 | System and method for collecting and analyzing network equipment information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107634848B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108446305A (en) * | 2018-01-30 | 2018-08-24 | 上海天旦网络科技发展有限公司 | The system and method for various dimensions service data statistics |
| CN108449375A (en) * | 2018-01-30 | 2018-08-24 | 上海天旦网络科技发展有限公司 | The system and method for network interconnection data grabber distribution |
| CN109271336A (en) * | 2018-10-10 | 2019-01-25 | 宋兴奎 | Intelligent information managing device, method and system |
| CN110022248A (en) * | 2019-04-19 | 2019-07-16 | 山东浪潮云信息技术有限公司 | Link flow statistical method and system, traffic statistics host and statistics request end |
| CN110069411A (en) * | 2019-04-15 | 2019-07-30 | 网易（杭州）网络有限公司 | Client performance quality report generation method, device, medium and electronic equipment |
| CN110191024A (en) * | 2019-05-31 | 2019-08-30 | 中国联合网络通信集团有限公司 | Network traffic monitoring method and device |
| CN110474896A (en) * | 2019-08-06 | 2019-11-19 | 厦门科灿信息技术有限公司 | Data communications method and relevant device based on Modbus consensus standard |
| CN111162949A (en) * | 2019-12-31 | 2020-05-15 | 国网山西省电力公司信息通信分公司 | An Interface Monitoring Method Based on Java Bytecode Embedding Technology |
| CN111506605A (en) * | 2020-04-02 | 2020-08-07 | 尚娱软件（深圳）有限公司 | Data analysis method, device, equipment and computer readable storage medium |
| CN112702232A (en) * | 2020-12-21 | 2021-04-23 | 盛科网络（苏州）有限公司 | IPFIX flow statistical method and device based on user-defined data |
| CN112783120A (en) * | 2020-12-31 | 2021-05-11 | 济南大陆机电股份有限公司 | Industrial metering data acquisition method and system based on driving |
| CN113242151A (en) * | 2021-06-04 | 2021-08-10 | 上海天旦网络科技发展有限公司 | Specific data extraction method and system based on massive network data |
| CN113364624A (en) * | 2021-06-04 | 2021-09-07 | 上海天旦网络科技发展有限公司 | Mixed cloud flow acquisition method and system based on edge computing |
| CN113824605A (en) * | 2020-06-18 | 2021-12-21 | 中兴通讯股份有限公司 | Network flow sampling method, network device and storage medium |
| CN115174496A (en) * | 2022-05-23 | 2022-10-11 | 北京大学 | Processing terminal and switch for intra-network aggregation transmission |
| CN115987812A (en) * | 2022-12-27 | 2023-04-18 | 上海天旦网络科技发展有限公司 | Mass session data aggregation analysis method, system, device and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030004949A1 (en) * | 2001-06-29 | 2003-01-02 | Jianrong Hong | Multi-thread shared memory message queue buffer systems, methods and computer program products for storing data in a database |
| CN102611626A (en) * | 2012-03-30 | 2012-07-25 | 北京英诺威尔科技股份有限公司 | System and method for analyzing network flow |
| CN103546343A (en) * | 2013-10-18 | 2014-01-29 | 中国南方电网有限责任公司 | Network flow display method and system for network flow analyzing systems |
| CN105262837A (en) * | 2015-11-03 | 2016-01-20 | 上海唐舜电信科技有限公司 | Tri-network integration application terminal accessing device based on cloud computing and realization method |
| CN106899443A (en) * | 2015-12-18 | 2017-06-27 | 北京神州泰岳软件股份有限公司 | The acquisition method and equipment of a kind of Netflow datas on flows |
-
2017
- 2017-08-07 CN CN201710666927.2A patent/CN107634848B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030004949A1 (en) * | 2001-06-29 | 2003-01-02 | Jianrong Hong | Multi-thread shared memory message queue buffer systems, methods and computer program products for storing data in a database |
| CN102611626A (en) * | 2012-03-30 | 2012-07-25 | 北京英诺威尔科技股份有限公司 | System and method for analyzing network flow |
| CN103546343A (en) * | 2013-10-18 | 2014-01-29 | 中国南方电网有限责任公司 | Network flow display method and system for network flow analyzing systems |
| CN105262837A (en) * | 2015-11-03 | 2016-01-20 | 上海唐舜电信科技有限公司 | Tri-network integration application terminal accessing device based on cloud computing and realization method |
| CN106899443A (en) * | 2015-12-18 | 2017-06-27 | 北京神州泰岳软件股份有限公司 | The acquisition method and equipment of a kind of Netflow datas on flows |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108449375A (en) * | 2018-01-30 | 2018-08-24 | 上海天旦网络科技发展有限公司 | The system and method for network interconnection data grabber distribution |
| CN108446305A (en) * | 2018-01-30 | 2018-08-24 | 上海天旦网络科技发展有限公司 | The system and method for various dimensions service data statistics |
| CN109271336A (en) * | 2018-10-10 | 2019-01-25 | 宋兴奎 | Intelligent information managing device, method and system |
| CN110069411A (en) * | 2019-04-15 | 2019-07-30 | 网易（杭州）网络有限公司 | Client performance quality report generation method, device, medium and electronic equipment |
| CN110022248A (en) * | 2019-04-19 | 2019-07-16 | 山东浪潮云信息技术有限公司 | Link flow statistical method and system, traffic statistics host and statistics request end |
| CN110191024A (en) * | 2019-05-31 | 2019-08-30 | 中国联合网络通信集团有限公司 | Network traffic monitoring method and device |
| CN110474896A (en) * | 2019-08-06 | 2019-11-19 | 厦门科灿信息技术有限公司 | Data communications method and relevant device based on Modbus consensus standard |
| CN110474896B (en) * | 2019-08-06 | 2022-01-04 | 厦门科灿信息技术有限公司 | Data communication method based on Modbus protocol standard and related equipment |
| CN111162949A (en) * | 2019-12-31 | 2020-05-15 | 国网山西省电力公司信息通信分公司 | An Interface Monitoring Method Based on Java Bytecode Embedding Technology |
| CN111506605A (en) * | 2020-04-02 | 2020-08-07 | 尚娱软件（深圳）有限公司 | Data analysis method, device, equipment and computer readable storage medium |
| CN113824605A (en) * | 2020-06-18 | 2021-12-21 | 中兴通讯股份有限公司 | Network flow sampling method, network device and storage medium |
| CN113824605B (en) * | 2020-06-18 | 2025-01-07 | 中兴通讯股份有限公司 | Network flow sampling method, network device and storage medium |
| CN112702232B (en) * | 2020-12-21 | 2022-04-01 | 苏州盛科通信股份有限公司 | IPFIX flow statistical method and device based on user-defined data |
| CN112702232A (en) * | 2020-12-21 | 2021-04-23 | 盛科网络（苏州）有限公司 | IPFIX flow statistical method and device based on user-defined data |
| CN112783120A (en) * | 2020-12-31 | 2021-05-11 | 济南大陆机电股份有限公司 | Industrial metering data acquisition method and system based on driving |
| CN113364624A (en) * | 2021-06-04 | 2021-09-07 | 上海天旦网络科技发展有限公司 | Mixed cloud flow acquisition method and system based on edge computing |
| CN113242151A (en) * | 2021-06-04 | 2021-08-10 | 上海天旦网络科技发展有限公司 | Specific data extraction method and system based on massive network data |
| CN115174496A (en) * | 2022-05-23 | 2022-10-11 | 北京大学 | Processing terminal and switch for intra-network aggregation transmission |
| CN115174496B (en) * | 2022-05-23 | 2024-02-13 | 北京大学 | A processing terminal and switch for intra-network aggregation transmission |
| CN115987812A (en) * | 2022-12-27 | 2023-04-18 | 上海天旦网络科技发展有限公司 | Mass session data aggregation analysis method, system, device and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107634848B (en) | 2020-10-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107634848A (en) | A kind of system and method for collection analysis network equipment information | |
| US11681678B2 (en) | Fast circular database | |
| US8179799B2 (en) | Method for partitioning network flows based on their time information | |
| EP2240854B1 (en) | Method of resolving network address to host names in network flows for network device | |
| US20090144414A1 (en) | Method for summarizing flow information from network devices | |
| CN110149239B (en) | Network flow monitoring method based on sFlow | |
| US20120026914A1 (en) | Analyzing Network Activity by Presenting Topology Information with Application Traffic Quantity | |
| US10069797B2 (en) | 10Gbps line rate stream to disk with fast retrieval (metadata) and network statistics | |
| CN112632129A (en) | Code stream data management method, device and storage medium | |
| CN114979186B (en) | Flow link analysis method and system based on Flink component | |
| CN107147535A (en) | A Distributed Statistical Analysis Method of Network Measurement Data | |
| CN112486914A (en) | Data packet storage and fast check method and system | |
| Canini et al. | Per flow packet sampling for high-speed network monitoring | |
| Plagemann et al. | Using data stream management systems for traffic analysis–a case study– | |
| WO2017124660A1 (en) | System and method for associating multi-stage assembly transactions | |
| CN113242151A (en) | Specific data extraction method and system based on massive network data | |
| CN106648722A (en) | Flume receiving side data processing method and device based on big data | |
| CN206164759U (en) | A cross-network multi-source heterogeneous data acquisition device | |
| CN112181929A (en) | Cloud management platform log processing method and device, electronic device and storage medium | |
| Elsen et al. | goProbe: a scalable distributed network monitoring solution | |
| CN107682180B (en) | Method for collecting performance index data of communication network equipment | |
| CN115695216A (en) | Big data analysis method for internet traffic flow direction | |
| US20180285396A1 (en) | Performance data storage | |
| CN116170352A (en) | Network traffic processing method and device, electronic equipment and storage medium | |
| CN116016288A (en) | Flow monitoring method, device, equipment and storage medium of industrial equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |