WO2022247308A1 - 流量测量方法、装置及相关设备 - Google Patents
流量测量方法、装置及相关设备 Download PDFInfo
- Publication number
- WO2022247308A1 WO2022247308A1 PCT/CN2022/071728 CN2022071728W WO2022247308A1 WO 2022247308 A1 WO2022247308 A1 WO 2022247308A1 CN 2022071728 W CN2022071728 W CN 2022071728W WO 2022247308 A1 WO2022247308 A1 WO 2022247308A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- flow
- message
- forwarding network
- storage table
- server
- Prior art date
Links
- 238000000691 measurement method Methods 0.000 title claims abstract description 38
- 238000000034 method Methods 0.000 claims abstract description 70
- 238000005206 flow analysis Methods 0.000 claims abstract description 11
- 238000004458 analytical method Methods 0.000 claims description 71
- 238000005259 measurement Methods 0.000 claims description 53
- 230000005540 biological transmission Effects 0.000 claims description 12
- 238000005538 encapsulation Methods 0.000 claims description 8
- 210000000352 storage cell Anatomy 0.000 description 70
- 230000006870 function Effects 0.000 description 20
- 210000004027 cell Anatomy 0.000 description 16
- 238000010586 diagram Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 6
- 238000007726 management method Methods 0.000 description 3
- 238000004806 packaging method and process Methods 0.000 description 3
- 241000406668 Loxodonta cyclotis Species 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000344 soap Substances 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
Definitions
- the present application relates to the communication field, and in particular to a flow measurement method, device and related equipment.
- Traffic measurement is an important part of network management, providing indispensable information for service quality improvement, capacity planning, network billing, congestion control, anomaly detection in data centers and backbone networks. For example, when the network is congested, information about the data flow causing the congestion can be found as soon as possible through flow measurement.
- the currently more popular traffic measurement method is the in-band network telemetry (in-band network telemetry, INT) method, but the inventors of the present application have found that this method has the problem of incomplete link information measured, and will consume a large amount of bandwidth resources And computing resources, the problem of high measurement cost.
- INT in-band network telemetry
- the present application provides a traffic measurement method, device and related equipment, which can solve the problem of incomplete measured link information and the problem of consuming a large amount of bandwidth resources and computing resources existing in the prior art.
- a traffic measurement method which is applied to the transmission process in which the data generated by the client is transmitted to the server through multiple forwarding network elements, the server is connected to a traffic analysis device, and the multiple forwarding network elements
- Each forwarding network element is a network device that supports adding information for processing the message to the message in the data flow, and the method includes:
- the server receives the message sent by the forwarding network element connected to it, and the message includes the identifier of the data flow and the information of processing the message by the multiple forwarding network elements;
- the server obtains a flow storage table, and the flow storage table is used to store the number of packets in the data stream;
- the server counts the number of packets in the data flow into the flow storage table according to the identifier of the data flow and the information on processing the packets by the plurality of forwarding network elements, and obtains the updated flow storage table;
- the server sends the updated traffic storage table to the traffic analysis device.
- the message received by the server includes the information of multiple forwarding network elements processing the message, including the forwarding network element connected to the server (that is, the last forwarding network element that the message passes before reaching the server, Hereinafter referred to as the information of the last forwarding network element) processing packets, that is to say, when the server counts the number of packets in the data flow to the flow storage table, it counts the information of the last forwarding network element processing packets Unlike in the prior art, when the last forwarding network element connected to the server sends information to the traffic analysis device, it does not send the information of its own processing of the packet to the traffic analysis device, resulting in inaccurate link information for traffic measurement. whole.
- the server After the server receives the flow identifier and the information on processing packets by multiple forwarding network elements, it counts the number of packets in the data flow according to the flow identifier and information on processing packets by multiple forwarding network elements.
- the flow storage table obtains the updated flow storage table, and then sends the updated flow storage table to the traffic analysis device, unlike in the prior art, the last forwarding network element directly processes the flow identification and multiple forwarding network elements
- the message information is sent to the traffic analysis device, and the traffic analysis device performs traffic statistics according to the flow identifier sent by the last forwarding network element and the information of multiple forwarding network elements processing the message.
- the above solution can reduce the bandwidth resources required to send data to the traffic analysis device, and does not need the traffic analysis device to perform traffic statistics, and can reduce the consumption of computing resources of the traffic analysis device, thereby reducing Measuring the role of cost.
- the method before the server sends the updated traffic storage table to the traffic analysis device, the method further includes: the server performs an operation on the updated traffic storage table encapsulation.
- the server encapsulates the updated traffic storage table, and then sends the encapsulated traffic storage table to the traffic analysis device, which can further reduce the bandwidth resource consumed for sending data to the traffic analysis device, and further reduce the measurement cost.
- the identifier of the data flow includes one or more combinations of the following: an Internet protocol (internet protocol, IP) address of the client, a port through which the client sends the data flow number, the IP address of the server, the port number for the server to receive the data stream, the transport layer protocol used by the client to transmit the data stream to the server, virtual local area network (virtual local area network) , VLAN) identifier.
- IP Internet protocol
- VLAN virtual local area network
- the information on processing the packet by each forwarding network element includes one or more combinations of the following: an identifier of each forwarding network element, an The port number for receiving the message, the port number for each forwarding network element sending the message, the queue number for the message to enter, the queue number for the message to leave, the number of the queue for each forwarding network element The time when the message is received, and the time when each forwarding network element sends the message.
- a traffic measurement method is provided, which is applied to the transmission process in which the data flow generated by the client is transmitted to the server through multiple forwarding network elements, and the statistical device is connected to the multiple forwarding network elements to send the data flow
- the statistics device is connected to a traffic analysis device, and each forwarding network element in the multiple forwarding network elements supports adding its processing information to the packets in the data flow.
- a network device that describes the information of the message includes:
- the statistical device receives a message sent by the last forwarding network element, the message includes the identifier of the data flow, and the message carries information on processing the message by the multiple forwarding network elements;
- the statistical device obtains a flow storage table, and the flow storage table is used to store the number of packets in the data flow;
- the statistics device counts the number of packets in the data flow to the flow storage table according to the identifier of the data flow and the information on processing the packets by the plurality of forwarding network elements, and obtains the updated flow storage table;
- the statistical device sends the updated traffic storage table to the traffic analysis device.
- the method before the statistical device sends the updated traffic storage table to the traffic analysis device, the method further includes: the statistical device performs an operation on the updated traffic storage table encapsulation.
- the identifier of the data flow includes one or more of the following combinations: the IP address of the client, the port number through which the client sends the data flow, the IP address of the server address, the port number of the server receiving the data flow, the transport layer protocol used by the client to transmit the data flow to the server, and the VLAN identifier.
- the information on processing the packet by each forwarding network element includes one or more combinations of the following: an identifier of each forwarding network element, an The port number for receiving the message, the port number for each forwarding network element sending the message, the queue number for the message to enter, the queue number for the message to leave, the number of the queue for each forwarding network element The time when the message is received, and the time when each forwarding network element sends the message.
- a traffic measurement device which is applied to the transmission process in which the data generated by the client is transmitted to the server through multiple forwarding network elements, and is specifically applied to the server, and the server is also connected to a traffic analysis device
- Each forwarding network element in the plurality of forwarding network elements is a network device that supports adding information for processing the message to the message in the data flow, and the device includes:
- the receiving module is configured to receive the message sent by the forwarding network element connected to the server, the message includes the identifier of the data flow, and the message carries the information processed by the multiple forwarding network elements information about the message;
- An acquisition module configured to acquire a flow storage table, where the flow storage table is used to store the number of packets in the data stream;
- a statistics module configured to count the number of packets in the data flow to the flow storage table according to the identifier of the data flow and the information on processing the packets by the multiple forwarding network elements, and obtain an updated Flow storage table;
- a sending module configured to send the updated flow storage table to the flow analysis device.
- the device further includes: an encapsulation module, configured to encapsulate the updated flow storage table.
- the identifier of the data flow includes one or more of the following combinations: the IP address of the client, the port number through which the client sends the data flow, the IP address of the server address, the port number of the server receiving the data flow, the transport layer protocol used by the client to transmit the data flow to the server, and the VLAN ID.
- the information on processing the packet by each forwarding network element includes one or more combinations of the following: an identifier of each forwarding network element, an The port number for receiving the message, the port number for each forwarding network element sending the message, the queue number for the message to enter, the queue number for the message to leave, the number of the queue for each forwarding network element The time when the message is received, and the time when each forwarding network element sends the message.
- a traffic measurement device which is applied to the transmission process of the data generated by the client through multiple forwarding network elements and transmitted to the server, and is specifically applied to a statistical device, and the statistical device is connected to the multiple forwarding networks
- the last forwarding network element that sends the data flow to the server in the unit, the statistical device is also connected to a traffic analysis device, and each forwarding network element in the multiple forwarding network elements is to support the data flow in the A network device that adds information on processing the message to the message in the message, the device includes:
- a receiving module configured to receive a message sent by the last forwarding network element, the message includes the identifier of the data flow, and the message carries information on processing the message by the multiple forwarding network elements ;
- An acquisition module configured to acquire a flow storage table, where the flow storage table is used to store the number of packets in the data stream;
- a statistics module configured to count the number of packets in the data flow to the flow storage table according to the identifier of the data flow and the information on processing the packets by the multiple forwarding network elements, and obtain an updated Flow storage table;
- a sending module configured to send the updated flow storage table to the flow analysis device.
- a non-transitory computer-readable storage medium stores instructions, and the instructions are used to implement any possible implementation manner or the second aspect of the first aspect above.
- a computer program product including a computer program.
- the computer program When the computer program is read and executed by a cluster of computer equipment, the cluster of computer equipment executes any possible implementation manner or the first implementation mode of the first aspect above.
- the method provided by any possible implementation in the second aspect.
- a computing device cluster including at least one computing device, each computing device includes a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that The computing device executes the method provided in any possible implementation manner of the first aspect or any possible implementation manner of the second aspect.
- the computing device cluster includes a computing device, and the computing device includes a processor and a memory; the processor is configured to execute instructions stored in the memory, so that the computing device performs the above-mentioned first aspect. Any possible implementation or the method provided by any possible implementation of the second aspect.
- the computing device cluster includes at least two computing devices, and each computing device includes a processor and a memory; the processors of the at least two computing devices are used to execute the Stored instructions, so that the cluster of computing devices executes the method provided in any possible implementation manner of the first aspect or any possible implementation manner of the second aspect.
- Fig. 1 is a schematic diagram of the first flow storage table involved in the present application
- Fig. 2 is the structural representation of the flow measurement system involved in the present application.
- Fig. 3 is a schematic flow chart of a flow measurement method provided by the present application.
- Fig. 4 is a schematic diagram of changes in the first flow storage table provided by the present application.
- Fig. 5 is a schematic diagram of a second flow storage table provided by the present application.
- Fig. 6 is a schematic diagram of an exemplary first flow storage table and an exemplary second flow storage table provided by the present application;
- Fig. 7 is a schematic flow chart of another flow measurement method provided by the present application.
- Fig. 8 is a schematic structural diagram of a flow measurement system provided by the present application.
- FIG. 9 is a schematic diagram of a traffic query process provided by the present application.
- FIG. 10 is a schematic diagram of a traffic query process provided by the present application.
- Fig. 11 is a schematic structural diagram of a flow measuring device provided by the present application.
- FIG. 12 is a schematic structural diagram of a computing device cluster provided by the present application.
- Fig. 13 is a schematic structural diagram of a computing device provided by the present application.
- first and second in the embodiments of the present application are used for description purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, a feature defined as “first” and “second” may explicitly or implicitly include one or more of these features.
- “at least one” means one or more, and “multiple” means two or more.
- “And/or” describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A, B can be singular or plural.
- the character “/” generally indicates that the contextual objects are an “or” relationship.
- “At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.
- At least one item (unit) of a, b or c can represent: a, b, c, a-b, a-c, b-c or a-b-c, wherein a, b, c can be single or multiple.
- the flow rate of the data stream which may also be referred to as the length of the data stream, or simply referred to as the flow rate, refers to the number of messages (also referred to as data packets) included in a data stream.
- the quintuple is a set composed of five parameters: source IP address, source port number, destination IP address, destination port number and transport layer protocol.
- the source IP address refers to the IP address of the client that sends out the data flow
- the source port number refers to the port number that the client sends out the data flow
- the destination IP address refers to the IP address of the server that receives the data flow
- the destination port number refers to the port number that the server receives.
- the port number of the data stream refers to the protocol used by the client and the server when transmitting the data stream.
- the five-tuple can distinguish different data streams, and the corresponding data stream is unique.
- INT is a traffic measurement protocol, which collects information (such as the identity document of INT network element) on the data plane for each INT network element (referring to network devices such as switches or routers supporting INT technology) to process packets. , ID), the port number of the packet entering and exiting the INT network element, the queue number of the packet entering and exiting the INT network element, the time of the packet entering and exiting the INT network element, etc.), and then the last INT network element collects the INT network elements process message information and send it to traffic analysis equipment, so as to realize fine-grained acquisition of network status. This process has no control and participation from the control plane, which can greatly reduce the pressure on the controller.
- the probabilistic data structure is a data structure based on hash (also called hash).
- the probabilistic data structure records the information of each data flow in the network at the cost of sacrificing certain traffic measurement accuracy instead of The information of each message can reduce memory overhead.
- the probabilistic data structure saves key-value data with the same hash value (also called hash value) in the first flow storage table (see Figure 1) by setting a hash function (also called a hash function). in the same cell.
- the statistical value in the storage cell is used as the traffic measurement result, which is an estimate of the real traffic of each data flow. Ignoring the occurrence of hash collisions (collision), the error can be well controlled under a certain threshold.
- probabilistic data structures possess a theoretically provable balance of flow estimation accuracy and memory.
- Currently commonly used probabilistic data structures include: Bloom filter, cardinality estimation method, and sketch.
- sketch has a minimum sketch (count-min sketch, abbreviated as cm sketch), a conservative update sketch (conservative- update sketch (referred to as cu sketch), count sketch (count sketch), FlowRadar, UnivMon, etc.
- cm sketch has a minimum sketch (count-min sketch, abbreviated as cm sketch), a conservative update sketch (conservative- update sketch (referred to as cu sketch), count sketch (count sketch), FlowRadar, UnivMon, etc.
- cm sketch the most popular probabilistic data structure in the field of flow measurement is cm sketch.
- the first flow storage table includes d storage layers (rows), and each storage layer includes w storage cells (also called hash buckets).
- the number of bits of the storage cells in the first flow storage table is ⁇ , therefore, the flow range that each storage cell can record is the same.
- Each storage layer is associated with a hash function, and the hash functions h 11 , h 12 , . . . , h 1d associated with the d storage layers are independent in pairs.
- both d and w are natural numbers greater than 0.
- ⁇ is 8 bits (bit), that is to say, the flow range that the memory cell can record is 0-255. In the initial state, the initial values of all storage cells on the first flow storage table are 0 or empty.
- cm sketch The working principle of cm sketch is: when a message including a flow identifier arrives at a device deployed with cm sketch, the device uses h 11 , h 12 , ..., h 1d to hash the flow identifier included in the message, In this way, d storage cells are located in the first flow storage table, and then the device increases the statistical value of the storage cell whose statistical value has not overflowed among the located d storage cells by 1, and the statistical value of the storage cell whose statistical value has overflowed remains unchanged, so as to obtain the updated first flow storage table.
- the minimum statistical value in the d storage cells is the device's estimated value of the real traffic of the data flow.
- the statistical value has not overflowed, indicating that the statistical value is less than the maximum value that can be recorded in the cell where the statistical value is located, and the statistical value has overflowed, indicating that the statistical value is equal to the maximum value that can be recorded in the cell where the statistical value is located. For example, suppose cell A The statistical value A' of cell B is 100, the statistical value B' of cell B is 1000, and the maximum value that can be recorded in cells A and B is both 255. B' has overflowed.
- Fig. 2 is a kind of traffic measurement system involved in the present application, as shown in Fig. 2, the system includes: client 110, multiple forwarding network elements 120, server 130 and traffic analysis device 140, client 110 , the multiple forwarding network elements 120, the server 130 and the traffic analysis device 140 can communicate through a network, and the network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.
- the client 110 refers to the device that sends out the data flow.
- the data flow sent by the client 110 may be forwarded by some or all of the multiple forwarding network elements 120 , and finally reaches the server 130 .
- the client 110 can be a terminal device, such as a smart phone, a tablet computer, a mobile notebook, a wearable device, etc., or a server, such as a personal computer, a cloud server, etc.
- the operating system of the client 110 may be IOS, Android, Windows, Linux, etc., which is not specifically limited here.
- the forwarding network element 120 is a network device, such as a switch or a router, for implementing data flow forwarding tasks.
- the server 130 refers to a device receiving a data stream, which may be a terminal device, such as a smart phone, a tablet computer, a mobile notebook, a wearable device, etc., or may be a personal computer, a server, etc.
- the operating system of the server 130 may be IOS, Android, Windows, Linux, etc., which is not specifically limited here.
- the traffic analysis device 140 is a device for implementing a traffic analysis function.
- the traffic analysis device 140 can monitor the transmission process of the data stream and obtain a traffic analysis result. After obtaining the traffic analysis result, the traffic analysis device 140 may receive the query request input by the user, and feed back the traffic query result to the user.
- the traffic analysis device 140 may be a personal computer, a server, and the like.
- the flow measurement system shown in Figure 2 usually uses the INT method to realize the flow measurement task.
- the multiple forwarding network elements 120 in the system shown in Figure 2 are INT network elements, and the process of realizing traffic measurement in the system shown in Figure 2 is:
- the first forwarding network element 120 When a message is sent by the client 110 and arrives at the first forwarding network element 120, the first forwarding network element 120 will add a data flow identifier (hereinafter referred to as the flow identifier) to the message and the first forwarding network element 120 Information about processing packets.
- the flow identification can be any one or more of the five parameters included in the quintuple, and can also be a VLAN identification (identity document, ID), etc.
- the information for forwarding network element 120 to process the message can be forwarding network element 120
- the flow identifiers added by the first forwarding network element 120 to the packets belonging to the same data flow are the same.
- the first forwarding network element 120 After adding the flow identifier and the information of the first forwarding network element 120 to process the message, the first forwarding network element 120 sends the message to the transit network element 120 (meaning that the message arrives at the last forwarding network element 120 after the first forwarding network element 120 The forwarding network element 120 passed by the network element 120 before sends a message carrying information about the processing of the message by the first forwarding network element 120 .
- the transit network element 120 continues to add the information of the message processed by the network element on the message, and then forwards the message to the next forwarding network with the added information of the message processed by the network element. Yuan 120 forwarding.
- the last forwarding network element 120 processes the message through the forwarding network element 120 (that is, the first forwarding network element 120 and the transit network element 120) passed by the message carried by the message carried by the message.
- the information (hereinafter referred to as the first carrying information) is stripped from the message to obtain the independent first carrying information and the message, and then the first carrying information is sent to the traffic analysis device 140, and the message is forwarded to the server 130 .
- the last forwarding network element 120 usually sends the multiple pieces of first carried information to the traffic analysis device 140 after stripping off multiple packets to obtain multiple pieces of first carried information.
- the traffic analysis device 140 After receiving the plurality of first carried information, the traffic analysis device 140 performs analysis and statistics according to the plurality of first carried information, and obtains traffic statistics results, such as the traffic of each data flow, each traffic flow passing through each forwarding network element 120 data flow, etc.
- the last forwarding network element 120 does not continue to add information about its own processing of the message to the message before sending the first carried information to the flow analysis device 140, that is, the first The carried information does not include the information of the last forwarding network element 120 processing the packet, and the first carried information sent by the last forwarding network element 120 received by the traffic analysis device 140 does not include the information of the last forwarding network element 120 processing the packet, so , this method has the problem that the measured link information is incomplete.
- the last forwarding network element 120 sends the stripped multiple first carried information together to the traffic analysis device 140. If the data volume of the multiple first carried information is very large, this will not only consume a large amount of bandwidth resources, but also A large amount of computing resources of the traffic analysis device 140 will be consumed, and the measurement cost is relatively high.
- the present application provides a flow measurement method, which can be applied to the flow measurement system shown in Figure 2, and the multiple forwarding network elements 120 in the system shown in Figure 2 are all supported in the data flow
- the network element that adds the information of processing the message to the message such as the INT network element, has a probabilistic data structure deployed on the server 130, and the process of realizing flow measurement by the system shown in Figure 2 is as follows:
- the first forwarding network element 120 When a message is sent by the client 110 and arrives at the first forwarding network element 120, the first forwarding network element 120 will add a flow identifier and information about processing the message by the first forwarding network element 120 to the message.
- the first forwarding network element 120 After adding the flow identifier and the information of the first forwarding network element 120 processing the message, the first forwarding network element 120 sends a message carrying the information of the first forwarding network element 120 processing the message to the transit network element 120 .
- the transit network element 120 continues to add the information of the message processed by the network element on the message, and then forwards the message to the next forwarding network with the added information of the message processed by the network element. Yuan 120 forwarding.
- the last forwarding network element 120 When the message is forwarded to the last forwarding network element 120, the last forwarding network element 120 continues to add the information of the message processed by the network element on the message, and then sends the message added with the information of the message processed by the network element to the message.
- the server 130 forwards.
- the server 130 processes the forwarding network element 120 (that is, the first forwarding network element 120, the transit network element 120, and the last forwarding network element 120) that the message carried by the message passes through.
- the information of the text (hereinafter referred to as the second carried information) is stripped from the message to obtain the independent second carried information and the message.
- the server 130 After the server 130 strips off the second carried information, since the probabilistic data structure is deployed on the server 130, the server 130 can count the flow of the data flow to the first flow storage table according to the second carried information, and obtain the updated The first flow storage table. Then, the server 130 sends the obtained updated first traffic storage table to the traffic analysis device 140 .
- the server 130 after the server 130 strips off multiple packets to obtain a plurality of second carrying information, it can count the traffic of the data flow in the first flow storage table according to the multiple second carrying information, and obtain the updated The first flow storage table, the updated flow storage table may include the flow of each data flow, the flow of each data flow passing through each forwarding network element 120, and the like.
- the traffic measurement method provided in this application implements traffic measurement
- the stripping operation of the second carried information is not performed by the last forwarding network element 120, but the stripping operation of the second carried information is performed by the server 130, Moreover, the second carried information stripped by the server 130 includes the information of the last forwarding network element 120 to process the message. Therefore, this method can solve the problem of incomplete measured link information existing in the existing INT method.
- the server 130 counts the flow of the data flow into the first flow storage table according to the second carrying information, and after obtaining the updated first flow storage table, the updated first flow storage table is A traffic storage table is sent to the traffic analysis device 140, unlike the existing INT method, the last forwarding network element 120 directly sends the first carried information to the traffic analysis device 140, and the traffic analysis device 140 performs the first carry information Analyze and summarize to obtain the traffic of the data stream. From the above introduction to the probabilistic data structure and the first flow storage table, it can be seen that using the first flow storage table to count the flow of the data flow can reduce the memory overhead.
- the updated data in the first flow storage table obtained by the server 130 The amount of data is smaller than the second data carrying information, and the server 130 sends the updated first flow storage table to the flow analysis device 140.
- it can reduce the consumption of sending data to the flow analysis device 140. bandwidth resources, and the updated first flow storage table already includes the flow of the data flow, and the flow analysis device 140 is not required to perform analysis, which can reduce the consumption of computing resources on the side of the flow analysis device 140 and reduce measurement costs.
- the probability data structure deployed on the server 130 is cm sketch, and the sequence of messages sent by the last forwarding network element 120 received by the server 130 is P1, P2 , ..., Pn as examples, in conjunction with the schematic flow chart shown in Figure 3, the flow measurement method provided by the application is described in detail, as shown in Figure 3, the method includes:
- the server 130 receives a message Pi including the second carrying information Pi' sent by the last forwarding network element 120.
- i and n are both natural numbers, 1 ⁇ i ⁇ n.
- the server 130 After receiving the packet Pi including the second carrying information Pi', the server 130 will strip the packet Pi to obtain the second carrying information Pi', and the second carrying information Pi' includes the flow identification ID Pi .
- the server 130 uses hash functions h 11 , h 12 , . . . , h 1d to hash the ID Pi included in the second carried information Pi' to obtain d first hash values.
- the d first hash values are h 11 (ID Pi ), h 12 (ID Pi ), ..., h 1d (ID Pi ).
- the server 130 determines the first target storage cell from the first flow storage table according to the d first hash values.
- the server 130 increases the statistical value of the first target storage cell by 1.
- the target storage cell is the above four storage cells, and then the statistical values of the above four storage cells are all increased by 1, and the updated first flow storage table is shown in Table C in FIG. 4 .
- the server 130 executes steps S101 to S104 for each received message to obtain an updated first flow storage table.
- the server 130 sends the updated first traffic storage table to the traffic analysis device 140.
- the server 130 before the server 130 sends the updated first traffic storage table to the traffic analysis device 140, it can also encapsulate the updated first traffic storage table, so as to reduce the traffic from the server 130 The bandwidth occupied when the analysis device 140 sends the updated first flow storage table.
- the server 130 may use an algorithm with an encapsulation function (such as elastic sketch) to encapsulate the updated first flow storage table.
- the traffic analysis device 140 After the traffic analysis device 140 receives the updated first traffic storage table sent by the server 130, it can realize the query of the traffic of a certain data flow, the query of the significant flow, and the query of the data flow received by the server 130 within a certain time window. number of messages or query the total number of messages received by the server 130 within a certain time window. Taking the traffic analysis device 140 to query the traffic of a certain data stream as an example, refer to FIG. 9 and the related description of FIG. 9 below for the process.
- the flow measurement method provided in the present application can also obtain the flow of each data flow passing through each forwarding network element 120, and the implementation method is as follows:
- the second carried information received by the server 130 may also include information on packet processing by multiple forwarding network elements 120, and the information on packet processing by multiple forwarding network elements 120 may be The IDs of multiple forwarding network elements 120, therefore, the probabilistic data structure deployed on the server 130 can be configured to include multiple first flow storage tables as shown in Figure 1, and configure the multiple first flow storage tables to associate a Hash function f 1 .
- the server end 130 can use the hash function f1 to hash the ID of each forwarding network element 120 included in the second carrying information, thereby Locate the first traffic storage table corresponding to each forwarding network element 120 in a traffic storage table, and then perform steps S102 to S106, and the traffic analysis device 140 can obtain the updated first traffic storage table corresponding to each forwarding network element 120 A flow storage table, so as to realize the acquisition of the flow of each data flow passing through each forwarding network element 120 .
- T 1 f 1 (the ID of the forwarding network element 120)%m 1
- m 1 represents the number of multiple first flow storage tables.
- the traffic analysis device 140 can query a certain data flow passing through a certain forwarding network element 120 tasks such as querying the number of data streams received by a certain forwarding network element 120 within a certain time window, or querying the total number of packets received by a certain forwarding network element 120 within a certain time window.
- the traffic analysis device 140 can query the traffic of a certain data flow passing through a certain forwarding network element 120 as an example, refer to FIG. 10 and the related description of FIG. 10 below for the process.
- the number of bits ⁇ of the storage cells included in the first flow storage table is the same, usually 8 bits. Therefore, it can be understood that in the flow measurement method provided by this application, if The probabilistic data structure deployed on the server 130 is the existing cm sketch or cu sketch, etc., and the flow measurement method provided by this application is used to measure the flow of large data flows or the flow of small data flows. The number of bits in the cell is the same.
- the number of bits in the storage cells included in the first flow storage table is 8 bits
- the number of packets included in the small data flow A is 2
- the number of packets included in the large data flow B is 1000
- the forwarding network element 120 is in
- the statistical value is 2
- 2 is far less than the maximum value 255 that can be recorded in the storage cell.
- the memory is wasted, and the forwarding network element 120 is measuring the flow of large data flow B , the statistical value has overflowed after reaching 255. Therefore, the estimated value of the flow of the large data flow B by the forwarding network element 120 is 255, which is quite different from the real flow of 1000 of the large data flow B.
- the flow measurement of the large data flow B Inaccurate.
- the present application also provides another flow measurement method, in which the probability data structure deployed on the server 130 is not the existing probability data structure, but a new probability data structure (That is, the tower sketch described below), this probabilistic data structure can improve memory utilization and improve the accuracy of flow measurement of large data streams.
- the tower sketch saves the key-value data with the same hash value in the same storage cell in the second traffic storage table by setting the hash function.
- the statistical value in the storage cell is used as the traffic measurement result, which is an estimate of the real traffic of each data flow.
- the second flow storage table is shown in FIG. 5 . It can be seen that the second flow storage table is the same as the first flow storage table shown in FIG. 1 in that both include d storage layers.
- the difference between the second flow storage table and the first flow storage table is that each storage layer of the second flow storage table includes a different number of storage cells, and the number of bits of the storage cells included in different storage layers is different.
- the number of storage cells included in one storage layer is w 1
- the number of bits is ⁇ 1
- the number of storage cells included in the second storage layer is w 2
- the number of bits is ⁇ 2
- the dth storage layer includes The number of storage cells is w d
- the number of digits is ⁇ d .
- storage cells belonging to different storage layers can record different flow ranges.
- Each storage layer is associated with a hash function, and the hash functions h 21 , h 22 , . . . , h 2d associated with the d storage layers are independent in pairs.
- the count values of all storage cells in the second flow storage table shown in FIG. 5 are 0 or empty in an initial state.
- the working principle of tower sketch is similar to the working principle of cm sketch mentioned above.
- the flow storage table 1 shown in FIG. 6 is an exemplary first flow storage table
- the flow storage table 2 shown in FIG. 6 is an exemplary second flow storage table. It can be seen from Figure 6:
- the flow storage table 1 includes 5 storage layers, the storage layer 11 to the storage layer 15 all include 8 storage cells, and the number of digits of the 40 storage cells is 8 bits, that is to say, the flow range that can be recorded by the 40 storage cells is equal to 0-255.
- the flow storage table 2 also includes 5 storage layers, the storage layer 21 includes 32 storage cells, and the number of digits is 2 bits, that is, the storage cells included in the storage layer 21 can record the flow range from 0 to 3, and the storage layer 22 includes 16 storage cells, the number of bits is 4 bits, that is, the storage cells included in the storage layer 22 can record the flow range of 0 to 15, and the storage layer 23 includes 8 storage cells, the number of bits is 8 bits, that is, the storage layer 23 includes The flow range that can be recorded in the storage cells is 0 ⁇ 255.
- the storage layer 24 includes 4 storage cells, and the number of digits is 16 bits. That is, the flow range that can be recorded in the storage cells included in the storage layer 24 is 0 ⁇ 65535. It includes 2 storage cells, both of which are 32 bits, that is, the storage cells included in the storage layer 25 can record traffic in the range of 0 to 4294967295.
- the device uses flow storage table 2 to store small data
- the flow rate of the data stream for example, when the flow rate of the small data stream is 12, use the storage cell with a memory occupation of 4 bits to record the flow rate of the data stream.
- the device since the maximum value that can be recorded in the storage cell in the flow storage table 2 is 4294967295, when the device uses the flow storage table 2 to measure the flow of a large data flow, it can use the storage cell that matches the flow of the large data flow to record For the flow of large data streams, for example, when the flow rate of large data streams is 1000, use a storage cell with a memory occupation of 16 bits to record the flow of the data stream. Recording the flow of the data flow in a grid will not cause a large difference between the measured flow of the large data flow and the real flow, so the accuracy of the measured flow of the large data flow can be improved.
- the flow storage table 2 includes more cells than the flow storage table 1 . It can be understood that when the number of storage cells included in the flow storage table 2 is greater than the number of storage cells included in the flow storage table 1, the probability of a hash collision occurring when the device uses the flow storage table 2 for flow measurement will also increase. is lower than the probability of hash collision when using flow storage table 1 for flow measurement, therefore, the accuracy of flow measurement using flow storage table 2 is higher than that of using flow storage table 1 for flow measurement.
- the server 130 receives a message Pi including the second carrying information Pi' sent by the last forwarding network element 120.
- the server 130 uses the hash functions h 21 , h 22 , . . . , h 2d to hash the ID Pi included in the second carried information Pi' to obtain d second hash values.
- the d second hash values are h 21 (ID Pi ), h 22 (ID Pi ), ..., h 2d (ID Pi ).
- the server 130 determines a second target storage cell from the second traffic storage table according to the d second hash values.
- the server 130 increases the statistical value of the second target storage cell by 1.
- the server 130 executes steps S201 to S204 for each received message to obtain an updated second flow storage table.
- the server 130 sends the updated second traffic storage table to the traffic analysis device 140.
- Another flow measurement method provided in this application can also obtain the flow of each data flow passing through each forwarding network element 120, and the implementation method is as follows:
- the second carried information received by the server 130 may also include information on packet processing by multiple forwarding network elements 120, and the information on packet processing by multiple forwarding network elements 120 may be IDs of multiple forwarding network elements 120, therefore, the probabilistic data structure deployed on the server 130 can be configured to include multiple second flow storage tables as shown in FIG. Hash function f 2 .
- the server 130 can use the hash function f2 to hash the ID of each forwarding network element 120 included in the carrying information, so that the information in the multiple second traffic storage tables Locate the second traffic storage table corresponding to each forwarding network element 120, and then perform steps S202 to S206, and the traffic analysis device 140 can obtain the updated second traffic storage table corresponding to each forwarding network element 120 , so as to realize the acquisition of the traffic of each data flow passing through each forwarding network element 120 .
- T 2 f 2 (ID of the forwarding network element 120)%m 2
- m 2 represents the number of multiple second flow storage tables.
- the traffic analysis device 140 can query a certain data flow passing through a certain forwarding network element 120 tasks such as querying the number of data streams received by a certain forwarding network element 120 within a certain time window, or querying the total number of packets received by a certain forwarding network element 120 within a certain time window.
- the traffic measurement method provided by the present application can also realize obtaining the traffic of each data flow received by each port on each forwarding network element 120, obtaining the traffic of each data flow sent by each client 110, Obtain the flow of each data flow passing through each queue on each forwarding network element 120, and its specific implementation method is the same as the above-mentioned method of obtaining the flow of each data flow passing through each forwarding network element 120 Similarly, reference may be made to the relevant description above, and details will not be repeated here.
- the flow measurement system shown in Figure 2 can also be Add a statistical device 150 with strong computing power and storage capacity, as the execution subject of the traffic measurement method provided by this application, as shown in Figure 8, the statistical device 150 is connected with the last forwarding network element 120, traffic analysis device 140 and The server 130 is connected.
- the statistical device 150 may be a personal computer, a server, or the like.
- the process of implementing the traffic measurement method provided in this application is similar to that when the server 130 is the executor.
- the relevant description above please refer to the relevant description above, which will not be repeated here.
- FIG. 9 is a schematic flow diagram of a traffic analysis device 140 provided in the present application to query the traffic of a certain data stream. As shown in FIG. 9 , the process includes the following steps:
- the traffic analysis device 140 receives a query request input by a user that includes an ID f of a data flow f.
- the traffic analysis device 140 uses hash functions h 11 , h 12 , . . . , h 1d to hash ID f , so as to determine d storage cells corresponding to ID f in the updated first traffic storage table.
- the traffic analysis device 140 determines the minimum statistical value in the d storage cells corresponding to the ID f as the traffic query result.
- the traffic query result is an estimation of the traffic of the real data flow f.
- the traffic analysis device 140 feeds back the traffic query result to the user.
- FIG. 10 is a schematic flow diagram of a traffic analysis device 140 provided in the present application to query the traffic of a certain data flow passing through a certain forwarding network element 120. As shown in FIG. 10 , the process includes the following steps:
- the traffic analysis device 140 receives a query request input by a user including an ID e of a forwarding network element e and an ID f of a data flow f.
- the traffic analysis device 140 uses the hash function f1 to hash the ID e , so as to determine a first traffic storage table corresponding to the ID e among multiple updated first traffic storage tables.
- the traffic analysis device 140 uses hash functions h 11 , h 12 , ..., h 1d to hash ID f , thereby determining d storage cells corresponding to ID f in the first traffic storage table corresponding to ID e .
- the traffic analysis device 140 determines the minimum statistical value in the d storage cells corresponding to the ID f as the traffic query result.
- the traffic analysis device 140 feeds back the traffic query result to the user.
- FIG. 11 is a schematic structural diagram of a flow measurement device 200 provided in the present application.
- the execution body of the flow measurement method provided in the present application is the server 130 in the flow measurement system shown in FIG. 2
- the flow measurement The device 200 is applied to the server 130
- the flow measurement device 200 is applied to the statistical device 150 when the execution subject of the flow measurement method provided in this application is the statistical device 150 in the flow measurement system shown in FIG. 8 .
- the flow measurement device 200 includes:
- the receiving module 210 is configured to receive the message sent by the forwarding network element 120 connected to the server 130, the message includes the identifier of the data flow, and the message carries the multiple forwarding network
- the element 120 processes the information of the message, that is, the second carrying information mentioned above;
- the acquiring module 220 is configured to acquire a traffic storage table, that is, a first traffic storage table or a second traffic storage table, where the traffic storage table is used to store the number of packets in the data stream;
- a statistics module 230 configured to count the number of packets in the data flow into the traffic storage table according to the identifier of the data flow and the information on processing the packets by the plurality of forwarding network elements 120, and update them After the flow storage table;
- the sending module 240 is configured to send the updated traffic storage table to the traffic analysis device 140 .
- the receiving module 210 is configured to receive the message sent by the forwarding network element 120 connected to the statistical device 150, the message includes the identifier of the data flow, and the message carries the multiple forwarding network
- the element 120 processes the information of the message, that is, the second carrying information mentioned above;
- the acquiring module 220 is configured to acquire a flow storage table, that is, a first flow storage table or a second flow storage table, and the flow storage table is used to store the number of packets in the data stream;
- a statistics module 230 configured to count the number of packets in the data flow into the traffic storage table according to the identifier of the data flow and the information on processing the packets by the plurality of forwarding network elements 120, and update them After the flow storage table;
- the sending module 240 is configured to send the updated traffic storage table to the traffic analysis device 140 .
- the flow measurement device 200 further includes: an encapsulation module 250, configured to encapsulate the updated flow storage table.
- the identifier of the data flow includes one or more of the following combinations: the IP address of the client, the port number through which the client sends the data flow, the IP address of the server address, the port number of the server receiving the data flow, the transport layer protocol used by the client to transmit the data flow to the server, and the VLAN identifier.
- the information on processing the packet by each forwarding network element 120 includes one or more of the following combinations:
- each forwarding network element 120 The identifier of each forwarding network element 120, the port number of each forwarding network element 120 receiving the message, the port number of each forwarding network element 120 sending the message, the message entering The queue number of the message, the queue number from which the message leaves, the time when each forwarding network element 120 receives the message, and the time when each forwarding network element 120 sends the message.
- the flow measurement device 200 is only one example provided herein, and that the flow measurement device 200 may have more or fewer components than those shown in FIG. 11 , two or more components may be combined, or It can be realized with different configurations of components.
- the present application also provides a computing device cluster 30, and the computing device cluster 30 can be used to deploy the flow measuring device 200 shown in FIG. 11 to execute the flow measuring method provided in the present application.
- the computing device cluster 30 includes at least one computing device 300 .
- the computing device cluster 30 only includes one computing device 300
- all the modules in the flow measurement device shown in FIG. 11 can be deployed in the one computing device 300: receiving module 210, acquiring module module 230 , sending module 240 and packaging module 250 .
- each computing device 300 in the multiple computing devices 300 can be used to deploy some modules in the flow measuring device 200 shown in FIG.
- Two or more computing devices 300 in the computing devices 300 are jointly used to deploy one or more modules in the flow measuring device 200 shown in FIG. 11 .
- the computing device 300A can be used to deploy the receiving module 210 and the acquiring module 220, and the computing device 300B can be used to deploy the statistics module 230, the sending module 240 and the The encapsulation module 250, or, the receiving module 210, the acquisition module 220, and the statistics module 230 are deployed on the computing device 300A, and the statistics module 230, the sending module 240, and the encapsulation module 250 are deployed on the computing device 300B; it is assumed that multiple computing devices 300 include the computing device 300A , 300B, 300C and 300D, then the computing device 300A can be used to deploy the receiving module 210, the computing device 300B can be used to deploy the acquiring module 220, the computing device 300C can be used to deploy the statistics module 230, and the computing device 300D can be used to deploy the sending module 240 and packaging module 250.
- At least one computing device 300 included in the computing device cluster 30 may be all terminal devices, all may be cloud servers, or partly be cloud servers and partly be terminal devices, which are not specifically limited here.
- each computing device 300 in the computing device cluster 30 may include a processor 310, a memory 320, a communication interface 330, etc., and the memory 320 in one or more computing devices 300 in the computing device cluster 30
- the processor 310 may read the codes from the memory 320 and execute the codes to realize the present application.
- the communication interface 330 may be used to implement communication between each computing device 300 and other devices.
- each computing device 300 in the computing device cluster 30 may also communicate with other devices through a network connection.
- the network may be a wide area network or a local area network or the like.
- the computing device 300 includes: a processor 310 , a memory 320 and a communication interface 330 , wherein the processor 310 , the memory 320 and the communication interface 330 may be connected to each other through a bus 340 .
- the processor 310 , the memory 320 and the communication interface 330 may be connected to each other through a bus 340 .
- the processor 310 can read the code stored in the memory 320, and cooperate with the communication interface 330 to execute some or all steps of the flow measurement method performed by the flow measurement device 200 in the above embodiments of the present application.
- the processor 310 can have multiple specific implementation forms, for example, the processor 310 can be a central processing unit (central processing unit, CPU) or a graphics processing unit (graphics processing unit, GPU), and the processor 310 can also be a single-core processor or multi-core processor.
- the processor 310 may be a combination of a CPU and a hardware chip.
- the aforementioned hardware chip may be an application-specific integrated circuit (application-specific integrated circuit, ASIC), a programmable logic device (programmable logic device, PLD) or a combination thereof.
- the aforementioned PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a general array logic (generic array logic, GAL) or any combination thereof.
- the processor 310 may also be implemented solely by a logic device with built-in processing logic, such as an FPGA or a digital signal processor (digital signal processing, DSP).
- the memory 320 can store codes as well as data.
- the code includes: the code of the receiving module 210, the code of the acquiring module 220, the code of the statistics module 230, the code of the sending module 240, the code of the packaging module 250, etc.
- the data includes: the second carrying information, the first flow storage table, The updated first flow storage table, the second flow storage table, the updated second flow storage table, and so on.
- the memory 320 can be a non-volatile memory, for example, a read-only memory (read-only memory, ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
- the memory 320 can also be a volatile memory, and the volatile memory can be a random access memory (random access memory, RAM), which is used as an external cache.
- the communication interface 330 can be a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or using a wireless local area network interface) for communicating with other computing nodes or devices.
- the communication interface 330 can adopt a protocol family above the transmission control protocol/internet protocol (transmission control protocol/internet protocol, TCP/IP), for example, a remote function call (remote function call, RFC) protocol, simple object access protocol (simple object access protocol, SOAP) protocol, simple network management protocol (simple network management protocol, SNMP) protocol, common object request broker architecture (common object request broker architecture, CORBA) protocol and distributed protocol and many more.
- TCP/IP transmission control protocol/internet protocol
- RFC remote function call
- simple object access protocol simple object access protocol
- SOAP simple network management protocol
- SNMP simple network management protocol
- CORBA common object request broker architecture
- the bus 340 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA for short) bus or the like.
- PCI peripheral component interconnect
- EISA extended industry standard architecture
- the bus 340 can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 13 , but it does not mean that there is only one bus or one type of bus.
- the above-mentioned computing device 300 is used to execute the method in the above-mentioned embodiment of the flow measurement method, which belongs to the same concept as the above-mentioned method embodiment, and its specific implementation process is detailed in the above-mentioned method embodiment, and will not be repeated here.
- computing device 300 is only an example provided by the embodiment of the present application, and the computing device 300 may have more or fewer components than those shown in FIG. 13 , may combine two or more components, or It can be realized with different configurations of components.
- the embodiment of the present application also provides a non-transitory computer-readable storage medium, in which codes are stored, and when it is run on a processor, the flow measurement method described in the above-mentioned embodiments can be realized some or all of the steps.
- all or part may be implemented by software, hardware or any combination thereof.
- software When implemented using software, it may be implemented in whole or in part in the form of a computer program product.
- the computer program product may comprise code.
- the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices.
- the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, DSL) or wireless (eg, infrared, wireless, microwave, etc.) means.
- the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media.
- the available medium may be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium, or a semiconductor medium.
- the steps in the method of the embodiment of the present application can be adjusted in order, combined or deleted according to actual needs; the units in the device of the embodiment of the present application can be divided, combined or deleted according to actual needs.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
本申请提供一种流量测量方法、装置及相关设备,其中,方法包括:服务端接收与其连接的转发网元发送的报文并获取流量存储表,其中,报文包括数据流的标识和多个转发网元处理报文的信息,流量存储表用于存储数据流中报文的数量,然后服务端根据数据流的标识和多个转发网元处理报文的信息,将数据流中报文的数量统计到流量存储表,得到更新后的流量存储表,最后,服务端向流量分析设备发送更新后的流量存储表。该方法能够解决现有技术存在的测量的链路信息不完整的问题以及会消耗大量带宽资源和计算资源的问题。
Description
本申请要求于2021年5月25日提交中国专利局、申请号为202110573832.2、发明名称为“流量测量方法、装置及相关设备”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及通信领域,尤其涉及一种流量测量方法、装置及相关设备。
流量测量是网络管理的重要环节,为服务质量提升、容量规划、网络计费、拥塞控制、数据中心和主干网中的异常检测提供了不可或缺的信息。例如,当网络发生拥塞时,通过流量测量可以尽快找到造成拥塞的数据流的信息。
目前比较流行的流量测量方法为带内网络遥测(in-band network telemetry,INT)方法,但是,本申请发明人发现该方法存在着测量的链路信息不完整的问题,以及会消耗大量带宽资源和计算资源,测量成本高的问题。
发明内容
本申请提供了一种流量测量方法、装置及相关设备,可以解决现有技术存在的测量的链路信息不完整的问题以及会消耗大量带宽资源和计算资源的问题。
第一方面,提供了一种流量测量方法,应用于客户端所生成的数据流经多个转发网元传输至服务端的传输过程,所述服务端连接流量分析设备,所述多个转发网元中的每个转发网元为支持在所述数据流中的报文上添加其处理所述报文的信息的网络设备,所述方法包括:
所述服务端接收与其连接的转发网元发送的所述报文,所述报文包括所述数据流的标识和所述多个转发网元处理所述报文的信息;
所述服务端获取流量存储表,所述流量存储表用于存储所述数据流中报文的数量;
所述服务端根据所述数据流的标识和所述多个转发网元处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;
所述服务端向所述流量分析设备发送所述更新后的流量存储表。
上述方案中,服务端接收的报文包括的多个转发网元处理报文的信息中,包含了与服务端连接的转发网元(即报文到达服务端之前经过的最后一个转发网元,以下简称为最后一个转发网元)处理报文的信息,也就是说,服务端在将数据流中的报文的数量统计到流量存储表时,统计了最后一个转发网元处理报文的信息,不像现有技术中,与服务端连接的最后一个转发网元在向流量分析设备发送信息时,并没有将自己处理报文的信息发送给流量分析设备,导致流量测量的链路信息不完整。
另外,上述方案中,服务端在接收到流标识和多个转发网元处理报文的信息后,根据流标识和多个转发网元处理报文的信息将数据流中报文的数量统计到流量存储表,得到更新后的流量存储表,然后向流量分析设备发送更新后的流量存储表,不像现有技术中,是由最后一个转发网元直接将流标识和多个转发网元处理报文的信息发送给流量分析设备,由流量分析设备根据最后一个转发网元发送的流标识和多个转发网元处理报文的信息进行流量统计。 因此,上述方案相较于现有技术,可以减少向流量分析设备发送数据所需要消耗的带宽资源,以及无需流量分析设备进行流量统计,可以减少对流量分析设备的计算资源的消耗,起到降低测量成本的作用。
在一种可能的实现方式中,所述服务端向所述流量分析设备发送所述更新后的流量存储表之前,所述方法还包括:所述服务端对所述更新后的流量存储表进行封装。
上述方案中,服务端对更新后的流量存储表进行封装,然后向流量分析设备发送封装后的流量存储表,可以进一步减少向流量分析设备发送数据所需要消耗的带宽资源,进一步降低测量成本。
在一种可能的实现方式中,所述数据流的标识包括如下的一种或多种组合:所述客户端的互联协议(internet protocol,IP)地址、所述客户端发送所述数据流的端口号、所述服务端的IP地址、所述服务端接收所述数据流的端口号、所述客户端向所述服务端传输所述数据流所使用的传输层协议、虚拟局域网(virtual local area network,VLAN)标识。
上述方案中,数据流的标识有多种组合方式,因此可以实现不同维度的流量测量,如实现测量每个客户端发送的数据流的流量、实现测量每个客户端的每个端口发送的数据流的流量等。
在一种可能的实现方式中,所述每个转发网元处理所述报文的信息包括如下的一种或多种组合:所述每个转发网元的标识、所述每个转发网元接收所述报文的端口号、所述每个转发网元发送所述报文的端口号、所述报文进入的队列号、所述报文离开的队列号、所述每个转发网元接收所述报文的时间、所述每个转发网元发送所述报文的时间。
上述方案中,每个转发网元处理报文的信息有多种组合方式,因此可以实现不同维度的流量测量,如实现测量每个转发网元上经过的数据流的流量、实现测量每个转发网元的每个端口接收到的数据流的流量等。
第二方面,提供一种流量测量方法,应用于客户端所生成的数据流经多个转发网元传输至服务端的传输过程,统计设备连接所述多个转发网元中将所述数据流发送给所述服务端的最后一个转发网元,所述统计设备连接流量分析设备,所述多个转发网元中的每个转发网元为支持在所述数据流中的报文上添加其处理所述报文的信息的网络设备,所述方法包括:
所述统计设备接收所述最后一个转发网元发送的报文,所述报文包括所述数据流的标识,所述报文携带有所述多个转发网元处理所述报文的信息;
所述统计设备获取流量存储表,所述流量存储表用于存储所述数据流中报文的数量;
所述统计设备根据所述数据流的标识和所述多个转发网元处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;
所述统计设备向所述流量分析设备发送所述更新后的流量存储表。
在一种可能的实现方式中,所述统计设备向所述流量分析设备发送所述更新后的流量存储表之前,所述方法还包括:所述统计设备对所述更新后的流量存储表进行封装。
在一种可能的实现方式中,所述数据流的标识包括如下的一种或多种组合:所述客户端的IP地址、所述客户端发送所述数据流的端口号、所述服务端的IP地址、所述服务端接收所述数据流的端口号、所述客户端向所述服务端传输所述数据流所使用的传输层协议、VLAN标识。
在一种可能的实现方式中,所述每个转发网元处理所述报文的信息包括如下的一种或多种组合:所述每个转发网元的标识、所述每个转发网元接收所述报文的端口号、所述每个转发网元发送所述报文的端口号、所述报文进入的队列号、所述报文离开的队列号、所述每个 转发网元接收所述报文的时间、所述每个转发网元发送所述报文的时间。
第三方面,提供一种流量测量装置,应用于客户端所生成的数据流经多个转发网元传输至服务端的传输过程,具体应用于所述服务端,所述服务端还连接流量分析设备,所述多个转发网元中的每个转发网元为支持在所述数据流中的报文上添加其处理所述报文的信息的网络设备,所述装置包括:
接收模块,用于接收与所述服务端连接的转发网元发送的所述报文,所述报文包括所述数据流的标识,所述报文携带有所述多个转发网元处理所述报文的信息;
获取模块,用于获取流量存储表,所述流量存储表用于存储所述数据流中报文的数量;
统计模块,用于根据所述数据流的标识和所述多个转发网元处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;
发送模块,用于向所述流量分析设备发送所述更新后的流量存储表。
在一种可能的实现方式中,所述装置还包括:封装模块,用于对所述更新后的流量存储表进行封装。
在一种可能的实现方式中,所述数据流的标识包括如下的一种或多种组合:所述客户端的IP地址、所述客户端发送所述数据流的端口号、所述服务端的IP地址、所述服务端接收所述数据流的端口号、所述客户端向所述服务端传输所述数据流所使用的传输层协议、VLAN标识。
在一种可能的实现方式中,所述每个转发网元处理所述报文的信息包括如下的一种或多种组合:所述每个转发网元的标识、所述每个转发网元接收所述报文的端口号、所述每个转发网元发送所述报文的端口号、所述报文进入的队列号、所述报文离开的队列号、所述每个转发网元接收所述报文的时间、所述每个转发网元发送所述报文的时间。
第四方面,提供一种流量测量装置,应用于客户端所生成的数据流经多个转发网元传输至服务端的传输过程,具体应用于统计设备,所述统计设备连接所述多个转发网元中将所述数据流发送给所述服务端的最后一个转发网元,所述统计设备还连接流量分析设备,所述多个转发网元中的每个转发网元为支持在所述数据流中的报文上添加其处理所述报文的信息的网络设备,所述装置包括:
接收模块,用于接收所述最后一个转发网元发送的报文,所述报文包括所述数据流的标识,所述报文携带有所述多个转发网元处理所述报文的信息;
获取模块,用于获取流量存储表,所述流量存储表用于存储所述数据流中报文的数量;
统计模块,用于根据所述数据流的标识和所述多个转发网元处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;
发送模块,用于向所述流量分析设备发送所述更新后的流量存储表。
第五方面,提供了一种非瞬态计算机可读存储介质,所述非瞬态计算机可读存储介质存储有指令,所述指令用于实现如上述第一方面任意可能的实现方式或者第二方面任意可能的实现方式提供的方法。
第六方面,提供了一种计算机程序产品,包括计算机程序,当所述计算机程序被计算机设备集群读取并执行时,使得所述计算机设备集群执行如上述第一方面任意可能的实现方式或者第二方面任意可能的实现方式提供的方法。
第七方面,提供了一种计算设备集群,包括至少一个计算设备,每个计算设备包括处理器和存储器;至少一个计算设备的处理器用于执行至少一个计算设备的存储器中存储的指令,以使得该计算设备执行如上述第一方面任意可能的实现方式或者第二方面任意可 能的实现方式提供的方法。
在一种可能的实现方式中,该计算设备集群包括一个计算设备,该计算设备包括处理器和存储器;该处理器用于执行该存储器中存储的指令,以使得该计算设备执行如上述第一方面任意可能的实现方式或者第二方面任意可能的实现方式提供的方法。
在一种可能的实现方式中,该计算设备集群包括至少两个计算设备,每个计算设备包括处理器和存储器;该至少两个计算设备的处理器用于执行该至少两个计算设备的存储器中存储的指令,以使得该计算设备集群执行如上述第一方面任意可能的实现方式或者第二方面任意可能的实现方式提供的方法。
为了更清楚地说明本申请实施例技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍。
图1是本申请涉及的第一流量存储表的示意图;
图2是本申请涉及的流量测量***的结构示意图;
图3是本申请提供的一种流量测量方法的流程示意图;
图4是本申请提供的第一流量存储表的变化示意图;
图5是本申请提供的第二流量存储表的示意图;
图6是本申请提供的示例性的第一流量存储表和示例性的第二流量存储表的示意图;
图7是本申请提供的另一种流量测量方法的流程示意图;
图8是本申请提供的一种流量测量***的结构示意图;
图9是本申请提供的一种流量查询过程的示意图;
图10是本申请提供的一种流量查询过程的示意图;
图11是本申请提供的一种流量测量装置的结构示意图;
图12是本申请提供的一种计算设备集群的结构示意图;
图13是本申请提供的一种计算设备的结构示意图。
下面结合本申请实施例中的附图对本申请实施例进行描述。
本申请实施例中的术语“第一”、“第二”仅用于描述目的,而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此,限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。
本申请实施例中,“至少一个”是指一个或者多个,“多个”是指两个或两个以上。“和/或”,描述关联对象的关联关系,表示可以存在三种关系,例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B的情况,其中A,B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下中的至少一项(个)”或其类似表达,是指的这些项中的任意组合,包括单项(个)或复数项(个)的任意组合。例如,a,b或c中的至少一项(个),可以表示:a、b、c、a-b、a-c、b-c或a-b-c,其中a、b、c可以是单个,也可以是多个。
下面先对本申请涉及的概念或者术语等进行介绍。
(1)数据流的流量,也可以称为数据流的长度,也可以简称为流量,指一条数据流中包括的报文(也可以称为数据包)的数量。
(2)五元组,是由源IP地址、源端口号、目的IP地址、目的端口号和传输层协议这五个参数组成的一个集合。其中,源IP地址指发送出数据流的客户端的IP地址,源端口号指客户端发送出数据流的端口号,目的IP地址指接收数据流的服务端的IP地址,目的端口号指服务端接收数据流的端口号,传输层协议指客户端和服务端在传输数据流时使用的协议。五元组能够区分不同的数据流,并且对应的数据流是唯一的。
(3)INT,是一种流量测量协议,它在数据平面收集每一个INT网元(指支持INT技术的交换机或者路由器等网络设备)处理报文的信息(如INT网元的标识(identity document,ID)、报文出入INT网元的端口号、报文出入INT网元的队列号、报文出入INT网元的时间等),然后由最后一个INT网元将收集到的链路上的INT网元处理报文的信息发送给流量分析设备,从而实现对网络状态细粒度的获取,这个过程没有控制层面的控制与参与,可以大大减少控制器的压力。
(4)概率数据结构,是一种基于哈希(也可以称为散列)的数据结构,概率数据结构以牺牲一定的流量测量准确度为代价,记录网络中每条数据流的信息而非每个报文的信息,可以减少内存开销。概率数据结构通过设置哈希函数(也可以称为散列函数),将具有相同哈希值(也可以称为散列值)的键值数据保存在第一流量存储表(参见图1)中相同的存储格中。存储格内的统计值作为流量测量结果,是每条数据流的真实流量的估计。忽略掉发生哈希碰撞(collision)的情况,错误可以在一定的阈值下得到很好的控制。与无错方法相比,这些算法使用的内存更少。简而言之,概率数据结构具备在理论上可证明的流量估计精度与内存的平衡特性。目前比较常用的概率数据结构包括:布隆过滤器(bloom filter)、基数估计法、sketch,其中,sketch有最小简图(count-min sketch,简称为cm sketch)、保守更新简图(conservative-update sketch,简称为cu sketch)、计数简图(count sketch)、FlowRadar、UnivMon等。目前,在流量测量领域最流行的概率数据结构为cm sketch。
(5)第一流量存储表,如图1所示,包括d个(行)存储层,每个存储层包括w个的存储格(也可以称为哈希桶)。第一流量存储表中的存储格的位数均为δ,因此,每个存储格可以记录的流量范围是相同的。每个存储层关联一个哈希函数,d个存储层关联的哈希函数h
11、h
12、…、h
1d两两独立。其中,d、w均为大于0的自然数,通常,δ为8比特(bit),也就是说,存储格可以记录的流量范围为0~255。在初始状态下,第一流量存储表上所有存储格的初始值为0或者为空。
(6)cm sketch的工作原理为:当一个包括流标识的报文到达部署了cm sketch的设备时,设备使用h
11、h
12、…、h
1d对报文包括的流标识进行哈希,从而在第一流量存储表中定位到d个存储格,然后设备将定位到的d个存储格中统计值未溢出的存储格的统计值增加1,统计值已溢出的存储格的统计值保持不变,从而得到更新后的第一流量存储表。当设备对一条数据流中的最后一个报文执行完上述操作的情况下,该d个存储格中的最小统计值即为设备对该数据流的真实流量的估计值。
其中,统计值未溢出,表示统计值小于统计值所在的存储格可以记录的最大值,统计值已溢出,表示统计值等于统计值所在的存储格可以记录的最大值,例如,假设存储格A的统计值A'为100,存储格B的统计值B'为1000,存储格A和B可以记录的最大值均为255,A'小于255,则A'未溢出,B'等于255,则B'已溢出。
其他概率数据结构的工作原理与cm sketch的工作原理相类似,此处不再展开赘述。
下面介绍本申请提供的流量测量方法、装置及相关设备可以应用的场景。
参见图2,图2为本申请涉及的一种流量测量***,如图2所示,该***包括:客户端 110、多个转发网元120、服务端130和流量分析设备140,客户端110、多个转发网元120、服务端130和流量分析设备140之间可以通过网络进行通信,网络可以是广域网、局域网、点对点连接等方式,或它们的任意组合。
客户端110,指发送出数据流的设备,客户端110发出的数据流可以经由多个转发网元120中的部分或者全部进行转发,最终到达服务端130。客户端110可以为终端设备,如智能手机、平板电脑、移动笔记本、可穿戴设备等,也可以为服务器,如个人计算机、云服务器等。客户端110的操作***可以是IOS、Android、Windows、Linux等,此处不作具体限定。
转发网元120是用于实现数据流转发任务的网络设备,如交换机或者路由器等。
服务端130,指接收数据流的设备,可以为终端设备,如智能手机、平板电脑、移动笔记本、可穿戴设备等,也可以为个人计算机、服务器等。服务端130的操作***可以是IOS、Android、Windows、Linux等,此处不作具体限定。
流量分析设备140,是用于实现流量分析功能的设备。流量分析设备140可以对数据流的传输过程进行监控,获得流量分析结果。流量分析设备140在获得流量分析结果后,可以接收用户输入的查询请求,向用户反馈流量查询结果。流量分析设备140可以为个人计算机、服务器等。
目前,图2所示的流量测量***通常使用INT方法实现流量测量任务。在使用INT方法时,图2所示***中的多个转发网元120是INT网元,图2所示***实现流量测量的过程为:
当报文由客户端110发出到达第一个转发网元120时,第一个转发网元120会在报文上添加数据流的标识(以下简称为流标识)和第一个转发网元120处理报文的信息。其中,流标识可以为五元组包括的五个参数中的任意一个或者多个,还可以为VLAN标识(identity document,ID)等,转发网元120处理报文的信息可以为转发网元120的ID、报文出入转发网元120的端口号、报文出入转发网元120的队列号、报文出入转发网元120的时间等。通常,第一个转发网元120在属于同一条数据流的报文上添加的流标识是相同的。
在添加完流标识和第一个转发网元120处理报文的信息后,第一个转发网元120向中转网元120(指报文在第一个转发网元120之后,到达最后一个转发网元120之前经过的转发网元120)发送携带有第一个转发网元120处理报文的信息的报文。当报文转发至中转网元120时,中转网元120在报文上继续添加本网元处理报文的信息,然后将添加了本网元处理报文的信息的报文向下一个转发网元120转发。
当报文转发至最后一个转发网元120时,最后一个转发网元120将报文携带的报文经过的转发网元120(即第一个转发网元120和中转网元120)处理报文的信息(以下简称为第一携带信息)从报文上剥离,得到独立的第一携带信息和报文,然后将第一携带信息发送给流量分析设备140,并将报文转发到服务端130。
在具体实现中,最后一个转发网元120通常是在从多个报文上剥离得到多个第一携带信息后,将多个第一携带信息一起发送给流量分析设备140。流量分析设备140在接收到多个第一携带信息后,根据多个第一携带信息进行分析统计,得到流量统计结果,如每条数据流的流量、每个转发网元120上经过的每条数据流的流量等。
上述INT方法在实现流量测量时,最后一个转发网元120在将第一携带信息发送至流量分析设备140之前,并没有在报文上继续添加自身处理报文的信息,也就是说,第一携带信息中不包括最后一个转发网元120处理报文的信息,流量分析设备140接收的最后一个转发网元120发送的第一携带信息不包括最后一个转发网元120处理报文的信息,因此,该方法存在着测量的链路信息不完整的问题。
另外,最后一个转发网元120是将剥离得到的多个第一携带信息一起发送给流量分析设备140,若多个第一携带信息的数据量非常大,这不仅会消耗大量带宽资源,而且还会消耗流量分析设备140的大量计算资源,测量成本较高。
针对上述问题,本申请提供一种流量测量方法,该流量测量方法可以应用于图2所示的流量测量***,图2所示***中的多个转发网元120均为支持在数据流中的报文上添加其处理报文的信息的网元,如INT网元,服务端130上部署有概率数据结构,图2所示***实现流量测量的过程为:
当报文由客户端110发出到达第一个转发网元120时,第一个转发网元120会在报文上添加流标识和第一个转发网元120处理报文的信息。
在添加完流标识和第一个转发网元120处理报文的信息后,第一个转发网元120向中转网元120发送携带有第一个转发网元120处理报文的信息的报文。当报文转发至中转网元120时,中转网元120在报文上继续添加本网元处理报文的信息,然后将添加了本网元处理报文的信息的报文向下一个转发网元120转发。
当报文转发至最后一个转发网元120时,最后一个转发网元120在报文上继续添加本网元处理报文的信息,然后将添加了本网元处理报文的信息的报文向服务端130转发。
当报文转发至服务端130时,服务端130将报文携带的报文经过的转发网元120(即第一个转发网元120、中转网元120和最后一个转发网元120)处理报文的信息(以下简称为第二携带信息)从报文上剥离,得到独立的第二携带信息和报文。
服务端130在剥离得到第二携带信息后,由于服务端130上部署有概率数据结构,因此服务端130可以根据第二携带信息将数据流的流量统计到第一流量存储表,得到更新后的第一流量存储表。然后,服务端130将获取的更新后的第一流量存储表发送给流量分析设备140。
在具体实现中,服务端130可以在从多个报文上剥离得到多个第二携带信息后,根据多个第二携带信息将数据流的流量统计到第一流量存储表,得到更新后的第一流量存储表,更新后的流量存储表中可以包括每条数据流的流量、每个转发网元120上经过的每条数据流的流量等。
可以看出,本申请提供的流量测量方法在实现流量测量时,并不是由最后一个转发网元120进行第二携带信息的剥离操作,而是由服务端130进行第二携带信息的剥离操作,且服务端130剥离得到的第二携带信息包括最后一个转发网元120处理报文的信息,因此,该方法可以解决现有的INT方法存在的测量的链路信息不完整的问题。
另外,在本申请提供的流量测量方法中,服务端130是根据第二携带信息将数据流的流量统计到第一流量存储表,得到更新后的第一流量存储表后,将更新后的第一流量存储表发送给流量分析设备140,不像现有的INT方法,是由最后一个转发网元120直接将第一携带信息发送给流量分析设备140,由流量分析设备140对第一携带信息进行分析汇总得到数据流的流量。由上文对概率数据结构和第一流量存储表的介绍可知,使用第一流量存储表统计数据流的流量可以减少内存开销,因此,服务端130得到的更新后的第一流量存储表的数据量要小于第二携带信息的数据量,服务端130将更新后的第一流量存储表发送给流量分析设备140,相较于现有技术,可以减少向流量分析设备140发送数据所需要消耗的带宽资源,而且更新后的第一流量存储表中已经包括数据流的流量,无需流量分析设备140进行分析,可以减少流量分析设备140侧的计算资源的消耗,降低测量成本。
为了便于更清楚地理解本申请提供的流量测量方法,这里以部署于服务端130的概率数据结构为cm sketch,服务端130接收到的最后一个转发网元120发送的报文序列为P1、P2、…、 Pn为例,结合图3所示的流程示意图对本申请提供的流量测量方法进行详细描述,如图3所示,该方法包括:
S101、服务端130接收最后一个转发网元120发送的包括第二携带信息Pi'的报文Pi。
其中,i、n均为自然数,1≤i≤n。
具体地,服务端130在接收到包括第二携带信息Pi'的报文Pi后,会从报文Pi上剥离得到第二携带信息Pi',第二携带信息Pi'包括流标识ID
Pi。
S102、服务端130使用哈希函数h
11、h
12、…、h
1d对第二携带信息Pi'包括的ID
Pi进行哈希,得到d个第一哈希值。
其中,d个第一哈希值为h
11(ID
Pi)、h
12(ID
Pi)、…、h
1d(ID
Pi)。
S103、服务端130根据d个第一哈希值,从第一流量存储表中确定第一目标存储格。
具体地,转发网元120可以先从第一流量存储表中确定d个与ID
Pi对应的存储格:第1个存储格=h
11(ID
Pi)%w,第二个存储格=h
12(ID
Pi)%w,…,第d个存储格=h
1d(ID
Pi)%w,然后,将d个与ID
Pi对应的存储格中统计值未溢出的存储格确定为第一目标存储格。
S104、服务端130将第一目标存储格的统计值增加1。
举例来讲,假设转发网元120在接收到报文Pi之前包括的第一流量存储表为图4中的表A,第一流量存储表中的每个存储格可以记录的流量范围为0~255,转发网元120在接收到报文Pi后,根据第二携带信息Pi'包括的流标识ID
Pi定位到的5个存储格为h
11(ID
Pi)%5=3,h
12(ID
Pi)%5=1,h
13(ID
Pi)%5=4,h
14(ID
Pi)%5=1,h
15(ID
Pi)%5=3,参见图4中的表B,表B中有阴影的表格表示定位到的存储格,从表B可以看出,统计值未溢出的存储格为第一个存储层中的第3个存储格、第三个存储层中的第4个存储格、第四个存储层中的第1个存储格和第五个存储层中的第3个存储格,其统计值对应为3、25、80、25,则转发网元120确定第一目标存储格为上述四个存储格,然后将上述四个存储格的统计值均增加1,更新后的第一流量存储表参见图4中的表C。
S105、服务端130对接收到的每个报文都执行步骤S101至步骤S104,得到更新后的第一流量存储表。
S106、服务端130向流量分析设备140发送更新后的第一流量存储表。
在本申请具体的实施例中,服务端130在向流量分析设备140发送更新后的第一流量存储表之前,还可以对更新后的第一流量存储表进行封装,以减少服务端130向流量分析设备140发送更新后的第一流量存储表时所占用的带宽。具体地,服务端130可以使用具有封装功能的算法(如elastic sketch),对更新后的第一流量存储表进行封装。
流量分析设备140在接收到服务端130发送的更新后的第一流量存储表之后,可以实现查询某条数据流的流量、查询显著流、查询某个时间窗口内服务端130接收到的数据流的条数或者查询某个时间窗口内服务端130接收到的报文的总数量等任务。以流量分析设备140实现查询某条数据流的流量为例,该过程参见图9以及下文对图9的相关描述。
本申请提供的流量测量方法也可以实现获取每个转发网元120上经过的每条数据流的流量,实现方式为:
由上文可知,服务端130接收到的第二携带信息除了包括流标识之外,还可以包括多个转发网元120处理报文的信息,多个转发网元120处理报文的信息可以为多个转发网元120的ID,因此,可以配置服务端130上部署的概率数据结构包括多个如图1所示的第一流量存储表,以及配置所述多个第一流量存储表关联一个哈希函数f
1。
当包括第二携带信息的报文到达了服务端130时,服务端130可以使用哈希函数f
1对第 二携带信息包括的每个转发网元120的ID进行哈希,从而在多个第一流量存储表中定位到与每个转发网元120对应的第一流量存储表,然后执行步骤S102至步骤S106,流量分析设备140便可以得到与每个转发网元120对应的更新后的第一流量存储表,从而实现每个转发网元120上经过的每条数据流的流量的获取。其中,在多个第一流量存储表中定位到与每个转发网元120对应的第一流量存储表T
1的公式为:T
1=f
1(转发网元120的ID)%m
1,m
1表示多个第一流量存储表的数量。
可以理解,流量分析设备140在接收到服务端130发送的多个转发网元120各自对应的更新后的第一流量存储表之后,可以实现查询某个转发网元120上经过的某条数据流的流量、查询某个时间窗口内某个转发网元120接收到的数据流的条数或者查询某个时间窗口内某个转发网元120接收到的报文的总数量等任务。以流量分析设备140实现查询某个转发网元120上经过的某条数据流的流量为例,该过程参见图10以及下文对图10的相关描述。
由上文对第一流量存储表的描述可知,第一流量存储表包括的存储格的位数δ是一样的,通常为8bit,因此,可以理解,在本申请提供的流量测量方法中,若服务端130上部署的概率数据结构为现有的cm sketch或者cu sketch等,则本申请提供的流量测量方法无论是对大数据流的流量进行测量还是对小数据流的流量进行测量,使用的存储格的位数是一样的。
但是,在真实的网络流通中,流量大小分布是高度倾斜的,倾斜意味着大多数数据流都是小数据流,这类数据流包括的报文数通常只有几个到一二百个,通常被称为鼠流,只有极少数的数据流是大数据流,这类数据流包括的报文数有好几百甚至上千上万个,通常被称为大象流。因此,本申请提供的流量测量方法相较于INT方法,虽然已经起到了节省内存资源的作用,但是内存资源的利用率仍然不够高,而且,在对大数据流的流量进行测量时测量的准确性也不够高。
举例来讲,假设第一流量存储表包括的存储格的位数均为8bit,小数据流A包括的报文数量为2,大数据流B包括的报文数量为1000,转发网元120在对小数据流A的流量进行测量时,统计值为2,2远远小于存储格可以记录的最大值255,这时内存是浪费的,转发网元120在对大数据流B的流量进行测量时,统计值达到255之后已经溢出,因此,转发网元120对大数据流B的流量估计值为255,这与真实的大数据流B的流量1000相差较大,大数据流B的流量测量不准确。
为了解决上述问题,本申请还提供另一种流量测量方法,在该流量测量方法中,部署于服务端130的概率数据结构不是现有的概率数据结构,而是一种新的概率数据结构(即下文所述tower sketch),该概率数据结构可以提高内存利用率,以及提高大数据流流量测量的准确性。
tower sketch,通过设置哈希函数,将具有相同哈希值的键值数据保存在第二流量存储表中的相同的存储格中。存储格内的统计值作为流量测量结果,是每条数据流的真实流量的估计。
第二流量存储表,如图5所示,可以看出,第二流量存储表与图1所示的第一流量存储表相同的地方在于,均包括d个存储层。第二流量存储表与第一流量存储表不同的地方在于,第二流量存储表的每个存储层包括不同个数的存储格,不同存储层包括的存储格的位数是不相同的,第一个存储层包括的存储格的个数为w
1,位数为δ
1,第二个存储层包括的存储格个数为w
2,位数为δ
2,…,第d个存储层包括的存储格的个数为w
d,位数为δ
d,因此,属于不同存储层的存储格可以记录的流量范围是不同的。每个存储层关联一个哈希函数,d个存储层关联的哈希函数h
21、h
22、…、h
2d两两独立。在本实施例中,图5所示的第二流量存储 表在初始状态下所有存储格的计数值均为0或者为空。
tower sketch的工作原理与上文所述cm sketch的工作原理相类似,具体可以参考上文对cm sketch的工作原理的相关描述。
下面以图6所示的流量存储表1和流量存储表2为例,对tower sketch可以提高内存利用率以及流量测量的准确性的原理进行解释说明。其中,图6所示的流量存储表1为一种示例性的第一流量存储表,图6所示的流量存储表2为一种示例性的第二流量存储表。从图6可以看出:
流量存储表1包括5个存储层,存储层11至存储层15均包括8个存储格,40个存储格的位数均为8bit,也就是说,该40个存储格可以记录的流量范围均为0~255。
流量存储表2也包括5个存储层,存储层21包括32个存储格,位数均为2bit,即存储层21包括的存储格可以记录的流量范围均为0~3,存储层22包括16个存储格,位数均为4bit,即存储层22包括的存储格可以记录的流量范围均为0~15,存储层23包括8个存储格,位数均为8bit,即存储层23包括的存储格可以记录的流量范围均为0~255,存储层24包括4个存储格,位数均为16bit,即存储层24包括的存储格可以记录的流量范围均为0~65535,存储层25包括2个存储格,位数均为32bit,即存储层25包括的存储格可以记录的流量范围均为0~4294967295。
由上述对流量存储表1的描述可以计算出,流量存储表1占用的内存=40*8bit=320bit,由上述对流量存储表2的描述可以计算出,流量存储表2占用的内存=2*32bit+4*16bit+8*8bit+16*4bit+32*2bit=320bit,也就是说,二者占用的内存是相同的。
在流量存储表1和流量存储表2占用的内存相同的情况下,由于流量存储表2每个存储层包括的存储格的位数是不同的,因此,设备在使用流量存储表2对小数据流的流量测量时,可以使用流量存储表2中与小数据流的流量匹配的存储格记录小数据流的流量,如在小数据流的流量为2时,使用占用内存为2bit的存储格记录该数据流的流量,如在小数据流的流量为12时,使用占用内存为4bit的存储格记录该数据流的流量,如在小数据流的流量为200时,使用占用内存为8bit的存储格记录该数据流的流量,相较于使用流量存储表1进行流量测量,减少了内存的浪费,提高了内存利用率。
同时,由于流量存储表2中的存储格可以记录的最大值为4294967295,因此,设备在使用流量存储表2对大数据流的流量测量时,可以使用与大数据流的流量匹配的存储格记录大数据流的流量,如在大数据流的流量为1000时,使用占用内存为16bit的存储格记录该数据流的流量,如在大数据流的流量为100000时,使用占用内存为32bit的存储格记录该数据流的流量,不会导致测量得到的大数据流的流量与真实流量相差较大,因此可以提高测量的大数据流的流量的准确性。
另外,由图6还可以看出,流量存储表2包括的存储格的个数多于流量存储表1包括的存储格的个数。可以理解,在流量存储表2包括的存储格的个数多于流量存储表1包括的存储格的个数的情况下,设备使用流量存储表2进行流量测量时发生哈希碰撞的几率也会低于使用流量存储表1进行流量测量时发生哈希碰撞的几率,因此,使用流量存储表2进行流量测量的准确性要高于使用流量存储表1进行流量测量的准确性。
这里以部署于服务端130的概率数据结构为tower sketch,服务端130接收到的最后一个转发网元120发送的报文序列为P1、P2、…、Pn为例,结合图7所示的流程示意图对本申请提供的另一种流量测量方法进行详细描述,如图7所示,该方法包括:
S201、服务端130接收最后一个转发网元120发送的包括第二携带信息Pi'的报文Pi。
S202、服务端130使用哈希函数h
21、h
22、…、h
2d对第二携带信息Pi'包括的ID
Pi进行哈希,得到d个第二哈希值。
其中,d个第二哈希值为h
21(ID
Pi)、h
22(ID
Pi)、…、h
2d(ID
Pi)。
S203、服务端130根据d个第二哈希值,从第二流量存储表中确定第二目标存储格。
具体地,服务端130可以先从第二流量存储表中确定d个与ID
Pi对应的存储格:第1个存储格=h
21(ID
Pi)%w
1、第2个存储格=h
22(ID
Pi)%w
2、…、第d个存储格=h
2d(ID
Pi)%w
d,然后将d个与ID
Pi对应的存储格中具有最小统计值且最小统计值未溢出的存储格确定为第二目标存储格。
S204、服务端130将第二目标存储格的统计值增加1。
S205、服务端130对接收到的每个报文都执行步骤S201至步骤S204,得到更新后的第二流量存储表。
S206、服务端130向流量分析设备140发送更新后的第二流量存储表。
本申请提供的另一种流量测量方法也可以实现获取每个转发网元120上经过的每条数据流的流量,实现方式为:
由上文可知,服务端130接收到的第二携带信息除了包括流标识之外,还可以包括多个转发网元120处理报文的信息,多个转发网元120处理报文的信息可以为多个转发网元120的ID,因此,可以配置服务端130上部署的概率数据结构包括多个如图5所示的第二流量存储表,以及配置所述多个第二流量存储表关联一个哈希函数f
2。
当包括携带信息的报文到达了服务端130时,服务端130可以使用哈希函数f
2对携带信息包括的每个转发网元120的ID进行哈希,从而在多个第二流量存储表中定位到与每个转发网元120对应的第二流量存储表,然后执行步骤S202至步骤S206,流量分析设备140便可以得到与每个转发网元120对应的更新后的第二流量存储表,从而实现每个转发网元120上经过的每条数据流的流量的获取。其中,在多个第二流量存储表中定位到与每个转发网元120对应的第二流量存储表T
2的公式为:T
2=f
2(转发网元120的ID)%m
2,m
2表示多个第二流量存储表的数量。
可以理解,流量分析设备140在接收到服务端130发送的多个转发网元120各自对应的更新后的第二流量存储表之后,可以实现查询某个转发网元120上经过的某条数据流的流量、查询某个时间窗口内某个转发网元120接收到的数据流的条数或者查询某个时间窗口内某个转发网元120接收到的报文的总数量等任务。
还可以理解,本申请提供的流量测量方法还可以实现获取每个转发网元120上的每个端口接收的每条数据流的流量、获取每个客户端110发送的每条数据流的流量、获取每个转发网元120上的每个队列经过的每条数据流的流量等,其具体实现方式与上文所述实现获取每个转发网元120上经过的每条数据流的流量的方式相类似,可以参考上文相关描述,此处不再展开赘述。
需要说明的是,虽然上文在对本申请提供的流量测量方法进行描述时,均以服务端130为执行主体,但是,在本申请具体的实施例中,还可以在图2所示流量测量***中增加计算能力和存储能力均较强的统计设备150,作为本申请提供的流量测量方法的执行主体,如图8所示,统计设备150分别与最后一个转发网元120、流量分析设备140和服务端130连接。在具体实现中,统计设备150可以为个人计算机、服务器等。
统计设备150为执行主体时,其实现本申请提供的流量测量方法的过程与服务端130作为执行主体时的实现过程是相似的,具体可以参考上文中的相关描述,此处不再展开赘述。
图9为本申请提供的一种流量分析设备140实现查询某条数据流的流量的流程示意图,如图9所示,该过程包括如下步骤:
S301、流量分析设备140接收用户输入的包括数据流f的标识ID
f的查询请求。
S302、流量分析设备140使用哈希函数h
11、h
12、…、h
1d对ID
f进行哈希,从而在更新后的第一流量存储表中确定d个与ID
f对应的存储格。
S303、流量分析设备140将d个与ID
f对应的存储格中的最小统计值确定为流量查询结果。
可以理解,流量查询结果为对真实的数据流f的流量的估计。
S304、流量分析设备140将流量查询结果反馈给用户。
图10为本申请提供的一种流量分析设备140实现查询某个转发网元120上经过的某条数据流的流量的流程示意图,如图10所示,该过程包括如下步骤:
S401、流量分析设备140接收用户输入的包括转发网元e的ID
e和数据流f的标识ID
f的查询请求。
S402、流量分析设备140使用哈希函数f
1对ID
e进行哈希,从而在多个更新后的第一流量存储表中确定与ID
e对应的第一流量存储表。
S403、流量分析设备140使用哈希函数h
11、h
12、…、h
1d对ID
f进行哈希,从而在与ID
e对应的第一流量存储表中确定d个与ID
f对应的存储格。
S404、流量分析设备140将d个与ID
f对应的存储格中的最小统计值确定为流量查询结果。
S405、流量分析设备140将流量查询结果反馈给用户。
上文详细阐述了本申请提供的流量测量方法,基于相同的发明构思,下面继续阐述本申请提供的流量测量装置。
参见图11,图11是本申请提供的一种流量测量装置200的结构示意图,在本申请提供的流量测量方法的执行主体为图2所示流量测量***中的服务端130时,该流量测量装置200应用于服务端130,在本申请提供的流量测量方法的执行主体为图8所示流量测量***中的统计设备150时,该流量测量装置200应用于统计设备150。如图11所示,流量测量装置200包括:
(1)在该流量测量装置200应用于服务端130的情况下:
接收模块210,用于接收与所述服务端130连接的转发网元120发送的所述报文,所述报文包括所述数据流的标识,所述报文携带有所述多个转发网元120处理所述报文的信息,即上文所述第二携带信息;
获取模块220,用于获取流量存储表,即第一流量存储表或者第二流量存储表,所述流量存储表用于存储所述数据流中报文的数量;
统计模块230,用于根据所述数据流的标识和所述多个转发网元120处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;
发送模块240,用于向所述流量分析设备140发送所述更新后的流量存储表。
(2)在该流量测量装置200应用于统计设备150的情况下:
接收模块210,用于接收与所述统计设备150连接的转发网元120发送的所述报文,所述报文包括所述数据流的标识,所述报文携带有所述多个转发网元120处理所述报文的信息,即上文所述第二携带信息;
获取模块220,用于获取流量存储表,即第一流量存储表或者第二流量存储表,所述流 量存储表用于存储所述数据流中报文的数量;
统计模块230,用于根据所述数据流的标识和所述多个转发网元120处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;
发送模块240,用于向所述流量分析设备140发送所述更新后的流量存储表。
在一种可能的实现方式中,流量测量装置200还包括:封装模块250,用于对所述更新后的流量存储表进行封装。
在一种可能的实现方式中,所述数据流的标识包括如下的一种或多种组合:所述客户端的IP地址、所述客户端发送所述数据流的端口号、所述服务端的IP地址、所述服务端接收所述数据流的端口号、所述客户端向所述服务端传输所述数据流所使用的传输层协议、VLAN标识。
在一种可能的实现方式中,所述每个转发网元120处理所述报文的信息包括如下的一种或多种组合:
所述每个转发网元120的标识、所述每个转发网元120接收所述报文的端口号、所述每个转发网元120发送所述报文的端口号、所述报文进入的队列号、所述报文离开的队列号、所述每个转发网元120接收所述报文的时间、所述每个转发网元120发送所述报文的时间。
具体地,上述流量测量装置200执行各种操作的具体实现,可参照上述流量测量方法实施例中相关内容中的描述,为了说明书的简洁,这里不再赘述。
应当理解,流量测量装置200仅为本申请提供的一个例子,并且,流量测量装置200可具有比图11示出的部件更多或更少的部件,可以组合两个或更多个部件,或者可具有部件的不同配置实现。
本申请还提供一种计算设备集群30,所述计算设备集群30可以用于部署图11所示的流量测量装置200,以执行本申请提供的流量测量方法。如图12所示,该计算设备集群30包括至少一个计算设备300。
具体地,在所述计算设备集群30仅包括一个计算设备300的情况下,可以在该一个计算设备300中部署图11所示流量测量装置中的全部模块:接收模块210、获取模块220、统计模块230、发送模块240和封装模块250。
在所述计算设备集群30包括多个计算设备300的情况下,多个计算设备300中的每个计算设备300可以用于部署图11所示流量测量装置200中的部分模块,或者,多个计算设备300中的两个或者两个以上的计算设备300共同用于部署图11所示流量测量装置200中的一个或者多个模块。
举例来讲,假设多个计算设备300包括计算设备300A和计算设备300B,则计算设备300A可以用于部署接收模块210和获取模块220,计算设备300B可以用于部署统计模块230、发送模块240和封装模块250,或者,计算设备300A上部署接收模块210、获取模块220和统计模块230,计算设备300B上部署统计模块230、发送模块240和封装模块250;假设多个计算设备300包括计算设备300A、300B、300C和300D,则计算设备300A可以用于部署接收模块210,计算设备300B可以用于部署获取模块220,计算设备300C可以用于部署统计模块230,计算设备300D可以用于部署发送模块240和封装模块250。
在具体实现中,所述计算设备集群30中包括的至少一个计算设备300可以全部是终端设备,也可以全部是云服务器,还可以部分是云服务器部分是终端设备,此处不作具体限定。
更具体地,所述计算设备集群30中的每个计算设备300可以包括处理器310、存储器320以及通信接口330等,所述计算设备集群30中的一个或者多个计算设备300中的存储器320 可以存有相同的用于执行本申请实施例提供的流量测量方法的代码(也可以称为指令或者程序指令等),处理器310可以从存储器320中读取代码,并执行代码以实现本申请实施例提供的流量测量方法,通信接口330可以用于实现每个计算设备300与其他设备之间的通信。
在一些可能的实现方式中,计算设备集群30中的每个计算设备300也可以通过网络与其他设备连接进行通信。其中,所述网络可以是广域网或局域网等等。
下面以流量测量装置200的全部模块部署于一个计算设备300上为例,结合图13对本申请提供的计算设备300进行详细描述。
参见图13,计算设备300包括:处理器310、存储器320以及通信接口330,其中,处理器310、存储器320以及通信接口330之间可以通过总线340相互连接。其中,
处理器310可以读取存储器320中存储的代码,与通信接口330配合执行本申请上述实施例中由流量测量装置200执行的流量测量方法的部分或者全部步骤。
处理器310可以有多种具体实现形式,例如处理器310可以为中央处理器(central processing unit,CPU)或图形处理器(graphics processing unit,GPU),处理器310还可以是单核处理器或多核处理器。处理器310可以由CPU和硬件芯片的组合。上述硬件芯片可以是专用集成电路(application-specific integrated circuit,ASIC),可编程逻辑器件(programmable logic device,PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device,CPLD),现场可编程逻辑门阵列(field-programmable gate array,FPGA),通用阵列逻辑(generic array logic,GAL)或其任意组合。处理器310也可以单独采用内置处理逻辑的逻辑器件来实现,例如FPGA或数字信号处理器(digital signal processing,DSP)等。
存储器320可以存储有代码以及数据。其中,代码包括:接收模块210的代码、获取模块220的代码、统计模块230的代码、发送模块240的代码以及封装模块250的代码等,数据包括:第二携带信息、第一流量存储表、更新后的第一流量存储表、第二流量存储表、更新后的第二流量存储表等等。
在实际应用中,存储器320可以是非易失性存储器,例如,只读存储器(read-only memory,ROM)、可编程只读存储器(programmable ROM,PROM)、可擦除可编程只读存储器(erasable PROM,EPROM)、电可擦除可编程只读存储器(electrically EPROM,EEPROM)或闪存。存储器320也可以是易失性存储器,易失性存储器可以是随机存取存储器(random access memory,RAM),其用作外部高速缓存。
通信接口330可以为有线接口(例如以太网接口)或无线接口(例如蜂窝网络接口或使用无线局域网接口),用于与其他计算节点或装置进行通信。当通信接口330为有线接口时,通信接口330可以采用传输控制协议/网际协议(transmission control protocol/internet protocol,TCP/IP)之上的协议族,例如,远程函数调用(remote function call,RFC)协议、简单对象访问协议(simple object access protocol,SOAP)协议、简单网络管理协议(simple network management protocol,SNMP)协议、公共对象请求代理体系结构(common object request broker architecture,CORBA)协议以及分布式协议等等。
总线340可以是外设部件互连标准(peripheral component interconnect,PCI)总线或扩展工业标准结构(extended industry standard architecture,简称EISA)总线等。所述总线340可以分为地址总线、数据总线、控制总线等。为便于表示,图13中仅用一条粗线表示,但并不表示仅有一根总线或一种类型的总线。
上述计算设备300用于执行上述流量测量方法实施例中的方法,与上述方法实施例属于同一构思,其具体实现过程详见上述方法实施例,这里不再赘述。
应当理解,计算设备300仅为本申请实施例提供的一个例子,并且,计算设备300可具有比图13示出的部件更多或更少的部件,可以组合两个或更多个部件,或者可具有部件的不同配置实现。
本申请实施例还提供一种非瞬态计算机可读存储介质,非瞬态计算机可读存储介质中存储有代码,当其在处理器上运行时,可以实现上述实施例中记载的流量测量方法的部分或者全部步骤。
在上述实施例中,对各个实施例的描述都各有侧重,某个实施例中没有详述的部分,可以参见其它实施例的相关描述。
在上述实施例中,可以全部或部分地通过软件、硬件或者其任意组合来实现。当使用软件实现时,可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品可以包含代码。当计算机程序产品被计算机读取并执行时,可以实现上述方法实施例中记载的数据表热度区分方法的部分或者全部步骤。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中,或者从一个计算机可读存储介质向另一个计算机可读存储介质传输,例如,所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线)或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如软盘、硬盘、磁带)、光介质、或者半导体介质等。
本申请实施例方法中的步骤可以根据实际需要进行顺序调整、合并或删减;本申请实施例装置中的单元可以根据实际需要进行划分、合并或删减。
以上对本申请实施例进行了详细介绍,本文中应用了具体个例对本申请的原理及实施方式进行了阐述,以上实施例的说明只是用于帮助理解本申请的方法及其核心思想;同时,对于本领域的一般技术人员,依据本申请的思想,在具体实施方式及应用范围上均会有改变之处,综上所述,本说明书内容不应理解为对本申请的限制。
Claims (12)
- 一种流量测量方法,其特征在于,应用于客户端所生成的数据流经多个转发网元传输至服务端的传输过程,所述服务端连接流量分析设备,所述多个转发网元中的每个转发网元为支持在所述数据流中的报文上添加其处理所述报文的信息的网络设备,所述方法包括:所述服务端接收与其连接的转发网元发送的所述报文,所述报文包括所述数据流的标识和所述多个转发网元处理所述报文的信息;所述服务端获取流量存储表,所述流量存储表用于存储所述数据流中报文的数量;所述服务端根据所述数据流的标识和所述多个转发网元处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;所述服务端向所述流量分析设备发送所述更新后的流量存储表。
- 根据权利要求1所述的方法,其特征在于,所述服务端向所述流量分析设备发送所述更新后的流量存储表之前,所述方法还包括:所述服务端对所述更新后的流量存储表进行封装。
- 根据权利要求1或2所述的方法,其特征在于,所述数据流的标识包括如下的一种或多种组合:所述客户端的互联协议IP地址、所述客户端发送所述数据流的端口号、所述服务端的IP地址、所述服务端接收所述数据流的端口号、所述客户端向所述服务端传输所述数据流所使用的传输层协议、虚拟局域网VLAN标识。
- 根据权利要求1至3任一项所述的方法,其特征在于,所述每个转发网元处理所述报文的信息包括如下的一种或多种组合:所述每个转发网元的标识、所述每个转发网元接收所述报文的端口号、所述每个转发网元发送所述报文的端口号、所述报文进入的队列号、所述报文离开的队列号、所述每个转发网元接收所述报文的时间、所述每个转发网元发送所述报文的时间。
- 一种流量测量方法,其特征在于,应用于客户端所生成的数据流经多个转发网元传输至服务端的传输过程,统计设备连接所述多个转发网元中将所述数据流发送给所述服务端的最后一个转发网元,所述统计设备连接流量分析设备,所述多个转发网元中的每个转发网元为支持在所述数据流中的报文上添加其处理所述报文的信息的网络设备,所述方法包括:所述统计设备接收所述最后一个转发网元发送的报文,所述报文包括所述数据流的标识,所述报文携带有所述多个转发网元处理所述报文的信息;所述统计设备获取流量存储表,所述流量存储表用于存储所述数据流中报文的数量;所述统计设备根据所述数据流的标识和所述多个转发网元处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;所述统计设备向所述流量分析设备发送所述更新后的流量存储表。
- 一种流量测量装置,其特征在于,应用于客户端所生成的数据流经多个转发网元传输至服务端的传输过程,具体应用于所述服务端,所述服务端还连接流量分析设备,所述多个转发网元中的每个转发网元为支持在所述数据流中的报文上添加其处理所述报文的信息的网络设备,所述装置包括:接收模块,用于接收与所述服务端连接的转发网元发送的所述报文,所述报文包括所述数据流的标识,所述报文携带有所述多个转发网元处理所述报文的信息;获取模块,用于获取流量存储表,所述流量存储表用于存储所述数据流中报文的数量;统计模块,用于根据所述数据流的标识和所述多个转发网元处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;发送模块,用于向所述流量分析设备发送所述更新后的流量存储表。
- 根据权利要求6所述的装置,其特征在于,所述装置还包括:封装模块,用于对所述更新后的流量存储表进行封装。
- 根据权利要求6或7所述的装置,其特征在于,所述数据流的标识包括如下的一种或多种组合:所述客户端的IP地址、所述客户端发送所述数据流的端口号、所述服务端的IP地址、所述服务端接收所述数据流的端口号、所述客户端向所述服务端传输所述数据流所使用的传输层协议、VLAN标识。
- 根据权利要求6至8任一项所述的装置,其特征在于,所述每个转发网元处理所述报文的信息包括如下的一种或多种组合:所述每个转发网元的标识、所述每个转发网元接收所述报文的端口号、所述每个转发网元发送所述报文的端口号、所述报文进入的队列号、所述报文离开的队列号、所述每个转发网元接收所述报文的时间、所述每个转发网元发送所述报文的时间。
- 一种流量测量装置,其特征在于,应用于客户端所生成的数据流经多个转发网元传输至服务端的传输过程,具体应用于统计设备,所述统计设备连接所述多个转发网元中将所述数据流发送给所述服务端的最后一个转发网元,所述统计设备还连接流量分析设备,所述多个转发网元中的每个转发网元为支持在所述数据流中的报文上添加其处理所述报文的信息的网络设备,所述装置包括:接收模块,用于接收所述最后一个转发网元发送的报文,所述报文包括所述数据流的标识,所述报文携带有所述多个转发网元处理所述报文的信息;获取模块,用于获取流量存储表,所述流量存储表用于存储所述数据流中报文的数量;统计模块,用于根据所述数据流的标识和所述多个转发网元处理所述报文的信息,将所述数据流中报文的数量统计到所述流量存储表,得到更新后的流量存储表;发送模块,用于向所述流量分析设备发送所述更新后的流量存储表。
- 一种非瞬态计算机可读存储介质,其特征在于,所述非瞬态计算机可读存储介质存储有指令,所述指令用于实现权利要求1至4任一项所述的方法。
- 一种计算设备集群,其特征在于,包括至少一个计算设备,每个计算设备包括处理器和存储器;所述至少一个计算设备的处理器用于执行所述至少一个计算设备的存储器中存储的指令,以使得所述计算设备集群执行如权利要求1至4中任一项所述的方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110573832.2 | 2021-05-25 | ||
CN202110573832.2A CN115396345A (zh) | 2021-05-25 | 2021-05-25 | 流量测量方法、装置及相关设备 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022247308A1 true WO2022247308A1 (zh) | 2022-12-01 |
Family
ID=84114345
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/071728 WO2022247308A1 (zh) | 2021-05-25 | 2022-01-13 | 流量测量方法、装置及相关设备 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN115396345A (zh) |
WO (1) | WO2022247308A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115988574A (zh) * | 2023-03-15 | 2023-04-18 | 阿里巴巴(中国)有限公司 | 基于流表的数据处理方法、***、设备和存储介质 |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116389322B (zh) * | 2023-06-02 | 2023-08-15 | 腾讯科技(深圳)有限公司 | 流量数据处理方法、装置、计算机设备和存储介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108199924A (zh) * | 2018-01-26 | 2018-06-22 | 北京邮电大学 | 基于带内网络遥测的全网流量可视化方法及装置 |
US20200021490A1 (en) * | 2018-07-10 | 2020-01-16 | Cable Television Laboratories, Inc | Systems and methods for advanced core network controls |
US20200067792A1 (en) * | 2018-08-21 | 2020-02-27 | Argela Yazilim Ve Bilisim Teknolojileri San Ve Tic A S | System and method for in-band telemetry target selection |
CN112491661A (zh) * | 2020-12-11 | 2021-03-12 | 苏州浪潮智能科技有限公司 | 数据中心交换机的时延探测方法、装置、设备及介质 |
-
2021
- 2021-05-25 CN CN202110573832.2A patent/CN115396345A/zh active Pending
-
2022
- 2022-01-13 WO PCT/CN2022/071728 patent/WO2022247308A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108199924A (zh) * | 2018-01-26 | 2018-06-22 | 北京邮电大学 | 基于带内网络遥测的全网流量可视化方法及装置 |
US20200021490A1 (en) * | 2018-07-10 | 2020-01-16 | Cable Television Laboratories, Inc | Systems and methods for advanced core network controls |
US20200067792A1 (en) * | 2018-08-21 | 2020-02-27 | Argela Yazilim Ve Bilisim Teknolojileri San Ve Tic A S | System and method for in-band telemetry target selection |
CN112491661A (zh) * | 2020-12-11 | 2021-03-12 | 苏州浪潮智能科技有限公司 | 数据中心交换机的时延探测方法、装置、设备及介质 |
Non-Patent Citations (3)
Title |
---|
DAI MIAN, CHENG GUANG; ZHOU YU-YANG: "Survey on Measurement Methods in Software-defined Networking", JOURNAL OF SOFTWARE, vol. 30, no. 6, 27 March 2019 (2019-03-27), pages 1853 - 1874, XP093008457, ISSN: 1000-9825, DOI: 10.13328/j.cnki.jos.005832 * |
NIKHIL HANDIGOL †, BRANDON HELLER †, VIMALKUMAR JEYAKUMAR †, DAVID MAZIÈRES, NICK MCKEOWN { NIKHILH,BRANDO: "I Know What Your Packet Did Last Hop: Using Packet Histories to Troubleshoot Networks", USENIX, USENIX, THE ADVANCED COMPUTING SYSTEMS ASSOCIATION, 2 April 2014 (2014-04-02), Usenix, the Advanced Computing Systems Association , pages 78 - 92, XP061024532 * |
YANG KAICHENG; LI YUANPENG; LIU ZIRUI; YANG TONG; ZHOU YU; HE JINTAO; XUE JING'AN; ZHAO TONG; JIA ZHENGYI; YANG YONGQIANG: "SketchINT: Empowering INT with TowerSketch for Per-flow Per-switch Measurement", 2021 IEEE 29TH INTERNATIONAL CONFERENCE ON NETWORK PROTOCOLS (ICNP), IEEE, 1 November 2021 (2021-11-01), pages 1 - 12, XP034061441, DOI: 10.1109/ICNP52444.2021.9651940 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115988574A (zh) * | 2023-03-15 | 2023-04-18 | 阿里巴巴(中国)有限公司 | 基于流表的数据处理方法、***、设备和存储介质 |
CN115988574B (zh) * | 2023-03-15 | 2023-08-04 | 阿里巴巴(中国)有限公司 | 基于流表的数据处理方法、***、设备和存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN115396345A (zh) | 2022-11-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7039685B2 (ja) | トラフィック測定方法、デバイス、およびシステム | |
WO2022247308A1 (zh) | 流量测量方法、装置及相关设备 | |
CN110971445B (zh) | 一种网络的oam方法及装置 | |
CN111769998B (zh) | 一种网络时延状态的探测方法及装置 | |
CN112511325B (zh) | 网络拥塞控制方法、节点、***及存储介质 | |
CN112039796B (zh) | 数据包传输方法和装置、存储介质和电子设备 | |
US20160218975A1 (en) | Information processing system, information processing apparatus and control method of information processing system | |
WO2020135087A1 (zh) | 一种通信方法、装置及*** | |
CN113225253B (zh) | 一种报文转发方法及装置 | |
CN113328902A (zh) | 网络性能检测方法、装置及网络设备 | |
US20200366626A1 (en) | Forwarding Entry Update Method and Apparatus | |
CN114050994A (zh) | 一种基于SRv6的网络遥测方法 | |
US20210036942A1 (en) | Systems and methods for identifying persistently congested queues | |
CN113542148A (zh) | 一种报文聚合方法、装置、网络网卡及可读存储介质 | |
Martins et al. | Using probabilistic data structures for monitoring of multi-tenant P4-based networks | |
CN113316212B (zh) | 一种基站前传数据流的传输方法及装置 | |
CN112019492A (zh) | 访问控制方法、装置及存储介质 | |
CN115002808A (zh) | 一种信息转发方法、装置、设备及存储介质 | |
CN115277504A (zh) | 一种网络流量监控方法、装置和*** | |
US11477126B2 (en) | Network device and method for processing data about network packets | |
CN114978808A (zh) | 一种数据转发方法、装置、电子设备及存储介质 | |
WO2022227788A1 (zh) | 网络切片资源信息的收集方法、装置及存储介质 | |
CN112511449A (zh) | 一种报文流乱序检测方法、报文处理方法及装置 | |
CN115484202B (zh) | 一种基于int的轻量化路径探测方法 | |
WO2023241520A1 (zh) | 随流检测方法、装置、设备、***及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22810047 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 22810047 Country of ref document: EP Kind code of ref document: A1 |