CN107846327A - A kind of processing method and processing device of network management performance data - Google Patents
A kind of processing method and processing device of network management performance data Download PDFInfo
- Publication number
- CN107846327A CN107846327A CN201711115345.1A CN201711115345A CN107846327A CN 107846327 A CN107846327 A CN 107846327A CN 201711115345 A CN201711115345 A CN 201711115345A CN 107846327 A CN107846327 A CN 107846327A
- Authority
- CN
- China
- Prior art keywords
- dimension
- file
- data
- analysis
- network element
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 17
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000004458 analytical method Methods 0.000 claims abstract description 129
- 238000000034 method Methods 0.000 claims abstract description 33
- 238000007726 management method Methods 0.000 claims description 88
- 238000013144 data compression Methods 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000013507 mapping Methods 0.000 description 4
- 238000005194 fractionation Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000000151 deposition Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000002679 ablation Methods 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000032696 parturition Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/06—Generation of reports
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a kind of processing method and processing device of network management performance data, this method, including:Pre-set at least one analysis dimension;Obtain at least one network management performance data file of at least one network element;According at least one analysis dimension, each network management performance data file is split, generates at least one dimension file, wherein, the network management performance data in each dimension file belongs to the same analysis dimension;Store at least one dimension file.The invention provides a kind of processing method and processing device of network management performance data, it is possible to increase inquiry velocity.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of processing method and processing device of network management performance data.
Background technology
Network management performance data is generally referred to as network element by network management system taken at regular intervals, the management object of network management system.Webmaster
System forms a network management performance number when gathering network management performance data typically in units of the larger network element in overlay area
According to file, these network management performance data files are stored for inquiry.
It is visible by foregoing description, it is one network management performance data text of formation in units of network element in the prior art
Part, these network management performance data files are simply stored according to a dimension, in the network management performance data that inquiry needs,
Need to inquire about each network management performance data file, the network management performance of needs is searched from each network management performance data file
Data, inquiry velocity are slower.
The content of the invention
The embodiments of the invention provide a kind of processing method and processing device of network management performance data, it is possible to increase inquiry velocity.
On the one hand, the embodiments of the invention provide a kind of processing method of network management performance data, including:
Pre-set at least one analysis dimension;
Obtain at least one network management performance data file of at least one network element;
According at least one analysis dimension, each network management performance data file is split, generation is at least
One dimension file, wherein, the network management performance data in each dimension file belongs to the same analysis dimension;
Store at least one dimension file.
Further,
This method further comprises:
In advance according at least one analysis dimension, storage catalogue is set;
Storage at least one dimension file, including:
For each dimension file, according to present analysis dimension corresponding to current dimension file, by the leading dimension
File storage is spent to arrive described in the storage catalogue under store path corresponding to present analysis dimension.
Further,
This method further comprises:Pre-set buffer queue;
At least one dimension file of generation, including:
For each analysis dimension, perform:
A1:To be split out from the network management performance data file data that belong to present analysis dimension be saved in it is described slow
Deposit in queue;
A2:Data are read from the buffer queue, the data of reading are written to corresponding to the present analysis dimension
In current dimension file;
A3:Judge whether the data in the buffer queue are successfully written in the current dimension file, if it is,
The dequeuing data in the current dimension file will be written in the buffer queue, otherwise, returns to A2.
Further,
This method further comprises:
When receiving the querying condition of outside input, perform:
According to the storage catalogue and the querying condition, at least one target storage of the querying condition is determined for compliance with
Path;
A thread is distributed for each target store path;
For each thread, using continuously being read under target store path corresponding to current thread from the current thread
Take the dimension file for meeting the querying condition.
Further,
Storage at least one dimension file, including:
At least one dimension file for belonging to the same analysis dimension is subjected to Coutinuous store.
Further,
This method further comprises:
For memory cell corresponding to each network element setting;
Storage at least one dimension file, including:
For each network element, at least one dimension file for belonging to current device network element is stored to institute
State in the memory cell corresponding to current device network element.
Further,
At least one dimension file of generation, including:
For each analysis dimension, it will be split out from the network management performance data file and belong to present analysis dimension
Data compression be binary data stream, the binary data stream is generated into the dimension corresponding to the present analysis dimension
File.
Further,
At least one analysis dimension includes:The dimension of acquisition time, network element dimension, depend on the facility network
One or more of the dimension of the lower level network element of member, dimension of time granularity of analysis.
Further,
After at least one dimension file of generation, further comprise:
Index file corresponding to each dimension file of generation, wherein, each index file is used to identify correspondingly
The dimension file in data original position.
On the other hand, the embodiments of the invention provide a kind of processing unit of network management performance data, including:
Dimension setup module, for setting at least one analysis dimension;
Acquisition module, for obtaining at least one network management performance data file of at least one network element;
Generation module, for according at least one analysis dimension, being carried out to each network management performance data file
Split, generate at least one dimension file, wherein, the network management performance data in each dimension file belongs to same described
Analyze dimension;
Memory module, for storing at least one dimension file.
Further,
The device further comprises:
Catalogue setup module, for according at least one analysis dimension, setting storage catalogue;
The memory module, for for each dimension file, according to present analysis corresponding to current dimension file
Dimension, by under the current dimension file storage to store path corresponding to present analysis dimension described in the storage catalogue.
Further,
The device further comprises:Buffer queue;
The generation module, for for each analysis dimension, performing:
A1:To be split out from the network management performance data file data that belong to present analysis dimension be saved in it is described slow
Deposit in queue;
A2:Data are read from the buffer queue, the data of reading are written to corresponding to the present analysis dimension
In current dimension file;
A3:Judge whether the data in the buffer queue are successfully written in the current dimension file, if it is,
The dequeuing data in the current dimension file will be written in the buffer queue, otherwise, returns to A2.
Further,
The device further comprises:
Enquiry module, for when receiving the querying condition of outside input, performing:
According to the storage catalogue and the querying condition, at least one target storage of the querying condition is determined for compliance with
Path;
A thread is distributed for each target store path;
For each thread, using continuously being read under target store path corresponding to current thread from the current thread
Take the dimension file for meeting the querying condition.
Further,
The memory module, at least one dimension file for belonging to the same analysis dimension to be connected
Renew storage.
Further,
The device further comprises:
Setup module is stored, for memory cell corresponding to being set for each network element;
The memory module, for for each network element, at least one institute of current device network element will to be belonged to
The storage of dimension file is stated into the memory cell corresponding to the current device network element.
Further,
The generation module, for that for each analysis dimension, will be split from the network management performance data file
The data compression for going out to belong to present analysis dimension is binary data stream, and the binary data stream is generated into the present analysis
The dimension file corresponding to dimension;
Further,
At least one analysis dimension includes:The dimension of acquisition time, network element dimension, depend on the facility network
One or more of the dimension of the lower level network element of member, dimension of time granularity of analysis.
Further,
The generation module, for after at least one dimension file of generation, generating each dimension file
Corresponding index file, wherein, each index file is used for the starting for identifying the data in the corresponding dimension file
Position.
In embodiments of the present invention, network management performance data file is split, at least one dimension text is generated after fractionation
Part so that the network management performance data in each dimension file belongs to same analysis dimension, when user needs to inquire about some analysis
During the network management performance data of dimension, because network management performance data is stored according to analysis dimension, can more quickly it look into
The data of needs are ask, improve inquiry velocity.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the required accompanying drawing used in technology description to be briefly described, it should be apparent that, drawings in the following description are the present invention
Some embodiments, for those of ordinary skill in the art, on the premise of not paying creative work, can also basis
These accompanying drawings obtain other accompanying drawings.
Fig. 1 is a kind of flow chart of the processing method for network management performance data that one embodiment of the invention provides;
Fig. 2 is a kind of schematic diagram for storage catalogue that one embodiment of the invention provides;
Fig. 3 is the flow chart of the processing method for another network management performance data that one embodiment of the invention provides;
Fig. 4 is a kind of schematic diagram of the processing unit for network management performance data that one embodiment of the invention provides;
Fig. 5 is the schematic diagram of the processing unit for another network management performance data that one embodiment of the invention provides.
Embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention
In accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is
Part of the embodiment of the present invention, rather than whole embodiments, based on the embodiment in the present invention, those of ordinary skill in the art
The every other embodiment obtained on the premise of creative work is not made, belongs to the scope of protection of the invention.
As shown in figure 1, the embodiments of the invention provide a kind of processing method of network management performance data, this method can include
Following steps:
Step 101:Pre-set at least one analysis dimension;
Step 102:Obtain at least one network management performance data file of at least one network element;
Step 103:According at least one analysis dimension, each network management performance data file is split,
At least one dimension file is generated, wherein, the network management performance data in each dimension file belongs to the same analysis
Dimension;
Step 104:Store at least one dimension file.
In embodiments of the present invention, network management performance data file is split, at least one dimension text is generated after fractionation
Part so that the network management performance data in each dimension file belongs to same analysis dimension, when user needs to inquire about some analysis
During the network management performance data of dimension, because network management performance data is stored according to analysis dimension, can more quickly it look into
The data of needs are ask, improve inquiry velocity.
In an embodiment of the present invention, this method further comprises:
In advance according at least one analysis dimension, storage catalogue is set;
Storage at least one dimension file, including:
For each dimension file, according to present analysis dimension corresponding to current dimension file, by the leading dimension
File storage is spent to arrive described in the storage catalogue under store path corresponding to present analysis dimension.
In embodiments of the present invention, storage catalogue is provided with for each dimension file, when needing to inquire about some network management performances
During data, the store path of the network management performance data of needs can be quickly found by storage catalogue, improves inquiry velocity.
In embodiments of the present invention, each dimension file can be embodied by store path corresponding to each dimension file
Analysis dimension.
In an embodiment of the present invention, this method further comprises:Pre-set buffer queue;
At least one dimension file of generation, including:
For each analysis dimension, perform:
A1:To be split out from the network management performance data file data that belong to present analysis dimension be saved in it is described slow
Deposit in queue;
A2:Data are read from the buffer queue, the data of reading are written to corresponding to the present analysis dimension
In current dimension file;
A3:Judge whether the data in the buffer queue are successfully written in the current dimension file, if it is,
The dequeuing data in the current dimension file will be written in the buffer queue, otherwise, returns to A2.
In embodiments of the present invention, it would be desirable to which the data being written in dimension file are saved in buffer queue, work as caching
When data in queue are successfully written in dimension file, just by the dequeuing data in buffer queue, when the number in buffer queue
During according to not being successfully written in dimension file, again to dimension file write data, it can so ensure not to be successfully written
It will not be lost to the data in dimension file, ensure that the reliability of data.
During to dimension file write data, buffer queue is used for the temporary data to be write, certain reaching
After the threshold value of size (such as:1GB), then by data it is continuously written into dimension file, ensures that continuously raising gulps down in storage
The amount of telling, reduce write-in number, reduce IOPS (Input/Output Operations Per Second, it is per second to be written and read behaviour
The number of work).Technically mainly in combination with two kinds of File Technologies of sparse file and Memory Mapping File.
Sparse file technology:Sparse file technology is operating system Floor layer Technology, it is allowed to the cavity of no data be present in file,
Purpose is to reduce the actual disk size taken of file, in buffer queue, writes data in batches in units of 128KB every time,
As the write-in inadequate 128KB of data actual size, the disk size actually taken in sparse file can be less than 128KB.
Storage mapping file technique:By on one piece of region of memory Map Disk fixed size (such as 1GB) it is slow
File is deposited, realizes the file operation based on internal storage access, the method binding cache file, which rolls, to be used, and once rolls fixed size
File, that is, correspond to a fixed size Memory Mapping File.
In an embodiment of the present invention, this method further comprises:
When receiving the querying condition of outside input, perform:
According to the storage catalogue and the querying condition, at least one target storage of the querying condition is determined for compliance with
Path;
A thread is distributed for each target store path;
For each thread, using continuously being read under target store path corresponding to current thread from the current thread
Take the dimension file for meeting the querying condition.
In embodiments of the present invention, meeting the target store path of querying condition may have multiple, be each target at this moment
Store path distributes a thread, realizes multiple threads, improves treatment effeciency, real for each target store path
Existing single thread batch processing, improve the handling capacity of disk.
In an embodiment of the present invention, storage at least one dimension file, including:
At least one dimension file for belonging to the same analysis dimension is subjected to Coutinuous store.
In embodiments of the present invention, the dimension file of same analysis dimension is subjected to Coutinuous store, these are same reading and writing
During the data of one analysis dimension, it is possible to achieve it is continuous to read, substantially increase the handling capacity of disk.
In an embodiment of the present invention, this method further comprises:
For memory cell corresponding to each network element setting;
Storage at least one dimension file, including:
For each network element, at least one dimension file for belonging to current device network element is stored to institute
State in the memory cell corresponding to current device network element.
In embodiments of the present invention, disk is divided into each memory cell, is only stored in each memory cell and belong to same
The data storage of one network element can more easily inquire about the webmaster of each equipment webpage in same memory cell
Can data.Each memory cell can be named according to the ID of corresponding network element.
In an embodiment of the present invention, at least one dimension file of generation, including:
For each analysis dimension, it will be split out from the network management performance data file and belong to present analysis dimension
Data compression be binary data stream, the binary data stream is generated into the dimension corresponding to the present analysis dimension
File.
In embodiments of the present invention, can be compressed by GZIP algorithms.Network management performance in dimension file is obtained
, it is necessary to first be decompressed to the binary data stream in dimension file during data.
In an embodiment of the present invention, at least one analysis dimension includes:Dimension, the network element dimension of acquisition time
Spend, depend on one or more of dimension, dimension of time granularity of analysis of the lower level network element of the network element.
In embodiments of the present invention, for the dimension of acquisition time, mesh can be created in units of week in storage catalogue
Record, belong under the data flow of the analysis dimension this catalogue.The catalogue can be named with date of first day weekly, form can
To be YYYYMMDD, such as:20171021.On acquisition time by Zhou Youxu store, can single file it is excessive, avoid because
Single file is excessive and reduces recall precision.
In embodiments of the present invention, for network element dimension, creatied directory in storage catalogue with network element, can be with
Named by the hashed value of the numbering of network element come catalogue, such as:By the numbering step-by-step and 255, Neng Goufang of network element
Only file is excessive under same catalogue, influences recall precision.
In embodiments of the present invention, for the dimension for the lower level network element for depending on network element, network element is depended on
Lower level network element can include:CPU, port, internal memory etc..Can be that each lower level network element establishes catalogue in storage catalogue.
In embodiments of the present invention, for analysis time granularity dimension, data are entered according to the time cycle of analysis
Row classification, such as:According to it is per minute, time granularity is stored per hour etc., for subsequent query.
In embodiments of the present invention, dimension according to the above analysis, in same all scopes, same network element, same depend on
The data of the time granularity of lower level network element, same analysis in network element, which are flowed under same catalogue, to be stored, according to looking into
, only need to be according to querying condition stitching path when inquiry condition is inquired about.
Corresponding to corresponding each network element for memory cell, identical all, identical network elements, identical equipment is depended on
Performance data under the lower level network element of network element, the time granularity of same analysis, same network element ID is deposited under storage catalogue
Same network element ID name memory cell under.
When newly-increased analysis dimension, storage catalogue is also required to be changed accordingly.
As shown in Fig. 2 the schematic diagram based on a kind of storage catalogue that dimension is analyzed in the embodiment of the present invention.As shown in Figure 2
Storage catalogue shown in, the one-level of the storage catalogue is catalogue corresponding to the dimension of acquisition time (being YYYYMMDD in figure), is adopted
The next stage for collecting catalogue corresponding to the dimension of time is catalogue corresponding to network element dimension, catalogue corresponding to network element dimension
Next stage be the dimension for the lower level network element for depending on network element corresponding to catalogue (specifically, depend under network element
Level network element includes CPU, port, internal memory etc.), the subordinate of catalogue corresponding to the dimension for the lower level network element for depending on network element is point
Catalogue corresponding to the dimension of the time granularity of analysis, wherein, M5 refers to that the time granularity of analysis is 5 minutes, and H refers to the time of analysis
Granularity is 1 hour.Each dimension file is stored with the dimension of the time granularity of analysis.
In an embodiment of the present invention, after at least one dimension file of generation, further comprise:
Index file corresponding to each dimension file of generation, wherein, each index file is used to identify correspondingly
The dimension file in data original position.
In embodiments of the present invention, the original position of the data in dimension file can be found by index file, is being looked into
During inquiry, the network management performance data of needs can be found according to the original position in index file.
In embodiments of the present invention, dimension file can be " .dat " file, and index file can be " .idx " file.Dimension
Physical length of 8 bytes of beginning of degree file for the data after storing this write-in, every time by file actual (tube) length after write-in
Degree is written to preceding four bytes, and backups to the 5-8 byte, for the positioning before write-in next time to file writing position.Under
The one before byte is read before secondary write-in, every 4 bytes are compared, take larger byte number as last time write-in length according to
According to enhancing robustness.
In each memory cell, dimension file and index file are generally comprised.
When data are written to memory cell, be first written to dat files, then safeguard this write-in after Data Position, after will
Position in dat files is written to idx index files, ensures dirty data, data will not occur because abnormal in dat ablation process
Write successfully, by the dequeuing data in buffer queue, whole process, which has, exception occurs at one, data at least can be based on file
Exist in buffer queue, ensure that data are not lost.
To the data being repeatedly written, when reading, the index occurred using last time can ensure to read data as most
The valid data being once stored in afterwards, ensure the reliability of digital independent.
In embodiments of the present invention, inquiry has two levels, and whole implementation method is all based on file, is used in deployment
(SuSE) Linux OS, Linux can be existed in internal memory always for last time using file, just be discharged until memory usage is higher,
One layer of hot spot data caching naturally has been done for this method, query caching is done to idx the and dat files frequently used.Another layer
Secondary, inquiry data, which are to ensure that, occurs without the problem of write-in data are unreadable, first the data in the buffer queue of file is entered
Row retrieval, after the data in memory cell are retrieved, ensure data it is readable all the time.
In embodiments of the present invention, buffer queue can use the mentality of designing of FIFO, can continuously be read when reading
More datas are taken, ensure handling capacity during write-in.
When the actual use configuration of cache file threshold size (such as:When 1G), legacy data file is sealed up for safekeeping, raw
The .cache file new into one provides data write-in.
In embodiments of the present invention, optimized for mass data storage and inquiry, add and be based in logic is write
The buffer queue of sparse file and Memory Mapping File, data buffer storage is realized, data cached run up to after a certain size again will be slow
Batch data write-in dimension file in depositing, realizes file Coutinuous store, improves write-in and inquiry handling capacity;Based on data cached
Copy distribution realize at one write, many places storage, realize that data calamity is standby.
In embodiments of the present invention, by optimizing for file design Storage, compression storing data mode and read-write, for
In network management performance data time, space, logical relation multidimensional analysis, scene is equally analyzed, the lower read-write of same hardware configuration gulps down
The amount of telling is dominant compared with relevant database and Key-Value database performances.
As shown in figure 3, the embodiments of the invention provide a kind of processing method of network management performance data, this method can include
Following steps:
Step 301:At least one analysis dimension is pre-set, in advance according at least one analysis dimension, storage is set
Catalogue, pre-set buffer queue.
Specifically, the storage catalogue that storage catalogue can be as shown in Figure 2.
Step 302:Obtain at least one network management performance data file of at least one network element.
Specifically, network management performance data file is obtained in units of network element.Such as:Obtaining the network management performance of this week
During data file, the network management performance data of each network element forms a file.
Here network element is the larger network element in overlay area, such as:Base station controller.
Step 303:According at least one analysis dimension, each network management performance data file is split.
Because network management performance data file is obtained in units of network element, wherein, may include multiple points
The data of dimension are analysed, by the step, the data of each analysis dimension can be splitted out.
Step 304:For each analysis dimension, it will be split out from network management performance data file and belong to present analysis dimension
Data be saved in buffer queue, data are read from buffer queue, the data of reading are written to present analysis dimension pair
In the current dimension file answered, when the data in buffer queue are successfully written in current dimension file, by buffer queue
The dequeuing data being written in current dimension file, wherein, the network management performance data in each dimension file belongs to same point
Analyse dimension.
Specifically, network management performance data is stored respectively according to analysis dimension, can more facilitates inquiry.
Step 305:For each dimension file, according to present analysis dimension corresponding to current dimension file, by leading dimension
File storage is spent into storage catalogue under store path corresponding to present analysis dimension, wherein, belong to same analysis dimension
At least one dimension file carries out Coutinuous store.
Specifically, each dimension file is subjected to Coutinuous store according to storage catalogue, can be according to storage mesh in inquiry
Record is quickly found out the network management performance data of needs.
As shown in Figure 4, Figure 5, the embodiments of the invention provide a kind of processing unit of network management performance data.Device embodiment
It can be realized, can also be realized by way of hardware or software and hardware combining by software.For hardware view, such as Fig. 4
It is shown, it is a kind of hardware structure diagram of equipment where a kind of processing unit of network management performance data provided in an embodiment of the present invention,
In addition to the processor shown in Fig. 4, internal memory, network interface and nonvolatile memory, setting where device in embodiment
It is standby generally to include other hardware, such as it is responsible for the forwarding chip of processing message.Exemplified by implemented in software, such as Fig. 5 institutes
Show, be by corresponding calculating in nonvolatile memory by the CPU of equipment where it as the device on a logical meaning
Machine programmed instruction reads what operation in internal memory was formed.A kind of processing unit for network management performance data that the present embodiment provides, bag
Include:
Dimension setup module 501, for setting at least one analysis dimension;
Acquisition module 502, for obtaining at least one network management performance data file of at least one network element;
Generation module 503, for according at least one analysis dimension, entering to each network management performance data file
Row is split, and generates at least one dimension file, wherein, the network management performance data in each dimension file belongs to same institute
State analysis dimension;
Memory module 504, for storing at least one dimension file.
In an embodiment of the present invention, the device further comprises:
Catalogue setup module, for according at least one analysis dimension, setting storage catalogue;
The memory module, for for each dimension file, according to present analysis corresponding to current dimension file
Dimension, by under the current dimension file storage to store path corresponding to present analysis dimension described in the storage catalogue.
In an embodiment of the present invention, the device further comprises:Buffer queue;
The generation module, for for each analysis dimension, performing:
A1:To be split out from the network management performance data file data that belong to present analysis dimension be saved in it is described slow
Deposit in queue;
A2:Data are read from the buffer queue, the data of reading are written to corresponding to the present analysis dimension
In current dimension file;
A3:Judge whether the data in the buffer queue are successfully written in the current dimension file, if it is,
The dequeuing data in the current dimension file will be written in the buffer queue, otherwise, returns to A2.
In an embodiment of the present invention, the device further comprises:
Enquiry module, for when receiving the querying condition of outside input, performing:
According to the storage catalogue and the querying condition, at least one target storage of the querying condition is determined for compliance with
Path;
A thread is distributed for each target store path;
For each thread, using continuously being read under target store path corresponding to current thread from the current thread
Take the dimension file for meeting the querying condition.
In an embodiment of the present invention, the memory module, at least the one of the same analysis dimension will to be belonged to
The individual dimension file carries out Coutinuous store.
In an embodiment of the present invention, the device further comprises:
Setup module is stored, for memory cell corresponding to being set for each network element;
The memory module, for for each network element, at least one institute of current device network element will to be belonged to
The storage of dimension file is stated into the memory cell corresponding to the current device network element.
In an embodiment of the present invention, the generation module, will be from the webmaster for for each analysis dimension
Split out that to belong to the data compression of present analysis dimension be binary data stream in performance data files, by the binary data
Stream generates the dimension file corresponding to the present analysis dimension.
In an embodiment of the present invention, at least one analysis dimension includes:Dimension, the network element dimension of acquisition time
Spend, depend on one or more of dimension, dimension of time granularity of analysis of the lower level network element of the network element.
In an embodiment of the present invention, the generation module, for after at least one dimension file of generation, giving birth to
Into index file corresponding to each dimension file, wherein, each index file is used to identify the corresponding dimension
The original position of data in file.
The contents such as the information exchange between each unit, implementation procedure in said apparatus, due to implementing with the inventive method
Example is based on same design, and particular content can be found in the narration in the inventive method embodiment, and here is omitted.
The embodiments of the invention provide a kind of computer-readable recording medium, including execute instruction, when the computing device of storage control
During the execute instruction, the storage control performs the processing of any one network management performance data provided in an embodiment of the present invention
Method.
The embodiments of the invention provide a kind of storage control, including:Processor, memory and bus;
The memory is used to store execute instruction, and the processor is connected with the memory by the bus, when
During the storage control operation, the execute instruction of memory storage described in the computing device, so that the storage control
Device performs the processing method of any one network management performance data provided in an embodiment of the present invention.
The each embodiment of the present invention at least has the advantages that:
1st, in embodiments of the present invention, network management performance data file is split, at least one dimension is generated after fractionation
File so that the network management performance data in each dimension file belongs to same analysis dimension, when user needs to inquire about some point
, can be more quick because network management performance data is stored according to analysis dimension when analysing the network management performance data of dimension
The data of needs are inquired, improve inquiry velocity.
2nd, in embodiments of the present invention, storage catalogue is provided with for each dimension file, when needing to inquire about some webmasters
During energy data, the store path of the network management performance data of needs can be quickly found by storage catalogue, improves inquiry speed
Degree.
3rd, in embodiments of the present invention, it would be desirable to which the data being written in dimension file are saved in buffer queue, when slow
When depositing the data in queue and being successfully written in dimension file, just by the dequeuing data in buffer queue, when in buffer queue
When data are not successfully written in dimension file, again to dimension file write data, it can so ensure not write successfully
Entering to the data in dimension file to lose, and ensure that the reliability of data.
4th, in embodiments of the present invention, meeting the target store path of querying condition may have multiple, be each mesh at this moment
Mark store path and distribute a thread, realize multiple threads, improve treatment effeciency, for each target store path,
Single thread batch processing is realized, improves the handling capacity of disk.
5th, in embodiments of the present invention, the dimension file of same analysis dimension is subjected to Coutinuous store, reads and writes these
During the data of same analysis dimension, it is possible to achieve it is continuous to read, substantially increase the handling capacity of disk.
It should be noted that herein, such as first and second etc relational terms are used merely to an entity
Or operation makes a distinction with another entity or operation, and not necessarily require or imply and exist between these entities or operation
Any this actual relation or order.Moreover, term " comprising ", "comprising" or its any other variant be intended to it is non-
It is exclusive to include, so that process, method, article or equipment including a series of elements not only include those key elements,
But also the other element including being not expressly set out, or also include solid by this process, method, article or equipment
Some key elements.In the absence of more restrictions, the key element limited by sentence " including one ", is not arranged
Except other identical factor in the process including the key element, method, article or equipment being also present.
One of ordinary skill in the art will appreciate that:Realizing all or part of step of above method embodiment can pass through
Programmed instruction related hardware is completed, and foregoing program can be stored in computer-readable storage medium, the program
Upon execution, the step of execution includes above method embodiment;And foregoing storage medium includes:ROM, RAM, magnetic disc or light
Disk etc. is various can be with the medium of store program codes.
It is last it should be noted that:Presently preferred embodiments of the present invention is the foregoing is only, is merely to illustrate the skill of the present invention
Art scheme, is not intended to limit the scope of the present invention.Any modification for being made within the spirit and principles of the invention,
Equivalent substitution, improvement etc., are all contained in protection scope of the present invention.
Claims (10)
- A kind of 1. processing method of network management performance data, it is characterised in thatAt least one analysis dimension is pre-set, including:Obtain at least one network management performance data file of at least one network element;According at least one analysis dimension, each network management performance data file is split, generation is at least one Dimension file, wherein, the network management performance data in each dimension file belongs to the same analysis dimension;Store at least one dimension file.
- 2. according to the method for claim 1, it is characterised in thatFurther comprise:In advance according at least one analysis dimension, storage catalogue is set;Storage at least one dimension file, including:For each dimension file, according to present analysis dimension corresponding to current dimension file, by the current dimension text Part storage is arrived described in the storage catalogue under store path corresponding to present analysis dimension.
- 3. according to the method for claim 1, it is characterised in thatFurther comprise:Pre-set buffer queue;At least one dimension file of generation, including:For each analysis dimension, perform:A1:The data that belong to present analysis dimension will be split out from the network management performance data file and are saved in the caching team In row;A2:Data are read from the buffer queue, the data of reading are written to current corresponding to the present analysis dimension In dimension file;A3:Judge whether the data in the buffer queue are successfully written in the current dimension file, if it is, by institute The dequeuing data being written in buffer queue in the current dimension file is stated, otherwise, returns to A2.
- 4. according to the method for claim 2, it is characterised in thatFurther comprise:When receiving the querying condition of outside input, perform:According to the storage catalogue and the querying condition, at least one target for being determined for compliance with the querying condition stores road Footpath;A thread is distributed for each target store path;For each thread, symbol is read using continuous under target store path corresponding to current thread from the current thread Close the dimension file of the querying condition.
- 5. according to any described method in claim 1-4, it is characterised in thatStorage at least one dimension file, including:At least one dimension file for belonging to the same analysis dimension is subjected to Coutinuous store;And/orFurther comprise:For memory cell corresponding to each network element setting;Storage at least one dimension file, including:For each network element, at least one dimension file storage for belonging to current device network element is worked as to described In the memory cell corresponding to preceding network element;And/orAt least one dimension file of generation, including:For each analysis dimension, the number that belongs to present analysis dimension will be split out from the network management performance data file According to boil down to binary data stream, the binary data stream is generated into the dimension text corresponding to the present analysis dimension Part;And/orAt least one analysis dimension includes:The dimension of acquisition time, network element dimension, depend on the network element One or more of the dimension of lower level network element, dimension of time granularity of analysis;And/orAfter at least one dimension file of generation, further comprise:Index file corresponding to each dimension file of generation, wherein, each index file is used to identify corresponding institute State the original position of the data in dimension file.
- A kind of 6. processing unit of network management performance data, it is characterised in that including:Dimension setup module, for setting at least one analysis dimension;Acquisition module, for obtaining at least one network management performance data file of at least one network element;Generation module, for according at least one analysis dimension, being split to each network management performance data file, At least one dimension file is generated, wherein, the network management performance data in each dimension file belongs to the same analysis Dimension;Memory module, for storing at least one dimension file.
- 7. device according to claim 6, it is characterised in thatFurther comprise:Catalogue setup module, for according at least one analysis dimension, setting storage catalogue;The memory module, for for each dimension file, according to present analysis dimension corresponding to current dimension file, By under the current dimension file storage to store path corresponding to present analysis dimension described in the storage catalogue.
- 8. device according to claim 6, it is characterised in thatFurther comprise:Buffer queue;The generation module, for for each analysis dimension, performing:A1:The data that belong to present analysis dimension will be split out from the network management performance data file and are saved in the caching team In row;A2:Data are read from the buffer queue, the data of reading are written to current corresponding to the present analysis dimension In dimension file;A3:Judge whether the data in the buffer queue are successfully written in the current dimension file, if it is, by institute The dequeuing data being written in buffer queue in the current dimension file is stated, otherwise, returns to A2.
- 9. device according to claim 7, it is characterised in thatFurther comprise:Enquiry module, for when receiving the querying condition of outside input, performing:According to the storage catalogue and the querying condition, at least one target for being determined for compliance with the querying condition stores road Footpath;A thread is distributed for each target store path;For each thread, symbol is read using continuous under target store path corresponding to current thread from the current thread Close the dimension file of the querying condition.
- 10. according to any described device in claim 6-9, it is characterised in thatThe memory module, at least one dimension file for belonging to the same analysis dimension continuously to be deposited Storage;And/orFurther comprise:Setup module is stored, for memory cell corresponding to being set for each network element;The memory module, for for each network element, at least one dimension of current device network element will to be belonged to File storage is spent into the memory cell corresponding to the current device network element;And/orThe generation module, for for each analysis dimension, category will to be split out from the network management performance data file It is binary data stream in the data compression of present analysis dimension, the binary data stream is generated into the present analysis dimension The corresponding dimension file;And/orAt least one analysis dimension includes:The dimension of acquisition time, network element dimension, depend on the network element One or more of the dimension of lower level network element, dimension of time granularity of analysis;And/orThe generation module, for after at least one dimension file of generation, each dimension file of generation to be corresponding Index file, wherein, each index file be used to identifying corresponding to data in the dimension file original position.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201711115345.1A CN107846327A (en) | 2017-11-13 | 2017-11-13 | A kind of processing method and processing device of network management performance data |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201711115345.1A CN107846327A (en) | 2017-11-13 | 2017-11-13 | A kind of processing method and processing device of network management performance data |
Publications (1)
Publication Number | Publication Date |
---|---|
CN107846327A true CN107846327A (en) | 2018-03-27 |
Family
ID=61681054
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201711115345.1A Pending CN107846327A (en) | 2017-11-13 | 2017-11-13 | A kind of processing method and processing device of network management performance data |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN107846327A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2020129612A1 (en) * | 2018-12-19 | 2020-06-25 | ソニー株式会社 | Information processing apparatus, information processing method, and information processing program |
CN112069270A (en) * | 2020-09-07 | 2020-12-11 | 青岛海信传媒网络技术有限公司 | File-based multi-dimensional data reading and writing method and display device |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102750377A (en) * | 2012-06-21 | 2012-10-24 | 深圳市谷米科技有限公司 | Massive data storage and retrieval method |
-
2017
- 2017-11-13 CN CN201711115345.1A patent/CN107846327A/en active Pending
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102750377A (en) * | 2012-06-21 | 2012-10-24 | 深圳市谷米科技有限公司 | Massive data storage and retrieval method |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2020129612A1 (en) * | 2018-12-19 | 2020-06-25 | ソニー株式会社 | Information processing apparatus, information processing method, and information processing program |
JP7435470B2 (en) | 2018-12-19 | 2024-02-21 | ソニーグループ株式会社 | Information processing device, information processing method, and information processing program |
CN112069270A (en) * | 2020-09-07 | 2020-12-11 | 青岛海信传媒网络技术有限公司 | File-based multi-dimensional data reading and writing method and display device |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11068455B2 (en) | Mapper tree with super leaf nodes | |
TWI719281B (en) | A system, machine readable medium, and machine-implemented method for stream selection | |
TWI702503B (en) | Systems, methods, and computer readable media to implement merge tree modifications for maintenance operations | |
TWI702506B (en) | System, machine readable medium, and machine-implemenated method for merge tree garbage metrics | |
US10642515B2 (en) | Data storage method, electronic device, and computer non-volatile storage medium | |
CN103635900B (en) | Time-based data partitioning | |
US9298726B1 (en) | Techniques for using a bloom filter in a duplication operation | |
US9223660B2 (en) | Storage device to backup content based on a deduplication system | |
KR20170054299A (en) | Reference block aggregating into a reference set for deduplication in memory management | |
CN103955530B (en) | Data reconstruction and optimization method of on-line repeating data deletion system | |
CN108009008A (en) | Data processing method and system, electronic equipment | |
CN103562914B (en) | The type that economizes on resources extends file system | |
US11232073B2 (en) | Method and apparatus for file compaction in key-value store system | |
US11157453B2 (en) | Parallel deduplication using automatic chunk sizing | |
CN107045531A (en) | A kind of system and method for optimization HDFS small documents access | |
CN108319654A (en) | Computing system, cold and hot data separation method and device, computer readable storage medium | |
CN107766374B (en) | Optimization method and system for storage and reading of massive small files | |
CN105117351A (en) | Method and apparatus for writing data into cache | |
CN113867627B (en) | Storage system performance optimization method and system | |
CN112148217B (en) | Method, device and medium for caching deduplication metadata of full flash memory system | |
CN105493080B (en) | The method and apparatus of data de-duplication based on context-aware | |
CN107665219A (en) | A kind of blog management method and device | |
CN113094374A (en) | Distributed storage and retrieval method and device and computer equipment | |
CN109407985B (en) | Data management method and related device | |
CN107846327A (en) | A kind of processing method and processing device of network management performance data |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180327 |
|
RJ01 | Rejection of invention patent application after publication |