CN103235820B - Date storage method and device in a kind of group system - Google Patents
Date storage method and device in a kind of group system Download PDFInfo
- Publication number
- CN103235820B CN103235820B CN201310153509.5A CN201310153509A CN103235820B CN 103235820 B CN103235820 B CN 103235820B CN 201310153509 A CN201310153509 A CN 201310153509A CN 103235820 B CN103235820 B CN 103235820B
- Authority
- CN
- China
- Prior art keywords
- data
- server
- initial
- mapping relation
- initial mapping
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000003860 storage Methods 0.000 title claims abstract description 39
- 238000013507 mapping Methods 0.000 claims abstract description 171
- 108010028984 3-isopropylmalate dehydratase Proteins 0.000 claims abstract description 44
- 238000013500 data storage Methods 0.000 claims description 16
- 238000007726 management method Methods 0.000 claims description 10
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 239000000203 mixture Substances 0.000 claims description 4
- 238000000151 deposition Methods 0.000 claims description 3
- 238000013506 data mapping Methods 0.000 claims description 2
- 238000012423 maintenance Methods 0.000 abstract description 8
- 101100328886 Caenorhabditis elegans col-2 gene Proteins 0.000 description 4
- 101100328884 Caenorhabditis elegans sqt-3 gene Proteins 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011017 operating method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses the storage method and apparatus of data in a kind of group system, described method includes: resolve the server service data gathered；If the data parsed are the data gathered based on IPMI protocol, determine the initial mapping relation table of correspondence according to the server identification ID parsed；Perform the step of converting of the data gathered based on IPMI protocol, to remove Heterogeneous data；Set up the target mapping relations table between numerical value and the table field name of all collected sensors；According to described target mapping relations table, generate SQL statement and the server service data of described collection is stored in data base.In the present invention, eliminated the isomerism of data by the form arranging different server initial mapping relation table with set up server target mapping relations table, by Servers-all service data centralized stores, it is simple to the unified management of data and the maintenance of cluster.
Description
Technical field
The present invention relates to trunking communicating technology field, number in a kind of group system
According to storage method and apparatus.
Background technology
The most properly functioning in order to obtain server cluster in real time, need server cluster is monitored,
It is monitored server cluster mainly including that according to communication protocol acquisition server service data (be envelope
The data of dress, it is simple to transmission), the server service data that storage gathers, according to the server fortune of storage
Row data are analyzed cluster and are run the most normal.Wherein said service data can include status number further
According to and performance data, described status data refers to the data that some numerical value is fixing, such as CPU model；
Described performance data can include some dynamic changes such as CPU usage, network traffics use information
Data.
When acquisition server service data, if the server model in cluster is different, assist based on SNMP
The data that view gathers have identical dimension, can unify storage management.But, based on IPMI protocol
Main by sensor acquisition data in server, the data dimension of collection is the most different, such as services
In device, the title of sensor is different, and kind of sensor is different, these isomerisms that can cause gathering data,
Data store more difficult.What Internet firm used at present is all to be provided by server production firm
IPMI manage system, if composition cluster server come from multiple manufacturer, then, to clothes
Just have many set IPMI during business device cluster monitoring and manage system, i.e. different model server service data guarantor
Exist in different data bases.When attendant to detect collection by the service data of Servers-all
Group running status time, need between different IPMI management systems, carry out transferring and comparing of data
Relatively, when cluster scale is bigger, the server service data amount of storage is bigger so that the maintenance of cluster
Difficulty is relatively big, inefficient.
Summary of the invention
The embodiment of the present application provides date storage method and device in a kind of group system, existing to solve
In the cluster being made up of different model server in technology, server service data exists isomerism cannot
Unified storage, the management scattered technical problem of maintenance that the cluster caused is overall, for having isomerism
Data provide a kind of unified data storage method, the server service data of all models are stored in
In same the table of data base, improve the concentration degree of server service data and the maintenance effect that cluster is overall
Rate.
For solving above-mentioned technical problem, the present invention provides date storage method and dress in a kind of group system
Putting, the present invention provides following technical scheme:
Date storage method in a kind of group system, described method includes:
The server service data gathered is resolved；
If the data parsed are the data gathered based on intelligent platform management IPMI protocol, according to parsing
The server identification ID gone out determines the initial mapping relation table that described server is corresponding, described initial mapping
Relation table have recorded the mapping pass in described server between at least one sensor name and table field name
System, uses identical table field name in the initial mapping table that different server is corresponding；
Perform based on IPMI protocol gather data step of converting to remove Heterogeneous data, described based on
The data step of converting of IPMI protocol collection includes: judge the name of all collected sensor parsed
Claim whether have record at described initial mapping relation table, the most then build according to initial mapping relation table
Target mapping relations table between numerical value and the table field name of vertical collected sensor；If it is not, described
Initial mapping relation table increases new table field name, and sets up described new table field name with Unrecorded
Mapping relations between collected sensor name；Perform described to set up quilt according to initial mapping relation table
The step of the target mapping relations table between numerical value and the table field name of collection sensor；
According to described target mapping relations table, generate SQL statement by described based on IPMI protocol collection
Data be stored in data base.
Preferably, the server identification ID that described basis parses determines corresponding initial of described server
Mapping relations table includes:
In the way of database retrieval, determine that server model identifies according to described server identification ID；
The initial mapping relation table that described server is corresponding is determined according to described server model mark.
Preferably, described initial mapping relation table is set up in the way of hash index table, then according to described
The model of server determines that the initial mapping table that described server is corresponding includes: according to server model pair
The mark answered, by hash mode search described in the initial mapping relation table that prestores.
Preferably, the described server service data to gathering resolves, including:
The queue being made up of readable documents descriptor is obtained according to Epoll mode；
The filec descriptor of queue heads instruction is obtained from readable documents descriptor queue；
The filec descriptor indicated according to described queue heads obtains server and runs initial data, including:
The byte of predetermined number is read, to determine that the filec descriptor that described queue heads indicates is retouched from Socket
The length of the server service data stated；According to the data length determined, retouch from described filec descriptor
The file stated reads the data of this length value；According to preset data form from the data of described length
Read the initial data of server.
Preferably, if the data parsed are the data gathered based on simple network management snmp protocol,
Perform the storing step of the data gathered based on snmp protocol；
The storing step of described data based on snmp protocol collection includes: set up the data that parse and
Mapping relations table between the parameter name that the described data parsed are corresponding；According to described mapping relations
Status data in table, updates described status data value in data base by SQL statement；By institute
The performance data stated in mapping relations table is stored in data base by SQL statement.
Preferably, described data base is relevant database.
The present invention also provides for data storage device in a kind of group system, it is characterised in that described device
Including:
Data resolution module, for resolving the server service data gathered；
Initial mapping table acquisition module, the data for parsing are the data gathered based on IPMI protocol
Time, determine, according to the server identification ID parsed, the initial mapping relation table that described server is corresponding,
Described initial mapping relation table have recorded at least one sensor name and table field name in described server
Between mapping relations, the initial mapping table that different server is corresponding uses identical table field name；
First performs module, for performing the conversion of data based on IPMI protocol collection to remove data
Isomerism, described first perform module includes: judge submodule, be used for judging all collected biographies
Whether the title of sensor has record at described initial mapping relation table；Generate submodule, for described
When judging that the result of submodule is as being, set up the numerical value of collected sensor according to initial mapping relation table
And target mapping relations table between table field name；Table field name increases module newly, for described judgement submodule
When the result of block is no, described initial mapping relation table increases new table field name, and sets up institute
State the mapping relations between new table field name and Unrecorded collected sensor name；Triggers module,
For triggering described generation submodule.
Data memory module, for according to described target mapping relations table, generates SQL statement by described
Data based on IPMI protocol collection are stored in data base.
Preferably, described initial mapping table acquisition module includes:
Server model mark determines submodule, is used for according to the server identification ID parsed with data
The mode of library searching determines that server model identifies；
Initial mapping relation table determines submodule, for determining described according to described server model mark
The initial mapping relation table that server is corresponding.
Preferably, described initial mapping relation table is set up in the way of hash index table, then initial mapping
Relation table determines submodule, specifically for the mark corresponding according to server model, passes through hash mode
The initial mapping relation table prestored described in lookup.
Preferably, described data resolution module includes:
Descriptor queue obtains submodule, for obtaining by readable documents descriptor according to Epoll mode
The queue of composition；
Descriptor reading submodule, for obtaining queue heads instruction from readable documents descriptor queue
Filec descriptor；
Initial data reading submodule, obtains clothes for the filec descriptor indicated according to described queue heads
Business device runs initial data and specifically includes: read the byte of predetermined number from Socket, to determine
State the length of the server service data of the filec descriptor description of queue heads instruction；According to the number determined
According to length, from the file that described filec descriptor describes, read the data of this length value；According to presetting
Data form reads the initial data of server from the data of described length.
Preferably, described device also includes:
Second performs module, when the data for parsing are data based on snmp protocol collection, holds
The storage of the data that row gathers based on snmp protocol；
Described second performs module includes: submodule set up by mapping relations table, for the data parsed
When being the data gathered based on snmp protocol, set up the data that parse and parameter corresponding to described data
Mapping relations table between title；Status data sub module stored, for according to described mapping relations table
In status data, update described status data value in data base by SQL statement；Performance number
According to sub module stored, for by the performance data in described mapping relations table, it is stored in by SQL statement
Data base.
In technical scheme, in cluster, each server has sensor name and the table of correspondence
Initial mapping relation table between field name, then according to the biography of described initial relation mapping table Yu collection
The numerical value of sensor sets up target mapping relations table.In described target mapping relations table, record is literary name section
Relation between name and the numerical value of collected sensor, maps pass by initial mapping relation table with target
It is that the comparison of table understands, the target mapping relations table now obtained only has numerical value corresponding to sensor
And the mapping relations between table field name, have not had sensor name.Meanwhile, right in the present invention
The literary name section name used in the initial mapping table that each server is corresponding is identical, and so far, target maps
In relation table, the data of record have not had isomerism.If regarding table field name as parameter name,
Corresponding sensor values is exactly parameter value, and this is a kind of simple one-to-one relationship, when storage,
As long as setting up a table with above-mentioned table field name, then leave corresponding collection value in correspondence
Literary name section is under one's name.Only for different servers, the physical significance that parameter value represents
May be different.Target mapping relations table has eliminated by the sensor name number that bring different with number
According to isomerism, now, it is possible to use unified SQL statement that the server of described collection is run number
According to being stored in data base.
Can be by based on IPMI protocol for different model server data by technical scheme
Unification is stored in a table, for attendant, needs to monitor the running status that cluster is overall
Time, it is only necessary in same table, carry out the reading of data and compare, even if cluster scale is relatively big,
The speed that in same table, lookup data compare also is faster, reduces the maintenance difficulties of cluster,
Improve the maintenance efficiency of cluster.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present application or technical scheme of the prior art, below by right
In embodiment or description of the prior art, the required accompanying drawing used is briefly described, it should be apparent that,
Accompanying drawing in describing below is only some embodiments described in the application, skill common for this area
From the point of view of art personnel, on the premise of not paying creative work, it is also possible to obtain it according to these accompanying drawings
His accompanying drawing.
Fig. 1 is the flow chart of date storage method embodiment 1 in a kind of group system of the present invention；
Fig. 2 is the flow chart of date storage method embodiment 2 in a kind of group system of the present invention
Fig. 3 is the flow chart of date storage method embodiment 3 in a kind of group system of the present invention；
Fig. 4 is the structural representation of data storage device embodiment 1 in a kind of group system of the present invention；
Fig. 5 is the first structural representation performing module 403 in apparatus of the present invention embodiment 1；
Fig. 6 is the structural representation of initial mapping table acquisition module 402 in apparatus of the present invention embodiment 1；
Fig. 7 is the structural representation of data storage device embodiment 2 in a kind of group system of the present invention；
Fig. 8 is the structural representation of data resolution module 401 in apparatus of the present invention embodiment 2；
Fig. 9 is the embodiment of the present invention a kind of structure chart under actual application scenarios.
Detailed description of the invention
In order to make those skilled in the art be more fully understood that the present invention program, below in conjunction with the present invention
Accompanying drawing in embodiment, is clearly and completely described the technical scheme in the embodiment of the present invention,
Obviously, described embodiment is only a part of embodiment of the present invention rather than whole embodiments.
Based on the embodiment in the present invention, those of ordinary skill in the art are not making creative work premise
Lower obtained every other embodiment, broadly falls into the scope of protection of the invention.
All embodiments of the invention all include saying premised on different model server by group system
Bright.With reference to shown in Fig. 1, for date storage method embodiment in a kind of group system that the present invention provides
The flow chart of 1, the present embodiment specifically may include that
Step 101: resolve the server service data gathered.
Under actual application scenarios, server service data is to collect according to certain acquisition protocols,
After having gathered server service data, needing to be packaged the data collected, after encapsulation, data are
Can transmit in the channel.So first having to carry out the parsing of data after receiving server service data,
To obtain the server raw operational data gathered.Parsing about data is referred in prior art
The implementation of data parsing, repeats no more here.
Step 102: if the data parsed are the data gathered based on IPMI protocol, according to parsing
Server identification ID determine the initial mapping relation table that described server is corresponding.
If the data parsed are based on IPMI protocol (Intelligent Platform
Management Interface, IPMI) data that gather, then corresponding not
There is isomerism, in order to remove data in the data collected based on IPMI protocol of same model server
Isomerism, the first step is exactly to obtain the initial mapping relation table that this server is corresponding.Concrete, according to
The data gathered based on IPMI protocol parsed, can obtain this server by data parsing
ID(identifies), by the way of database retrieval, then obtain the server corresponding with this server ID
Model (or model ID), finally, the server model according to determining is found by hash mode
The initial mapping relation table of corresponding described model server.
Preferably, described initial mapping relation table is set up in hash index table mode, for each service
Device can prestore an initial mapping relation table, in practical operation, can be the clothes of each model
Business device prestores a corresponding initial mapping relation table, and corresponding initially the reflecting of different model server
Penetrate and relation table uses identical table field name.Or, having resolved data after every time, set up and resolve
Mapping relations table between the sensor name and the table field name that go out is as initial mapping relation table.With in advance
Deposit initial mapping relation table corresponding to the server of each model, two of which that cluster includes not
The server of same model for example, for the ease of describe described model is designated as respectively the first model and
Second model, the first model server set up by hash index table mode shown in reference table 1 is corresponding
Initial mapping relation table A1；Table 2 is that the second model server set up by hash index table mode is corresponding
Initial mapping relation table A2.
Table 1 initial mapping relation table A1
| Table field name | Sensor name | |
| Col1 | Name1 | Example |
| Col2 | Name2 | Example |
Table 2 initial mapping relation table A2
| Table field name | Sensor name | |
| Col1 | Name3 | Example |
| Col2 | Name4 | Example |
Sensor name in Tables 1 and 2 described here be with Name1, Name2, Name3 and
Name4 mark.In sensor name in Tables 1 and 2 not necessarily corresponding model server
Including the title of all the sensors, and be probably in this concrete model server in all the sensors
The title of several sensors, this is that the time of making the product by the corresponding multiple servers of a concrete model is different
Cause.Such as, for the server of the first model, it is assumed that the number of it sensor included
It is 3, specifically temperature sensor, voltage sensor and current sensor.But, actually used
In there may exist the situations such as server updating.Such as, increase on the original basis new sensor or
Person deletes certain sensor, and now, the title for remaining original sensor keeps constant, same
The number of the sensor that the model time of making the product different server includes may be different.And for one
The corresponding initial mapping table of the server of model, thus the sensor name in this initial mapping table with
The title of the real sensor of each collection of server of this model may be incomplete same.
Understanding with reference to Tables 1 and 2, for the server of first model the second model, it is corresponding
Initial mapping table in the literary name section name that uses identical, sensor that the most same table field name is corresponding
Title is different.
Step 103: perform based on IPMI protocol gather data step of converting to remove Heterogeneous data,
Generate target mapping relations table.
Because data based on IPMI protocol collection exist isomerism, for the ease of the unified storage of data,
Before carrying out the storage of data, first have to data are carried out the conversion of data structure.Concrete data
The conversion process of structure is as follows:
After parsing server raw operational data, set up the clothes of actual acquisition according to the data parsed
In business device, the mapping relations table between sensor name with corresponding collection value (is breathed out with data in the present embodiment
The form of uncommon table is given).In order to make it easy to understand, with certain collected service of corresponding first model of table 3
The data Hash table B1 of device, table 4 are the data Hash of certain collected server of corresponding second model
Table B2 is that example carries out schematically illustrating of this step.
Table 3 data Hash table B1
| Sensor name | Collection value | |
| Name1 | 10 | Example |
| Name2 | 20 | Example |
Table 4 data Hash table B2
| Sensor name | Collection value | |
| Name3 | 30 | Example |
| Name4 | 40 | Example |
| Name5 | 50 |
After obtaining the data Hash table that server is corresponding, it is judged that the sensor name set that table 1 includes
Whether comprise the sensor name set that table 3 includes, all the sensors that table 3 includes in other words
Title the most all exists, if it is, referred to as table 3 is completely covered by table 1.If table
3 are completely covered by table 1, directly carry out merging of table 1 and table 3, set up table field name and collection value
Between target mapping table, for the server of described first model, shown in reference table 5, for
The target mapping relations table that table 1 obtains after merging with table 3, certain service that the i.e. first model is collected
The target mapping relations table of device.
The target mapping relations table of table 5 first model server
| Table field name | Collection value | |
| Col1 | 10 | Example |
| Col2 | 20 | Example |
For the described collected server of the first model, its initial mapping table and data Hash
Table is completely covered, and merges fairly simple.From table 2 with table 4 it can be seen that for collected
For certain server of second model, the initial mapping table of its correspondence and data Hash table are not complete
Cover.In this case, to set up for the sensor name not comprised by initial mapping relation table
And the mapping relations between newly-increased table field name.In practical operation, because server updating exists not
Predictability, it is possible to set up a table field name set, do not have when encountering initial mapping relation table
During the sensor name comprised, from described table field name set, obtain the table field name being not used by come
Set up the mapping relations between the sensor name not comprised.Such as, Col3 correspondence Name is used
The collection value of 5, then, shown in reference table 6, collected for the correspondence and the second model that obtain
The target mapping relations table of certain server.
The target mapping relations table of table 6 second model
| Table field name | Collection value | |
| Col1 | 30 | Example |
| Col2 | 40 | Example |
| Col3 | 50 | Example |
Described step 103 is illustrate as a example by a server of collection, other are being taken
When business device data process, principle is identical.
Step 104: generate SQL statement according to described target mapping relations table and assist described based on IPMI
The data that view gathers are stored in data base.
After generating target mapping relations table, by generating SQL statement by described based on IPMI protocol collection
Data be stored in data base.Wherein, the SQL statement that in target mapping relations table, data are corresponding includes
Insert inserts statement and update updates statement, concrete, by state in target mapping relations table
Type data are saved in data base by updating statement, and performance data is saved in data base by inserting statement.
Insert Sentence format:
insert into table_name(x1,x2,x3,x4,...)values(y1,y2,y3,y4,...)
Table field name and collection value in the most corresponding target mapping relations table of wherein xn, yn are right.
Update Sentence format:
update table_name set x1=y1,x2=y2,x3=y3,...where...
Table field name and collection value in the most corresponding target mapping relations table of wherein xn, yn are right, data
Traveling through target mapping relations table during warehouse-in, time complexity is O (N).
In the technical scheme of the present embodiment, in cluster each server have correspondence sensor name with
Initial mapping relation table between table field name, then according to described initial relation mapping table and collection
The numerical value of sensor sets up target mapping relations table.In described target mapping relations table, record is literary name
Relation between the numerical value of the collected sensor of Duan Mingyu, is mapped with target by initial mapping relation table
The comparison of relation table understands, and only has the number that sensor is corresponding in the target mapping relations table now obtained
Mapping relations between value and table field name, have not had sensor name.Meanwhile, in the present invention
Literary name section name for using in the initial mapping table that each server is corresponding is identical, and so far, target is reflected
Penetrate the data of record in relation table and there is no isomerism.If regarding table field name as parameter name
Words, corresponding sensor values is exactly parameter value, and this is a kind of simple one-to-one relationship, is depositing
Chu Shi, as long as setting up a table with above-mentioned table field name, then leaves in corresponding collection value
Corresponding literary name section is under one's name.Only for different servers, the thing that parameter value represents
Reason meaning may be different.Target mapping relations table has eliminated by sensor name and number different band
The Heterogeneous data come, now, it is possible to use unified SQL statement by the server of described collection
Service data is stored in data base.
Different model server can be gathered based on IPMI protocol by technical scheme
Data are unified to be stored in a table, for attendant, needs to monitor the operation that cluster is overall
During state, it is only necessary to carry out the reading of data in same table and compare, even if cluster scale is relatively big,
The speed that lookup data compare in same table is also faster, reduces the Maintenance Difficulty of cluster
Degree, improves the maintenance efficiency of cluster.
Data storage based on IPMI protocol collection is described, to data by described embodiment 1
Existing data parsing technology is used during parsing, but when available data resolves, by real-time reception
Service data is placed directly in queue caching, when needing to resolve data, directly reads from data queue
Data.Because data directly cache in queue, more operating-system resources can be taken, and team
Row finite capacity, is not suitable for the storage of high concurrent data.In order to solve depositing of high concurrency data
Storage, the present invention provides date storage method embodiment 2 in a kind of group system, uses a kind of new number
According to analysis mode, provide storage mode for high concurrency data.It is to gather based on IPMI protocol simultaneously
Data with based on snmp protocol (Simple Network Management Protocol, simple network
Management agreement) service data that gathers provides a kind of efficient storage mode.
With reference to shown in Fig. 2, for date storage method embodiment in a kind of group system that the present invention provides
The flow chart of 2, on the basis of embodiment 1, the data gathered with snmp protocol in the present embodiment
It is described in detail as a example by storage, although under the background of the data of snmp protocol collection, introduce new number
According to analytical form, but it is only used to analysis mode is illustrated, is not to say that this analysis mode
It is only applicable to the data that snmp protocol gathers.Implementation for step same as the previously described embodiments
With reference to the step in above-described embodiment, the present embodiment repeats no more.The present embodiment specifically may include that
Step 201: prestore every kind of initial mapping relation corresponding to model server in data base
Table.
Initial mapping relation table described here is primarily directed to data based on IPMI protocol collection and sets
Put.
Step 202: the server service data gathered is resolved.
Using Epoll interface data in the present embodiment, Epoll interface is that under Linux, multichannel is multiple
With the enhancing version of I/O interface select/poll, it can significantly improve program in a large amount of concurrently connections
System CPU utilization rate in the case of enlivening only on a small quantity, because it can multiplexing filec descriptor set
All must again prepare to be listened before transmitting result and developer need not being forced every time to wait event
Filec descriptor set；The when that another reason obtaining event exactly, it need not travel through whole quilt
The descriptor set intercepted, as long as traveling through those to be added the description of enqueue by kernel asynchronous the waking up up of I/O event
Symbol set is just.Described queue is a kind of " first in first out " (FIFO first in first out)
Linear list.
Receive the mode of data based on Epoll, the data parsing detailed process carried out is: first pass through
The mode of Epoll, obtains the filec descriptor that current time is readable from system, and can by obtain
Read filec descriptor and put in readable documents descriptor queue, read the readable documents in queue heads and describe
Symbol.And it is original to read server from corresponding file according to the readable documents descriptor in queue heads
Service data (the server service data before unencapsulated).Server is being obtained according to filec descriptor
During raw operational data, first reading from Socket and preset a byte, described presetting in a byte is deposited
Storage is the length of encapsulation of data in the file that represents of the readable documents descriptor in described queue heads；Root
Length value accordingly, reads the data of this length, then presses in file corresponding from this document descriptor
Form according to communicating pair agreement understands server raw operational data from the data of described length.Often
Secondary when carrying out data parsing, the readable documents descriptor all reading queue heads corresponding carries out data parsing.
If the data parsed are data based on IPMI protocol, the data described in reference example 1 are deposited
Storage mode stores, if the data parsed are data based on snmp protocol, enters step
203。
Step 203: determine the data gathered based on snmp protocol and parameter name corresponding to described data
Between mapping relations table.
If the data parsed be based on snmp protocol gather data, it is necessary first to determine based on
Mapping relations table between data and parameter name corresponding to described data that snmp protocol gathers.Map
Relation table just can enter the data of step 204 and store after determining.
Step 204: status data is stored relevant database.
For the status data collected, its multi collect to numerical value be probably identical, so not having
It is necessary the most individually to store the status data every time gathered, as long as in each more new database
Status data.In practical operation, can will pass through based on the status data of snmp protocol collection
SQL statement updates these state type data values in data base.
Step 205: performance data is stored described relevant database.
Preferably, described status data can be filtered from described mapping relations table, by remaining performance
Data are stored in data base by SQL statement, i.e. complete the storage gathering data based on snmp protocol.
The SQL statement form used during data loading based on IPMI protocol collection storage is applicable to SNMP association
The storage of the data that view gathers.In practical operation, status data can be stored separately with performance data
In different tables, it is simultaneously based on the data that IPMI protocol gathers and the number gathered based on snmp protocol
According to being stored separately in different tables.
As shown in the above, data storage ratio based on snmp protocol collection is based on IPMI protocol
The data storage gathered is simply the most, because for the server of different model based on snmp protocol collection
Data dimension be identical, the number of parameters i.e. gathered and parameter name are identical, as long as now
There are the mapping relations between parameter name with table field name just can obtain in the way of merging to use Hash table
Target Hash table between collection value and table field name, and the server for different model is corresponding
Mapping relations between parameter name and table field name are identical.
In the technical scheme of the present embodiment, employ a kind of data parsing mode unlike the prior art,
Server raw operational data is read, because operating file descriptor is not take up behaviour by filec descriptor
Make system resource, but application program itself, reduce the impact on operating system.It addition, queue
Middle placement is filec descriptor, if there being high concurrency data, as long as now being retouched by corresponding for data
State symbol to be buffered in queue, too much resource will not be taken, effectively solve high concurrent data
Storage problem.
When in prior art being monitored cluster, the data collected generally use RRD(Round
Robin Database, ring-type data base) store, so-called RRD indicates a little as a periphery
Annulus--these point be exactly according to the time store position.An annulus stores data, does not has
There is beginning and end.After Chang Shijian, data new in RRD can cover holding of legacy data, i.e. data
Property is poor for a long time.Accordingly, the present embodiment uses relevant database store data, can keep
The persistency of data.
Although for being to pass through based on IPMI protocol collection and data storage based on snmp protocol collection
Different embodiments are described separately, in actual applications, are simultaneous, because for clothes
For business device, sometimes, data should be gathered based on snmp protocol, simultaneously also will be based on IPMI
Agreement gathers data, now the flow process of embodiment 1 with embodiment 2 is integrated, with reference to shown in Fig. 3,
For the complete flowchart of data storage in cluster, wherein the realization of correlation step is referred to implement
The implementation of correlation step in example 1 and embodiment 2.
Corresponding to embodiment of the method, accordingly, data storage during the present invention also provides for a kind of group system
Device, with reference to shown in Fig. 4, implements for data storage device in a kind of group system that the present invention provides
The structural representation of example 1, described device includes:
Data resolution module 401, for resolving the server service data gathered；
Initial mapping table acquisition module 402, gathers based on IPMI protocol for the data parsed
During data, determine, according to the server identification ID parsed, the initial mapping relation that described server is corresponding
Table, described initial mapping relation table have recorded at least one sensor name and literary name in described server
Mapping relations between section name, use identical literary name section in the initial mapping table that different server is corresponding
Name；
First performs module 403, for performing the conversion of the data gathered based on IPMI protocol, with life
Become target mapping relations table.
Data memory module 404, for according to described target mapping relations table, generating SQL statement will
Described data based on IPMI protocol collection are stored in data base.
Wherein, it is preferred that with reference to shown in Fig. 5, for the described first structural representation performing module 403
Figure, described first performs module 403 includes:
Judge submodule 501, for judging that the title of all collected sensors is in described initial mapping
Whether relation table has record；
Generate submodule 502, for the described result judging submodule as being time, according to initial mapping
Relation table sets up target mapping relations table between the numerical value of collected sensor and table field name；
Table field name increases submodule 503 newly, for increasing new table in described initial mapping relation table
Field name, and set up the mapping between described new table field name and Unrecorded collected sensor name
Relation；
Triggers module, is used for triggering described generation submodule.
Preferably, with reference to shown in Fig. 6, for the structural representation of described initial mapping table acquisition module 402
Figure, described initial mapping relation table acquisition module 402 includes:
Server model mark determines submodule 601, is used for according to the server identification parsed with number
Determine that server model identifies according to the mode of library searching；
Initial mapping relation table determines submodule 602, for determining according to described server model mark
The initial mapping relation table that described server is corresponding.
Described initial mapping relation table is set up in hash index table mode, then initial mapping relation table determines
Submodule 603, specifically for the mark corresponding according to server model, searches institute by hash mode
State the initial mapping relation table prestored.
The function that in device, each module is realized is corresponding with method operating procedure in embodiment of the method 1, this
In repeat no more.
The structural representation of data storage device embodiment 2 in a kind of group system that the present invention provides,
With reference to shown in Fig. 7, for data storage device embodiment 2 in a kind of group system that the present invention provides
Structural representation, in addition to the module shown in Fig. 4, described device can also include:
Second performs module 701, when the data for parsing are data based on snmp protocol collection,
Perform the storage of the data gathered based on snmp protocol；
Described second performs module 701 includes: submodule 7011 set up by mapping relations table, is used for resolving
When the data gone out are data based on snmp protocol collection, set up the data and described data pair parsed
Mapping relations table between the parameter name answered；
Status data sub module stored 7012, is used for according to the status data in described mapping relations table,
Described status data value in data base is updated by SQL statement；
Performance data sub module stored 7013, for by the performance data in described mapping relations table, logical
Cross SQL statement and be stored in data base.
Preferably, with reference to shown in Fig. 8, for described data resolution module 401 knot in the present embodiment
Structure schematic diagram, described data resolution module 401 includes:
Descriptor queue obtains submodule 801, retouches by readable documents for obtaining according to Epoll mode
State the queue of symbol composition；
Descriptor reading submodule 802, refers to for obtaining queue heads from readable documents descriptor queue
The filec descriptor shown；
Initial data reading submodule 803, obtains for the filec descriptor indicated according to described queue heads
Take server operation initial data to specifically include: from Socket, read the byte of predetermined number, with really
The length of the server service data that the filec descriptor of fixed described queue heads instruction describes；According to determining
Data length, from the file that described filec descriptor describes, read the data of this length value；According to
Preset data form reads the initial data of server from the data of described length.
The function that in device, each module is realized is corresponding with method operating procedure in embodiment of the method 2, this
In repeat no more.
With reference to shown in Fig. 9, for the embodiment of the present invention a kind of realization under actual application scenarios, main line
Available filec descriptor is put in readable documents descriptor queue (shown in A1) by Epoll by journey.
Containing a number of thread in worker thread pond, it is responsible for taking out team from readable documents descriptor queue
Filec descriptor (shown in A2) in row head, therefrom reads data, resolves data and storage data (A3
Shown in), complete the unified storage of IPMI data.In the design of data base, according to gathering data age
The difference of property is stored separately.It is the last look of one form each field of storage of each table design simultaneously,
The last look of each item data can be got rapidly when accessing data base.
It should be noted that term " includes ", " comprising " or its any other variant are intended to non-
Comprising of exclusiveness, so that include the process of a series of key element, method, article or equipment not
Only include those key elements, but also include other key elements being not expressly set out, or also include for
The key element that this process, method, article or equipment are intrinsic.In the case of there is no more restriction,
The key element limited by statement " including ... ", it is not excluded that including the process of described key element, side
Method, article or equipment there is also other identical element.
For system embodiment, owing to it corresponds essentially to embodiment of the method, so relevant part
The part seeing embodiment of the method illustrates.System embodiment described above is only schematically
, the wherein said unit illustrated as separating component can be or may not be physically separate
, the parts shown as unit can be or may not be physical location, i.e. may be located at one
Individual place, or can also be distributed on multiple NE.Can be selected it according to the actual needs
In some or all of module realize the purpose of the present embodiment scheme.Those of ordinary skill in the art
In the case of not paying creative work, i.e. it is appreciated that and implements.
The above is only the detailed description of the invention of the present invention, it is noted that common for the art
For technical staff, under the premise without departing from the principles of the invention, it is also possible to make some improvements and modifications,
These improvements and modifications also should be regarded as protection scope of the present invention.
Claims (11)
1. date storage method in a group system, it is characterised in that described method includes:
The server service data gathered is resolved；
If the data parsed are the data gathered based on intelligent platform management IPMI protocol, according to parsing
The server identification ID gone out determines the initial mapping relation table that described server is corresponding, described initial mapping
Relation table have recorded the mapping pass in described server between at least one sensor name and table field name
System, uses identical table field name in the initial mapping table that different server is corresponding；
Perform based on IPMI protocol gather data step of converting to remove Heterogeneous data, described based on
The data step of converting of IPMI protocol collection includes: judge the name of all collected sensor parsed
Claim whether have record at described initial mapping relation table, the most then build according to initial mapping relation table
Target mapping relations table between numerical value and the table field name of vertical collected sensor；If it is not, described
Initial mapping relation table increases new table field name, and sets up described new table field name and non-record
Collected sensor name between mapping relations；Perform described according to the foundation of initial mapping relation table
The step of the target mapping relations table between numerical value and the table field name of collected sensor；
According to described target mapping relations table, unified SQL statement is used to assist described based on IPMI
The data that view gathers are stored in data base.
Method the most according to claim 1, it is characterised in that the service that described basis parses
Device mark ID determines that the initial mapping relation table that described server is corresponding includes:
In the way of database retrieval, determine that server model identifies according to described server identification ID；
The initial mapping relation table that described server is corresponding is determined according to described server model mark.
Method the most according to claim 2, it is characterised in that described initial mapping relation table with
The mode of hash index table is set up, then determine that described server is corresponding according to the model of described server
Initial mapping table includes: according to the mark that server model is corresponding, is searched by hash mode and deposits in advance
The described initial mapping relation table of storage.
Method the most according to claim 1, it is characterised in that the described server fortune to gathering
Row data resolve, including:
The queue being made up of readable documents descriptor is obtained according to Epoll mode；
The filec descriptor of queue heads instruction is obtained from readable documents descriptor queue；
The filec descriptor indicated according to described queue heads obtains server and runs initial data, including:
The byte of predetermined number is read, to determine that the filec descriptor that described queue heads indicates is retouched from Socket
The length of the server service data stated；According to the data length determined, retouch from described filec descriptor
The file stated reads the data of this length value；According to preset data form from the data of described length
Read the initial data of server.
Method the most according to claim 1, it is characterised in that if the data parsed be based on
The data that simple network management snmp protocol gathers, perform depositing of data based on snmp protocol collection
Storage step；
The storing step of described data based on snmp protocol collection includes: set up the data that parse and
Mapping relations table between the parameter name that the described data parsed are corresponding；According to described mapping relations
Status data in table, updates described status data value in data base by SQL statement；By institute
The performance data stated in mapping relations table is stored in data base by SQL statement.
6. according to the method described in any one of claim 1-5, it is characterised in that described data base is
Relevant database.
7. data storage device in a group system, it is characterised in that described device includes:
Data resolution module, for resolving the server service data gathered；
Initial mapping table acquisition module, the data for parsing are the data gathered based on IPMI protocol
Time, determine, according to the server identification ID parsed, the initial mapping relation table that described server is corresponding,
Described initial mapping relation table have recorded at least one sensor name and table field name in described server
Between mapping relations, the initial mapping table that different server is corresponding uses identical table field name；
First performs module, for performing the conversion of data based on IPMI protocol collection to remove data
Isomerism, described first perform module includes: judge submodule, be used for judging all collected biographies
Whether the title of sensor has record at described initial mapping relation table；Generate submodule, for described
When judging that the result of submodule is as being, set up the numerical value of collected sensor according to initial mapping relation table
And target mapping relations table between table field name；Table field name increases module newly, for described judgement submodule
When the result of block is no, described initial mapping relation table increases new table field name, and sets up institute
State the mapping relations between new table field name and Unrecorded collected sensor name；Trigger submodule
Block, is used for triggering described generation submodule；
Data memory module, for according to described target mapping relations table, uses unified SQL statement
Described data based on IPMI protocol collection are stored in data base.
Device the most according to claim 7, it is characterised in that described initial mapping table obtains mould
Block includes:
Server model mark determines submodule, is used for according to the server identification ID parsed with data
The mode of library searching determines that server model identifies；
Initial mapping relation table determines submodule, for determining described according to described server model mark
The initial mapping relation table that server is corresponding.
Device the most according to claim 8, it is characterised in that described initial mapping relation table with
The mode of hash index table is set up, then initial mapping relation table determines submodule, specifically for according to clothes
The mark that business type number is corresponding, searches the described initial mapping relation table prestored by hash mode.
Device the most according to claim 7, it is characterised in that described data resolution module bag
Include:
Descriptor queue obtains submodule, for obtaining by readable documents descriptor according to Epoll mode
The queue of composition；
Descriptor reading submodule, for obtaining queue heads instruction from readable documents descriptor queue
Filec descriptor；
Initial data reading submodule, obtains clothes for the filec descriptor indicated according to described queue heads
Business device runs initial data and specifically includes: read the byte of predetermined number from Socket, to determine
State the length of the server service data of the filec descriptor description of queue heads instruction；According to the number determined
According to length, from the file that described filec descriptor describes, read the data of this length value；According to presetting
Data form reads the initial data of server from the data of described length.
11. devices according to claim 7, it is characterised in that described device also includes:
Second performs module, when the data for parsing are data based on snmp protocol collection, holds
The storage of the data that row gathers based on snmp protocol；
Described second performs module includes: submodule set up by mapping relations table, for the data parsed
When being the data gathered based on snmp protocol, set up the data that parse and parameter corresponding to described data
Mapping relations table between title；Status data sub module stored, for according to described mapping relations table
In status data, update described status data value in data base by SQL statement；Performance number
According to sub module stored, for by the performance data in described mapping relations table, it is stored in by SQL statement
Data base.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310153509.5A CN103235820B (en) | 2013-04-27 | 2013-04-27 | Date storage method and device in a kind of group system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310153509.5A CN103235820B (en) | 2013-04-27 | 2013-04-27 | Date storage method and device in a kind of group system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103235820A CN103235820A (en) | 2013-08-07 |
| CN103235820B true CN103235820B (en) | 2016-10-05 |
Family
ID=48883861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310153509.5A Expired - Fee Related CN103235820B (en) | 2013-04-27 | 2013-04-27 | Date storage method and device in a kind of group system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103235820B (en) |
Families Citing this family (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106330990B (en) * | 2015-06-17 | 2020-02-11 | 阿里巴巴集团控股有限公司 | Performance monitoring and analyzing system and method of B/S structure |
| CN106909557B (en) * | 2015-12-23 | 2020-06-16 | 中国电信股份有限公司 | Memory cluster storage method and device and memory cluster reading method and device |
| CN106302554A (en) * | 2016-11-08 | 2017-01-04 | 郑州云海信息技术有限公司 | A kind of socket communication means, device and storage device |
| CN106529195A (en) * | 2016-12-02 | 2017-03-22 | 郑州云海信息技术有限公司 | Data acquisition and analysis method and device |
| CN107169075A (en) * | 2017-05-10 | 2017-09-15 | 深圳大普微电子科技有限公司 | Data access method, storage device and the storage system of feature based analysis |
| CN107463618B (en) * | 2017-07-04 | 2020-06-30 | 北京奇艺世纪科技有限公司 | Index creating method and device |
| CN107229838A (en) * | 2017-07-17 | 2017-10-03 | 成都嘉逸科技有限公司 | A kind of collecting method and data acquisition unit for medical monitoring equipment |
| CN108228426A (en) * | 2018-01-19 | 2018-06-29 | 郑州云海信息技术有限公司 | A kind of method and apparatus for recording SDR information |
| CN108595121B (en) * | 2018-04-17 | 2021-03-02 | 湖北文理学院 | Data storage method and device |
| CN111274051B (en) * | 2020-01-19 | 2023-08-22 | 中信银行股份有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111538775A (en) * | 2020-04-28 | 2020-08-14 | 中车株洲电力机车有限公司 | Train running state data analysis method and system and rail transit vehicle |
| CN111738909B (en) * | 2020-06-11 | 2023-09-26 | 杭州海康威视数字技术股份有限公司 | Image generation method and device |
| CN113992658B (en) * | 2021-10-28 | 2023-09-26 | 科大讯飞股份有限公司 | Data transmission method, related device, equipment, system and storage medium |
| CN117066743B (en) * | 2023-06-01 | 2024-04-16 | 广州富士汽车整线集成有限公司 | Automobile processing production line system comprising multi-vehicle type processing stations |
| CN117312435A (en) * | 2023-11-23 | 2023-12-29 | 首都信息发展股份有限公司 | Data acquisition method and device and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101212355A (en) * | 2006-12-29 | 2008-07-02 | 佛山市顺德区顺达电脑厂有限公司 | Server monitoring method |
| CN101916260A (en) * | 2010-07-21 | 2010-12-15 | 中国测绘科学研究院 | Method for establishing semantic mapping between disaster body and relational database |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7051363B2 (en) * | 2001-09-20 | 2006-05-23 | Intel Corporation | System and method for interfacing to different implementations of the intelligent platform management interface |
-
2013
- 2013-04-27 CN CN201310153509.5A patent/CN103235820B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101212355A (en) * | 2006-12-29 | 2008-07-02 | 佛山市顺德区顺达电脑厂有限公司 | Server monitoring method |
| CN101916260A (en) * | 2010-07-21 | 2010-12-15 | 中国测绘科学研究院 | Method for establishing semantic mapping between disaster body and relational database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103235820A (en) | 2013-08-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103235820B (en) | Date storage method and device in a kind of group system | |
| US11720537B2 (en) | Bucket merging for a data intake and query system using size thresholds | |
| US11327992B1 (en) | Authenticating a user to access a data intake and query system | |
| US11620288B2 (en) | Dynamically assigning a search head to process a query | |
| US10257057B2 (en) | System and a process for searching massive amounts of time-series | |
| US11657057B2 (en) | Revising catalog metadata based on parsing queries | |
| US11157497B1 (en) | Dynamically assigning a search head and search nodes for a query | |
| CN102236581B (en) | Mapping reduction method and system thereof for data center | |
| US11416465B1 (en) | Processing data associated with different tenant identifiers | |
| CN108667725A (en) | A kind of industrial AnyRouter and implementation method based on a variety of accesses and edge calculations | |
| CN102164186B (en) | Method and system for realizing cloud search service | |
| CN103902537A (en) | Multi-service log data storage processing and inquiring system and method thereof | |
| US11573955B1 (en) | Data-determinant query terms | |
| CN109299157A (en) | A kind of data export method and device of distributed big single table | |
| CN105227367A (en) | A kind of warning analysis processing method of low delay | |
| CN103886508A (en) | Mass farmland data monitoring method and system | |
| CN108241725B (en) | A kind of data hot statistics system and method | |
| CN105577411A (en) | Cloud service monitoring method and device based on service origin | |
| CN104993957A (en) | Method for providing cloud log service for distributed APP employing Log4j | |
| WO2017000592A1 (en) | Data processing method, apparatus and system | |
| US11892996B1 (en) | Identifying an indexing node to process data using a resource catalog | |
| CN107341249A (en) | The storage of server info and extracting method and system, extraction element | |
| CN111625595A (en) | Block chain management method, system and storage medium based on self-starting crawler technology | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN106354612A (en) | Service-oriented packaging method and system of cloud diagnosis resource |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: 100086 12, 1201, 3 building, 2 South Road, Haidian District Academy of Sciences, Beijing. Patentee after: BEIJING SOHU NEW MEDIA INFORMATION TECHNOLOGY Co.,Ltd. Address before: 100084 Beijing Haidian District Zhongguancun East Road 1 hospital 9 building Sohu cyber building 15 Floor Patentee before: BEIJING SOHU NEW MEDIA INFORMATION TECHNOLOGY Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20161005 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |