CN101170433A - Database storage method - Google Patents
Database storage method Download PDFInfo
- Publication number
- CN101170433A CN101170433A CNA2006101409868A CN200610140986A CN101170433A CN 101170433 A CN101170433 A CN 101170433A CN A2006101409868 A CNA2006101409868 A CN A2006101409868A CN 200610140986 A CN200610140986 A CN 200610140986A CN 101170433 A CN101170433 A CN 101170433A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- core buffer
- warehouse
- earlier
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database entering method. Current database entering method is aimed at processing large-quantity, scattered data with low efficiency and high resource consumption as well as poor overall network management. To solve the problem, the invention includes the following steps: (1) opening a memory buffering area idle memory to store temporary data to be stored in the database; (2) firstly storing multiple batches of database-entering data received from time to time in the memory buffering area and then sending to the database in a centralized manner. The invention improves the performance of network management and significantly increases the database-entering efficiency. The rate of resource occupancy has been significantly reduced. The invention is applicable to various database systems.
Description
Technical field
The invention provides a kind of database storage method, the method for warehouse-in efficiency in the especially large-scale relevant database application system.
Background technology
In the communications field, come the unified management network element by webmaster.Along with the growth and the functional diversities of business, the network element of webmaster management is more and more, and the veneer of managing in the network element is also more and more.This has just proposed higher requirement to the response speed of webmaster, the utilance of resource.
Performance management is one of several big functions of webmaster, also is one of the strongest function of real-time.SDH equipment alarm, performance with transmission network are example, each veneer will report 15 minutes a large amount of performances to webmaster every 15 minutes, the veneer of management is many more, 15 minutes performances that report are many more, and each veneer is not to report at one time, but, report to webmaster by NCP, Manager separately before and after 15 minutes.More after a little while, use traditional method at database, though warehouse-in efficiency is low, because 15 minutes performances that report are not too many, so it is little to take database resource.Increasing of careless database, the performance increases at double in 15 minutes that report, if also be to use original traditional storage method, not only warehouse-in efficiency is low, and causes database thread to take the many CPU of 70%-90%, thereby causes the inefficiency of whole webmaster.
Summary of the invention
Defective and deficiency at prior art exists the invention provides a kind of database storage method, reduce the frequency that transmits into database data to database by set up the core buffer that is stored into database data temporarily.
In order to reach the foregoing invention purpose, database storage method of the present invention may further comprise the steps:
(1) in free memory, opens up a core buffer that is used for temporarily being stored into database data;
(2) go into database data for many batches that earlier not timing are received and deposit described core buffer in, again these warehouse-in data centralizations are sent to database.
In the above-mentioned database storage method, step (1) is specially: the core buffer that distributes in free memory, in order to preserve the field value of database data, increase byte at the core buffer head simultaneously, go into zero-time and the termination time that database data enters core buffer in order to preservation.
In the above-mentioned database storage method, step (2) is specially: go into database data and deposit described core buffer in for many batches that earlier not timing are received, when the warehouse-in data volume in the core buffer meets or exceeds the core buffer capacity, again these warehouse-in data centralizations are sent to database.
In the above-mentioned database storage method, step (2) is specially: go into database data for many batches that earlier not timing are received and deposit described core buffer in, regularly these warehouse-in data centralizations are sent to database again.
In the above-mentioned database storage method, step (2) is specially: go into database data for many batches that earlier not timing are received and deposit described core buffer in, when data base querying in core buffer go into database data the time, again these warehouse-in data centralizations are sent to database.
In the above-mentioned database storage method, step (2) further may further comprise the steps:
What (21) database was received each equipment new goes into database data;
(22) database judges whether core buffer is full, if core buffer is full, then earlier goes into the disposable database that sends to of database data existing in the core buffer, deposits the new database data of going into again in core buffer in; If core buffer less than, directly in core buffer, deposit the new database data of going in.
In the above-mentioned database storage method, step (2) further is specially: go into database data and deposit described core buffer in for many batches that earlier not timing are received, when data base querying is gone into database data, judge that query time is whether at buffering area in the time period, if, earlier these warehouse-in data centralizations are sent to database, from database, inquire about again; If query time, is not then directly inquired about from database in the time period at buffering area.
Database storage method of the present invention has improved the performance of webmaster, and data loading efficient is significantly improved, and the database resource occupancy obviously reduces.
Description of drawings
Fig. 1 is the class figure of the invention process case;
Fig. 2 completely puts schematic flow sheet in storage for buffer area;
Fig. 3 is for regularly putting schematic flow sheet in storage;
Fig. 4 is inquiry warehouse-in schematic flow sheet.
Embodiment
The present invention is described in further detail below in conjunction with accompanying drawing:
The objective of the invention is to overcome above-mentioned a large amount of, the separate data warehouse-in efficiency is low, causes the characteristics of the overall performance difference of webmaster, makes full use of internal memory at a high speed, and a kind of storage method very simply efficiently is provided.The present invention has following characteristics: 1) buffer size is determined according to the structure of manual configuration and table; 2) buffer data refreshes management by three kinds of modes.3) in the core buffer, increase zero-time and termination time.4) make full use of the bottom batch warehousing interface that database provides, improve the utilance and the warehouse-in efficiency of buffering area.
Thinking of the present invention is: according to user's setting, in internal memory, split a zone that size is suitable, the buffer memory that is used for data, when data volume more after a little while, data directly exist in the internal memory, when data accumulation when a certain amount of, near or when surpassing core buffer, just disposable warehouse-in reduces the number of times of warehouse-in, thereby improves the efficient of warehouse-in.
Technical scheme of the present invention comprises following content:
The tissue of step 1, core buffer:
Distributing buffer zone in free memory in order to preserve the field value of RKB, increases N byte at the buffering area head simultaneously, in order to preserve zero-time and termination time.Buffering area is only preserved field value, does not preserve field name, thereby has improved the buffering area service efficiency, has also improved the efficient of warehouse-in simultaneously.More than these have benefited from making full use of bottom that database provides warehousing interface in batches.
The management of step 2, core buffer:
3 conditions that buffering area refreshes: one, regularly put in storage in batches by the time; Two, press the actual capacity of buffer area; Three, when inquiry, judge earlier query time whether at buffering area in the time period, if, refresh the buffer area earlier, again from data base querying; If do not exist, then need not refresh buffering area, directly from data base querying; Three conditions can work simultaneously.
In step 2, core buffer manages in 3 kinds of modes:
Mode one, is as shown in Figure 3 gone into database data for many batches that earlier not timing are received and is deposited described core buffer in, regularly these warehouse-in data centralizations is sent to database again.By time warehouse-in regularly in batches, can guarantee that the core buffer data in time put in storage, also can avoid the moment that mass data is inserted buffering area simultaneously, avoid causing resource to wait for and deadlock.
Mode two, as shown in Figure 2, at first, database is received each equipment new goes into database data; Then, database judges whether core buffer is full, if core buffer is full, then earlier goes into the disposable database that sends to of database data existing in the core buffer, deposits the new database data of going into again in core buffer in; If core buffer less than, directly in core buffer, deposit the new database data of going in.By the actual capacity of buffer area, after database is received the message of each equipment, judge whether core buffer is full earlier:, then, in core buffer, insert record more earlier behind the disposable warehouse-in of the data of buffering area if full; If less than, directly in core buffer, insert record.Can reduce core buffer to greatest extent like this and send data frequency to database.
Mode three, is as shown in Figure 4 gone into database data for many batches that earlier not timing are received and is deposited described core buffer in, when data base querying is gone into database data, judge that query time is whether at buffering area in the time period, if, earlier these warehouse-in data centralizations are sent to database, from database, inquire about again; If query time, is not then directly inquired about from database in the time period at buffering area.Can improve core buffer sends validity from data to database like this.
The present invention is further illustrated below in conjunction with concrete enforcement.Examples of implementation are based on the sybase database, describe process of the present invention in detail.
For convenience, fast use this function, this function is modularization, and interface externally is provided, and caller only needs calling interface to get final product.
One, interface
The init function is used for the initialization core buffer.Have 4 parameters, the 1st parameter is void*, is used in reference to the pointer to server; The 2nd parameter is char*, is used in reference to the pointer to concrete database; The 3rd parameter is char*, is used to specify concrete table, and the 4th parameter is acquiescence int, is used to give tacit consent to batch size that refreshes.
The addNew function is in order to increase new record.Have 2 parameters, the 1st parameter is void*, in order to point to the pointer of server; The 2nd parameter is int, and be newly-increased in which way in order to specific data.
The setField function is in order to be provided with the value of different field.This function has a plurality of overload function, in order to set the value of types such as int, char.
The flush function is in order to refresh the content of core buffer.
Two, inheritance
Different databases is realized concrete function on the basis of interface, Fig. 1 has shown the class figure of disparate databases inheritance.
Three, implementation
1, at first calls init () initialization internal memory.The size of internal memory can be given tacit consent to, and also can set (data that can read manual configuration from configuration file) in this function.
When 2, receiving record, after calling addNew (), use setField () function to give each field assignment.
3, when record reaches the record size of regulation, disposable the flush () function that calls calls the bottom warehousing interface that sybase provides, and refreshes the record warehouse-in of internal memory.See Fig. 2
4, by the time of setting, periodic refreshing buffer data.See Fig. 3.
When if 5 interfaces issue the order of this table of inquiry, judge earlier in the order time parameter whether at buffering area in the time, if, then refresh buffering area earlier, again inquiry; If do not exist, directly from data base querying.See Fig. 4.
6, repeating step 3,4, reach the purpose of warehouse-in in batches, thereby reduce the number of times of warehouse-in.
Use traditional storage method, the database resource occupancy is higher, has a strong impact on the overall performance of webmaster.After using embodiment of the present invention instead, warehouse-in efficiency has obviously improved.The database resource occupancy reaches normally, thereby has improved the performance of webmaster.
More than be preferable case study on implementation of the present invention, not in order to restriction protection scope of the present invention.
Claims (7)
1. database storage method is characterized in that: may further comprise the steps:
(1) in free memory, opens up a core buffer that is used for temporarily being stored into database data;
(2) go into database data for many batches that earlier not timing are received and deposit described core buffer in, again these warehouse-in data centralizations are sent to database.
2. database storage method according to claim 1, it is characterized in that: step (1) is specially: the core buffer that distributes in free memory, in order to preserve the field value of database data, increase byte at the core buffer head simultaneously, go into zero-time and the termination time that database data enters core buffer in order to preservation.
3. database storage method according to claim 1, it is characterized in that: step (2) is specially: go into database data and deposit described core buffer in for many batches that earlier not timing are received, when the warehouse-in data volume in the core buffer meets or exceeds the core buffer capacity, again these warehouse-in data centralizations are sent to database.
4. database storage method according to claim 1 is characterized in that: step (2) is specially: go into database data for many batches that earlier not timing are received and deposit described core buffer in, regularly these warehouse-in data centralizations are sent to database again.
5. database storage method according to claim 1, it is characterized in that: step (2) is specially: go into database data and deposit described core buffer in for many batches that earlier not timing are received, when data base querying in core buffer go into database data the time, again with these the warehouse-in data centralizations send to database.
6. database storage method according to claim 3 is characterized in that: step (2) further may further comprise the steps:
What (21) database was received each equipment new goes into database data;
(22) database judges whether core buffer is full, if core buffer is full, then earlier goes into the disposable database that sends to of database data existing in the core buffer, deposits the new database data of going into again in core buffer in; If core buffer less than, directly in core buffer, deposit the new database data of going in.
7. database storage method according to claim 5, it is characterized in that: step (2) further is specially: go into database data and deposit described core buffer in for many batches that earlier not timing are received, when data base querying is gone into database data, judge that query time is whether at buffering area in the time period, if, earlier these warehouse-in data centralizations are sent to database, from database, inquire about again; If query time, is not then directly inquired about from database in the time period at buffering area.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2006101409868A CN101170433A (en) | 2006-10-25 | 2006-10-25 | Database storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2006101409868A CN101170433A (en) | 2006-10-25 | 2006-10-25 | Database storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101170433A true CN101170433A (en) | 2008-04-30 |
Family
ID=39390912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2006101409868A Pending CN101170433A (en) | 2006-10-25 | 2006-10-25 | Database storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101170433A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101277272B (en) * | 2008-05-16 | 2010-12-15 | 北京航空航天大学 | Method for implementing magnanimity broadcast data warehouse-in |
| CN101980190A (en) * | 2010-10-15 | 2011-02-23 | 中兴通讯股份有限公司 | Method and device for quickly putting service data into base |
| CN101321089B (en) * | 2008-07-07 | 2011-04-20 | 中兴通讯股份有限公司 | Warehouse entry method of performance data in telecommunication network management system |
| CN102542054A (en) * | 2011-12-29 | 2012-07-04 | 厦门雅迅网络股份有限公司 | Method utilizing cache tables to improve insertion performance of data in database |
| CN102750377A (en) * | 2012-06-21 | 2012-10-24 | 深圳市谷米科技有限公司 | Massive data storage and retrieval method |
| CN105843819A (en) * | 2015-01-15 | 2016-08-10 | ***通信集团河南有限公司 | Method and device for exporting data |
| CN106227794A (en) * | 2016-07-20 | 2016-12-14 | 北京航空航天大学 | The storage method and apparatus of dynamic attribute data in temporal diagram data |
-
2006
- 2006-10-25 CN CNA2006101409868A patent/CN101170433A/en active Pending
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101277272B (en) * | 2008-05-16 | 2010-12-15 | 北京航空航天大学 | Method for implementing magnanimity broadcast data warehouse-in |
| CN101321089B (en) * | 2008-07-07 | 2011-04-20 | 中兴通讯股份有限公司 | Warehouse entry method of performance data in telecommunication network management system |
| CN101980190A (en) * | 2010-10-15 | 2011-02-23 | 中兴通讯股份有限公司 | Method and device for quickly putting service data into base |
| CN102542054A (en) * | 2011-12-29 | 2012-07-04 | 厦门雅迅网络股份有限公司 | Method utilizing cache tables to improve insertion performance of data in database |
| CN102542054B (en) * | 2011-12-29 | 2017-02-08 | 厦门雅迅网络股份有限公司 | Method utilizing cache tables to improve insertion performance of data in database |
| CN102750377A (en) * | 2012-06-21 | 2012-10-24 | 深圳市谷米科技有限公司 | Massive data storage and retrieval method |
| CN102750377B (en) * | 2012-06-21 | 2016-07-27 | 深圳市谷米科技有限公司 | Mass data storage and search method |
| CN105843819A (en) * | 2015-01-15 | 2016-08-10 | ***通信集团河南有限公司 | Method and device for exporting data |
| CN105843819B (en) * | 2015-01-15 | 2020-07-10 | ***通信集团河南有限公司 | Data export method and device |
| CN106227794A (en) * | 2016-07-20 | 2016-12-14 | 北京航空航天大学 | The storage method and apparatus of dynamic attribute data in temporal diagram data |
| CN106227794B (en) * | 2016-07-20 | 2019-09-17 | 北京航空航天大学 | The storage method and device of dynamic attribute data in tense diagram data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101170433A (en) | Database storage method | |
| CN202058147U (en) | Distribution type real-time database management system | |
| CN100407203C (en) | Method for processing mass data | |
| WO2007115477A1 (en) | Data synchronization method und system | |
| CN100543750C (en) | A kind of matrix type data caching method and device of using based on WEB | |
| CN101990256A (en) | Long-connection management device and method for managing link resources of long-connection communication | |
| CN104731799B (en) | Main memory DBM device | |
| CN105357286B (en) | A kind of real-time messages orientation method for pushing based on Web | |
| CN104778188A (en) | Distributed device log collection method | |
| WO2021077748A1 (en) | Method for achieving linear growth of traffic analysis service performance on basis of dpdk | |
| CN103645904A (en) | Cache realization method of interface calling | |
| CN100359474C (en) | Method for constructing Web server based on soft flow construction and server thereof | |
| CN110708256A (en) | CDN scheduling method, device, network equipment and storage medium | |
| CN102158550B (en) | IEC61850-based power quality transient data transmission method | |
| CN101355488B (en) | Method and system for controlling flow of information series business initiated by network | |
| CN102609464A (en) | Method and device for associative table query of MONGODB shards | |
| CN106777387A (en) | A kind of Internet of Things big data access method based on HBase | |
| CN101997701B (en) | Data processing method and a device thereof | |
| CN103678522B (en) | A kind of data acquisition of intelligent substation metering system and conversion method | |
| CN107346270A (en) | Method and system based on the sets cardinal calculated in real time | |
| CN100488114C (en) | Network element management method and system | |
| CN102750368A (en) | High-speed importing method of cluster data in data base | |
| CN109344137A (en) | A kind of log storing method and system | |
| CN102916902A (en) | Method and device for storing data | |
| CN100525211C (en) | Method and system for implementing integral dispatching in network element management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Open date: 20080430 |