CN103473271B - A kind of optimization storage method for mass data - Google Patents
A kind of optimization storage method for mass data Download PDFInfo
- Publication number
- CN103473271B CN103473271B CN201310363130.7A CN201310363130A CN103473271B CN 103473271 B CN103473271 B CN 103473271B CN 201310363130 A CN201310363130 A CN 201310363130A CN 103473271 B CN103473271 B CN 103473271B
- Authority
- CN
- China
- Prior art keywords
- data
- period
- cutting
- million magnitudes
- magnitudes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of optimization storage method for mass data, carry out as follows：S1, the data Jing Guo legitimacy verifies according to certain rule are divided into fritter, the data of each fritter are stored in corresponding table；S2, the table to data storage in step S1 judge, if the data in table reach million magnitudes, this table is then subjected to data cutting as time interval according to the period of setting, is divided into small table to store again, the data in the correspondence period are only stored in each small table.The accumulation that the present invention solves mass data causes database rapid expansion, unlimited to expand, and inquiry velocity is slow, reduces the drawbacks such as data utilization ratio, reaches reduction database loads, improves the effect of data utilization ratio.
Description
Technical field
The present invention relates to a kind of optimization storage method of data, and in particular to a kind of optimization storage method of mass data,
Belong to microcomputer data processing field.
Background technology
At present, developing rapidly due to data, data volume is increasing, and the storage of big data is with inquiring about as very big difficulty
Topic.How cloud is more and more applied in many message areas as the treatment technology of big data and fast data now
As needed come handle huge cloud data into naturally the problem of.It is directly to deposit to handle mass data most straightforward approach
Enter corresponding database, this method requires high to server performance, and if use will arrive huge database every time
In go the data needed for inquiry, inquiry velocity is slow, and efficiency is low, and if only making a simply point table is also unable to reach preferable effect
Really, the load of database is still very big, it is impossible to quickly data are positioned, it is impossible to improve data query speed.
The content of the invention
Goal of the invention：In view of the above-mentioned deficiencies in the prior art, it is an object of the present invention to provide a kind of reduction database loads, carry
The optimization storage method for mass data of high data query speed and utilization ratio.
Technical scheme：The present invention provides a kind of optimization storage method for mass data, carries out as follows：
S1, the data Jing Guo legitimacy verifies according to certain rule are divided into fritter, the data of each fritter are deposited
Enter in corresponding table；
S2, the table to data storage in step S1 judge, if the data in table reach million magnitudes, by this table
Data cutting is carried out as time interval according to the period of setting, is divided into small table to store again, is only stored in each small table
Data in the correspondence period.
The technical solution of the present invention is further defined as, in step S2, the period set is time interval by short
To the i long period, when carrying out data cutting according to the period of setting, i-th of period is first selected as time interval
Carry out data cutting；After data cutting, the small table of generation is judged, if also there are data reaches million magnitudes
Table, then select the i-th -1 period that again data are reached with the table of million magnitudes carries out data cutting, circulation as time interval
Operate as above, until no data reaches that the table of million magnitudes is present or the period of setting all uses.
Further, in step S2, the period set as one day, one week and one month, according to the time of setting
When Duan Jinhang data are cut, first select carry out data cutting as time interval within one month；After data cutting, to the small of generation
Table is judged, the table of million magnitudes is reached if there is no data, then completes this data storage；If also there are data to reach
To the table of million magnitudes, then select week age section that again data are reached with the table of million magnitudes carries out data cutting；Data are cut
After cutting, the small table of generation is judged, the table of million magnitudes is reached if there is no data, then completed this data and deposit
Storage；If also there is the table that data reach million magnitudes, one period of selection reaches the table of million magnitudes to data again
Data cutting is carried out, this data storage is completed.
Further, if the data in the small table being divided into again are carried out certainly beyond the scope of the period of setting
Dynamic deletion.
Further, in step S2, the table that data reach million magnitudes is carried out after data cutting, the data to segmentation are entered
Row judges, if the table after two and the merging of more than two data units first merges table still less than million magnitudes,
Then it is restored again into small table.
Beneficial effect：A kind of optimization storage method for mass data that the present invention is provided, by times to data
Reach that the table of million magnitudes carries out data cutting, the data to each period are separately maintained, and solve mass data
Accumulation causes database rapid expansion, unlimited to expand, and inquiry velocity is slow, reduces the drawbacks such as data utilization ratio, reaches reduction
Database loads, improve the effect of data utilization ratio.
Brief description of the drawings
A kind of flow chart for optimization storage method for mass data that Fig. 1 provides for the present invention.
Embodiment
Technical solution of the present invention is described in detail below, but protection scope of the present invention is not limited to the implementation
Example.
Embodiment：The present embodiment provides a kind of optimization storage method for mass data, and the present invention is applied to cloud
To that in the reception processing of big data, after cloud server mass data is parsed to data, will be optimized to data
Storage, the method flow diagram of specific optimization storage as shown in figure 1, carry out as follows：
S1, the data Jing Guo legitimacy verifies according to certain rule are divided into fritter, the data of each fritter are deposited
Enter in corresponding table.
Legitimacy verifies include the verification and the verification to data in bag to receiving packet, and packet, which is mainly, to be passed through
Whether md5 values are correct come the source for verifying bag, if bag is injected for network attack, if packet is legal, by packet
Parsed, the data to the inside are analyzed, if be required correct data.
The rule of data segmentation, depending on the corresponding table of data, all data can not possibly be stored in a table, solution
Data after analysis have fixed mark, and to show each several part data, which table this is present in, such as user profile has user's table
In, network traffic information is present in the table corresponding to network traffics, and fixed rule is the mark appointed.
The table of data storage is just built up when building storehouse, and does not need dynamic creating table.Different list data structure is not
With, it is that interdependence is again each independent between table and table that table, which is, and the data in a table are that data pass through in another table
Calculate what is be stored in after merging, it is each independent when using to be used.Data meeting one of the data if not handling in table in table
Straight increase, the purpose of this method is exactly processing increased data always, so as not to unlimited increase.
It is technology ripe at present, ability that verification to network data legitimacy, the segmentation to data and database, which build table,
The technical staff in domain can be realized using conventional knowledge, not discussed herein.
S2, the table to data storage in step S1 judge, if the data in table reach million magnitudes, by this table
Period according to setting carries out data cutting as time interval and the data of segmentation is judged, if two and two with
On data unit merge after table still less than million magnitudes, then first table is merged, is then restored again into small table.Each small table
In only store correspondence the period in data.Receive a pen data within such as every 15 minutes, the pen data is stored in nearest one day
In table, then every four pen data synthesizes a pen data and is stored in the tables of data of nearest one week, by that analogy, during by exceeding in each table
Between the data of scope delete automatically, the size and data volume of such database are all without invalid increase.
Specifically the method for progress Interval data is at times：When the period set is time interval from short to long i
Between section, according to setting period carry out data cutting when, first select i-th of period as time interval carry out data cut
Cut；After data cutting, the small table of generation is judged, if also there is the table that data reach million magnitudes, the is selected
I-1 period reaches that the table of million magnitudes carries out data cutting as time interval to data again, and circulation is operated as above, directly
Reach that the table of million magnitudes is present or the period of setting all uses to no data.If in the small table being divided into again
Data are then automatically deleted beyond the scope of the period of setting.
In the present embodiment, the period set carries out data as one day, one week and one month, according to the period of setting and cut
When cutting, first select carry out data cutting as time interval within one month；After data cutting, the small table of generation is judged,
The table of million magnitudes is reached if there is no data, then completes this data storage；If also there are data reaches million magnitudes
Table, then select week age section data are reached again million magnitudes table carry out data cutting；After data cutting, to life
Into small table judged, the table of million magnitudes is reached if there is no data, then completes this data storage；If also existed
Data reach the table of million magnitudes, then select a period that again data are reached with the table of million magnitudes carries out data cutting,
Complete this data storage.If the data in the small table being divided into again are carried out certainly beyond the scope of the period of setting
Dynamic deletion.
The present invention carries out data cutting by table huge to data at times, and the data to each period carry out list
Solely safeguard, solving the accumulation of mass data causes database rapid expansion, unlimited to expand, inquiry velocity is slow.
As described above, although the present invention has been represented and described with reference to specific preferred embodiment, it must not be explained
For to the limitation of itself of the invention., can be right under the premise of the spirit and scope of the present invention that appended claims are defined are not departed from
Various changes can be made in the form and details for it.
Claims (1)
1. a kind of optimization storage method for mass data, it is characterised in that carry out as follows：S1, it will pass through legal
Property verification data be divided into fritter according to certain rule, the data of each fritter are stored in corresponding table；S2, to step
The table of data storage is judged in S1, if the data in table reach million magnitudes, by period of this table according to setting
Data cutting is carried out as time interval, is divided into small table to store again, the number in the correspondence period is only stored in each small table
According to；
In step S2, i period of the period set as time interval from short to long, enter according to the period of setting
When row data are cut, i-th of period is first selected to carry out data cutting as time interval；After data cutting, to generation
Small table is judged, if also there is the table that data reach million magnitudes, selection the i-th -1 period as time interval again
The secondary table that data are reached with million magnitudes carries out data cutting, and circulation is operated as above, until no data reaches the table of million magnitudes
In the presence of or setting period all use；
In step S2, the period set, as one day, one week and one month, data cutting is carried out according to the period of setting
When, first select carry out data cutting as time interval within one month；After data cutting, the small table of generation is judged, such as
The table that data reach million magnitudes is not present in fruit, then completes this data storage；If also there are data reaches million magnitudes
Table, then select week age section that again data are reached with the table of million magnitudes carries out data cutting；After data cutting, to generation
Small table judged, the table of million magnitudes is reached if there is no data, then completes this data storage；If also there is number
According to the table for reaching million magnitudes, then select a period that again data are reached with the table of million magnitudes carries out data cutting, it is complete
Into this data storage；
If the data in the small table being divided into again are automatically deleted beyond the scope of the period of setting；
In step S2, the table that data reach million magnitudes is carried out after data cutting, the data to segmentation judge, if two
Table is then first merged, is then restored again into small still less than million magnitudes by the table after individual and more than two data units merging
Table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310363130.7A CN103473271B (en) | 2013-08-20 | 2013-08-20 | A kind of optimization storage method for mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310363130.7A CN103473271B (en) | 2013-08-20 | 2013-08-20 | A kind of optimization storage method for mass data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103473271A CN103473271A (en) | 2013-12-25 |
| CN103473271B true CN103473271B (en) | 2017-09-26 |
Family
ID=49798119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310363130.7A Active CN103473271B (en) | 2013-08-20 | 2013-08-20 | A kind of optimization storage method for mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103473271B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182546B (en) * | 2014-09-09 | 2017-10-27 | 北京国双科技有限公司 | The data query method and device of database |
| WO2016138614A1 (en) * | 2015-03-02 | 2016-09-09 | Microsoft Technology Licensing, Llc | Management of database queries against large datasets |
| CN106600735A (en) * | 2016-12-14 | 2017-04-26 | 天津飞鸟科技有限公司 | Fingerprint time recorder and mobile phone communication recognition intelligent system |
| CN107798048A (en) * | 2017-07-28 | 2018-03-13 | 昆明理工大学 | A kind of negative data library management method for radio heliograph Mass Data Management |
| CN109800252A (en) * | 2019-03-05 | 2019-05-24 | 深圳市国晨工程造价咨询有限公司 | A kind of engineering project Records Information Management System |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100484017C (en) * | 2004-09-08 | 2009-04-29 | 大唐移动通信设备有限公司 | Method for statistics of mass performance data in network element management system |
| CN101207513B (en) * | 2006-12-22 | 2012-09-05 | 中兴通讯股份有限公司 | Apparatus and method for saving historical data |
| CN101566986A (en) * | 2008-04-21 | 2009-10-28 | 阿里巴巴集团控股有限公司 | Method and device for processing data in online business processing |
| CN101697152A (en) * | 2009-10-23 | 2010-04-21 | 金蝶软件（中国）有限公司 | Database storage system and method and device for splitting data thereof |
| US20130066882A1 (en) * | 2011-09-09 | 2013-03-14 | Onzo Limited | Data storage method and system |
-
2013
- 2013-08-20 CN CN201310363130.7A patent/CN103473271B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN103473271A (en) | 2013-12-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103473271B (en) | A kind of optimization storage method for mass data | |
| CN103116605B (en) | A kind of microblog hot event real-time detection method based on monitoring subnet and system | |
| JP2015524952A5 (en) | ||
| CN103593433B (en) | A kind of diagram data processing method towards magnanimity time series data and system | |
| CN104615627B (en) | A kind of event public feelings information extracting method and system based on microblog | |
| CN104021205B (en) | Method and device for establishing microblog index | |
| CN104216889B (en) | Data dissemination analyzing and predicting method and system based on cloud service | |
| CN103488683B (en) | Microblog data management system and implementation method thereof | |
| CN110083722A (en) | A kind of electronic drawing lookup method, device, equipment and readable storage medium storing program for executing | |
| CN105471893B (en) | A kind of distributed equivalent data flow connection method | |
| CN104750718B (en) | The searching method and equipment of a kind of data information | |
| CN107239885A (en) | A kind of electricity power engineering towards supplier digitizes handover resource management system | |
| CN107346270A (en) | Method and system based on the sets cardinal calculated in real time | |
| CN103995886B (en) | A kind of various dimensions product-design knowledge pushes framework and construction method | |
| Wang et al. | Research of massive web log data mining based on cloud computing | |
| CN108712337B (en) | Multipath bandwidth scheduling method in high-performance network | |
| CN106557985A (en) | A kind of social network information propagating source method for solving based on random walk | |
| CN106648828A (en) | Field-oriented virtual machine fast deployment method | |
| CN106612298B (en) | A kind of content distribution method and system based on large-scale network node | |
| CN104992698A (en) | Song on-demand arrangement method and device for KTV song-on-demand system | |
| CN104516956B (en) | A kind of site information increment crawling method | |
| Jiang et al. | A Task Offloading Method with Edge for 5G‐Envisioned Cyber‐Physical‐Social Systems | |
| CN103458032A (en) | Method and system for dynamic statistics and information compression of spatial data access law | |
| CN106411622A (en) | Table entry processing method and device | |
| CN203689112U (en) | Centralized-control system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 301-302, 3rd Floor, Tiancheng Information Building, No. 88 South Tiancheng Road, High Speed Rail New City, Xiangcheng District, Suzhou City, Jiangsu Province, 215133 Patentee after: SUZHOU MAXNET NETWORK SAFETY TECHNOLOGY Co.,Ltd. Address before: 215021 International Science and Technology Park Phase III 8B, No. 1355 Jinjihu Avenue, Industrial Park, Suzhou City, Jiangsu Province Patentee before: SUZHOU MAXNET NETWORK SAFETY TECHNOLOGY Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |