CN103473271A - Optimized storing method for mass data - Google Patents
Optimized storing method for mass data Download PDFInfo
- Publication number
- CN103473271A CN103473271A CN2013103631307A CN201310363130A CN103473271A CN 103473271 A CN103473271 A CN 103473271A CN 2013103631307 A CN2013103631307 A CN 2013103631307A CN 201310363130 A CN201310363130 A CN 201310363130A CN 103473271 A CN103473271 A CN 103473271A
- Authority
- CN
- China
- Prior art keywords
- data
- magnitudes
- time period
- cutting
- reach
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an optimized storing method for mass data, which is carried out according to the following steps of S1. dividing data subjected to validity checking into small blocks according to a certain rule, and storing each small block of data into a corresponding list; S2. judging the lists for storing the data in S1, and if the data in the lists reach the magnitude order of millions, dividing the data of the lists by using set time periods as time intervals, and dividing the lists into small lists for storage again, wherein each small list is used for only storing the data within the corresponding time period. Through the optimized storing method for the mass data, the problems of rapid database expansion, limitless enlargement and low inquiry speed caused by the accumulation of the mass data, and the disadvantage of reducing data-utilizing efficiency and the like are solved, and the effects of reducing the load of a database and improving the data-utilizing efficiency are achieved.
Description
Technical field
The present invention relates to a kind of optimization storage means of data, be specifically related to a kind of optimization storage means of mass data, belong to the microcomputer data processing field.
Background technology
At present, due to the develop rapidly of data, data volume is increasing, and the storage of large data becomes a very large difficult problem with inquiry.Cloud, as the treatment technology of large data and fast data, more and more is applied in a lot of message areas now, how to process as required huge cloud data and has become natural problem.Processing the most direct method of mass data is directly to deposit corresponding database in, this method requires high to server performance, if each use all will be inquired about required data in huge database, inquiry velocity is slow, efficiency is low, and if only make simple submeter and also can't reach desirable effect, the load of database is still very large, can not position data fast, can't improve data query speed.
Summary of the invention
Goal of the invention: the object of the invention is to for the deficiencies in the prior art, a kind of storage means of the optimization for mass data that reduces database loads, improves data query speed and utilization ratio is provided.
Technical scheme: the invention provides a kind of storage means of the optimization for mass data, carry out as follows:
S1, will be divided into fritter according to certain rule through the data of legitimacy verification, the data of each fritter will be deposited in corresponding table;
S2, in step S1 the storage data table judged, if the data in table reach 1,000,000 magnitudes, this table is carried out to the data cutting according to the time period of setting as the time interval, again be divided into little table storage, only store the data in the corresponding time period in each little table.
Being further defined to of technical solution of the present invention, in step S2, the time period of described setting is a time interval i from short to long time period, while according to the time period of setting, carrying out the data cutting, first selects i time period to carry out the data cutting as the time interval; After the data cutting, the little table generated is judged, if also exist data to reach the table of 1,000,000 magnitudes, select the table that i-1 time period reaches 1,000,000 magnitudes to data again as the time interval to carry out the data cutting, circulation is operation as above, until countless certificate reaches, the table of 1,000,000 magnitudes exists or the time period of setting is all used.
Further, in step S2, the time period of described setting is one day, one week and one month, while according to the time period of setting, carrying out the data cutting, first selects within one month, as the time interval, carry out the data cutting; After the data cutting, the little table generated is judged, if there is no data reach the table of 1,000,000 magnitudes, complete this secondary data storage; If also exist data to reach the table of 1,000,000 magnitudes, the table of selecting the week age section again data to be reached to 1,000,000 magnitudes carries out the data cutting; After the data cutting, the little table generated is judged, if there is no data reach the table of 1,000,000 magnitudes, complete this secondary data storage; If also exist data to reach the table of 1,000,000 magnitudes, select the table that a time period reaches 1,000,000 magnitudes to data again to carry out the data cutting, complete this secondary data storage.
Further, if the data in the little table again be divided into exceed the scope of the time period of setting, delete automatically.
Further, in step S2, after the table that data are reached to 1,000,000 magnitudes carries out the data cutting, the data of cutting apart are judged, if the table after two and plural data unit merge still is less than 1,000,000 magnitudes, first table is merged, and then deposited in little table.
Beneficial effect: a kind of storage means of the optimization for mass data provided by the invention, logical table that at times data is reached to 1,000,000 magnitudes carries out the data cutting, data to each time period are safeguarded separately, the accumulation that has solved mass data causes the database rapid expansion, unlimited expansion, inquiry velocity is slow, has reduced the drawbacks such as data utilization ratio, reach the reduction database loads, improve the effect of data utilization ratio.
The accompanying drawing explanation
The process flow diagram that Fig. 1 is a kind of storage means of the optimization for mass data provided by the invention.
Embodiment
Below technical solution of the present invention is elaborated, but protection scope of the present invention is not limited to described embodiment.
embodiment:the present embodiment provides a kind of storage means of the optimization for mass data, the present invention is applied in the reception & disposal of cloud to large data, after Cloud Server reception mass data is resolved data, to be optimized storage to data, the method flow diagram of concrete optimization storage as shown in Figure 1, carries out as follows:
S1, will be divided into fritter according to certain rule through the data of legitimacy verification, the data of each fritter will be deposited in corresponding table.
The legitimacy verification comprises verification to receiving packet and to the verification of data in bag, whether packet is mainly to come the source of verification bag correct by the md5 value, whether be that network attack injects bag, if packet is legal, packet is resolved, to the data analysis of the inside, whether be needed correct data.
The rule of Data Segmentation, the table corresponding according to data determined, all data can not leave in a table, data after parsing have fixing sign show the each several part data this exist which the table in, such as user profile exists in subscriber's meter, network traffic information exists in the corresponding table of network traffics, and fixing rule is the sign of appointing.
The table of storage data is just built up when building storehouse, does not need dynamically to build table.Different list data structure is different, table be table with show between be interdependence again separately independently, the data in table are that in another table, data deposit in after merging by calculating, during use, independently are used separately.If the data that the data in table are not processed in table can increase always, the purpose of the method is exactly to process the data that always increase, and makes it can infinitely not increase.
To the verification of network data legitimacy, to data cut apart and database is built table for current proven technique, those skilled in the art can adopt conventional knowledge to realize, does not do discussion herein.
S2, in step S1 the storage data table judged, if the data in table reach 1,000,000 magnitudes, this table being carried out to the data cutting according to the time period of setting as the time interval is judged the data of cutting apart, if the table after two and plural data unit merge still is less than 1,000,000 magnitudes, first table is merged, and then deposited in little table.The data of only storage correspondence in the time period in each little table.Such as within every 15 minutes, receiving data, these data are deposited in the table of nearest a day, then the synthetic data of every four data deposit in the tables of data of nearest a week, by that analogy, the data that exceed time range in each table are deleted automatically, and so size and the data volume of database can invalidly not increase.
Concrete method of carrying out at times Interval data is: the time period of setting is a time interval i from short to long time period, while according to the time period of setting, carrying out the data cutting, first selects i time period to carry out the data cutting as the time interval; After the data cutting, the little table generated is judged, if also exist data to reach the table of 1,000,000 magnitudes, select the table that i-1 time period reaches 1,000,000 magnitudes to data again as the time interval to carry out the data cutting, circulation is operation as above, until countless certificate reaches, the table of 1,000,000 magnitudes exists or the time period of setting is all used.If the data in the little table again be divided into exceed the scope of the time period of setting, delete automatically.
In the present embodiment, the time period of setting is one day, one week and one month, while according to the time period of setting, carrying out the data cutting, first selects within one month, as the time interval, carry out the data cutting; After the data cutting, the little table generated is judged, if there is no data reach the table of 1,000,000 magnitudes, complete this secondary data storage; If also exist data to reach the table of 1,000,000 magnitudes, the table of selecting the week age section again data to be reached to 1,000,000 magnitudes carries out the data cutting; After the data cutting, the little table generated is judged, if there is no data reach the table of 1,000,000 magnitudes, complete this secondary data storage; If also exist data to reach the table of 1,000,000 magnitudes, select the table that a time period reaches 1,000,000 magnitudes to data again to carry out the data cutting, complete this secondary data storage.If the data in the little table again be divided into exceed the scope of the time period of setting, delete automatically.
To data, huge table carries out the data cutting to logical of the present invention at times, and the data of each time period are safeguarded separately, and the accumulation that has solved mass data causes the database rapid expansion, unlimited expansion, and inquiry velocity is slow.
As mentioned above, although meaned and explained the present invention with reference to specific preferred embodiment, it shall not be construed as the restriction to the present invention self.Under the spirit and scope of the present invention prerequisite that does not break away from the claims definition, can make in the form and details various variations to it.
Claims (5)
1. the storage means of the optimization for mass data, is characterized in that, carries out as follows:
S1, will be divided into fritter according to certain rule through the data of legitimacy verification, the data of each fritter will be deposited in corresponding table;
S2, in step S1 the storage data table judged, if the data in table reach 1,000,000 magnitudes, this table is carried out to the data cutting according to the time period of setting as the time interval, again be divided into little table storage, only store the data in the corresponding time period in each little table.
2. a kind of storage means of the optimization for mass data according to claim 1, it is characterized in that, in step S2, the time period of described setting is a time interval i from short to long time period, while according to the time period of setting, carrying out the data cutting, first select i time period to carry out the data cutting as the time interval; After the data cutting, the little table generated is judged, if also exist data to reach the table of 1,000,000 magnitudes, select the table that i-1 time period reaches 1,000,000 magnitudes to data again as the time interval to carry out the data cutting, circulation is operation as above, until countless certificate reaches, the table of 1,000,000 magnitudes exists or the time period of setting is all used.
3. a kind of storage means of the optimization for mass data according to claim 2, it is characterized in that, in step S2, the time period of described setting is one day, one week and one month, while according to the time period of setting, carrying out the data cutting, first select within one month, as the time interval, carry out the data cutting; After the data cutting, the little table generated is judged, if there is no data reach the table of 1,000,000 magnitudes, complete this secondary data storage; If also exist data to reach the table of 1,000,000 magnitudes, the table of selecting the week age section again data to be reached to 1,000,000 magnitudes carries out the data cutting; After the data cutting, the little table generated is judged, if there is no data reach the table of 1,000,000 magnitudes, complete this secondary data storage; If also exist data to reach the table of 1,000,000 magnitudes, select the table that a time period reaches 1,000,000 magnitudes to data again to carry out the data cutting, complete this secondary data storage.
4. a kind of storage means of the optimization for mass data according to claim 1, is characterized in that, if the data in the little table again be divided into exceed the scope of the time period of setting, deletes automatically.
5. a kind of storage means of the optimization for mass data according to claim 1, it is characterized in that, in step S2, after the table that data are reached to 1,000,000 magnitudes carries out the data cutting, the data of cutting apart are judged, if the table after two and plural data unit merge still is less than 1,000,000 magnitudes, first table is merged, and then deposited in little table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310363130.7A CN103473271B (en) | 2013-08-20 | 2013-08-20 | A kind of optimization storage method for mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310363130.7A CN103473271B (en) | 2013-08-20 | 2013-08-20 | A kind of optimization storage method for mass data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103473271A true CN103473271A (en) | 2013-12-25 |
| CN103473271B CN103473271B (en) | 2017-09-26 |
Family
ID=49798119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310363130.7A Active CN103473271B (en) | 2013-08-20 | 2013-08-20 | A kind of optimization storage method for mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103473271B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182546A (en) * | 2014-09-09 | 2014-12-03 | 北京国双科技有限公司 | Method and device for querying data in databases |
| CN106233287A (en) * | 2015-03-02 | 2016-12-14 | 微软技术许可有限责任公司 | Management to the data base querying of large data collection |
| CN106600735A (en) * | 2016-12-14 | 2017-04-26 | 天津飞鸟科技有限公司 | Fingerprint time recorder and mobile phone communication recognition intelligent system |
| CN107798048A (en) * | 2017-07-28 | 2018-03-13 | 昆明理工大学 | A kind of negative data library management method for radio heliograph Mass Data Management |
| CN109800252A (en) * | 2019-03-05 | 2019-05-24 | 深圳市国晨工程造价咨询有限公司 | A kind of engineering project Records Information Management System |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1747398A (en) * | 2004-09-08 | 2006-03-15 | 大唐移动通信设备有限公司 | Mass performance data statistical method in network element management system |
| CN101207513A (en) * | 2006-12-22 | 2008-06-25 | 中兴通讯股份有限公司 | Apparatus and method for saving historical data |
| CN101566986A (en) * | 2008-04-21 | 2009-10-28 | 阿里巴巴集团控股有限公司 | Method and device for processing data in online business processing |
| CN101697152A (en) * | 2009-10-23 | 2010-04-21 | 金蝶软件（中国）有限公司 | Database storage system and method and device for splitting data thereof |
| US20130066882A1 (en) * | 2011-09-09 | 2013-03-14 | Onzo Limited | Data storage method and system |
-
2013
- 2013-08-20 CN CN201310363130.7A patent/CN103473271B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1747398A (en) * | 2004-09-08 | 2006-03-15 | 大唐移动通信设备有限公司 | Mass performance data statistical method in network element management system |
| CN101207513A (en) * | 2006-12-22 | 2008-06-25 | 中兴通讯股份有限公司 | Apparatus and method for saving historical data |
| CN101566986A (en) * | 2008-04-21 | 2009-10-28 | 阿里巴巴集团控股有限公司 | Method and device for processing data in online business processing |
| CN101697152A (en) * | 2009-10-23 | 2010-04-21 | 金蝶软件（中国）有限公司 | Database storage system and method and device for splitting data thereof |
| US20130066882A1 (en) * | 2011-09-09 | 2013-03-14 | Onzo Limited | Data storage method and system |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182546A (en) * | 2014-09-09 | 2014-12-03 | 北京国双科技有限公司 | Method and device for querying data in databases |
| CN104182546B (en) * | 2014-09-09 | 2017-10-27 | 北京国双科技有限公司 | The data query method and device of database |
| CN106233287A (en) * | 2015-03-02 | 2016-12-14 | 微软技术许可有限责任公司 | Management to the data base querying of large data collection |
| CN106233287B (en) * | 2015-03-02 | 2019-07-02 | 微软技术许可有限责任公司 | Management to the data base querying of large data collection |
| CN106600735A (en) * | 2016-12-14 | 2017-04-26 | 天津飞鸟科技有限公司 | Fingerprint time recorder and mobile phone communication recognition intelligent system |
| CN107798048A (en) * | 2017-07-28 | 2018-03-13 | 昆明理工大学 | A kind of negative data library management method for radio heliograph Mass Data Management |
| CN109800252A (en) * | 2019-03-05 | 2019-05-24 | 深圳市国晨工程造价咨询有限公司 | A kind of engineering project Records Information Management System |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103473271B (en) | 2017-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103473271A (en) | Optimized storing method for mass data | |
| CN103902593B (en) | A kind of method and apparatus of Data Migration | |
| CN105912397A (en) | Resources management method and device | |
| CN102780641B (en) | Flow table aging method and device of quick forwarding engine, and switch | |
| CN104363282B (en) | A kind of cloud computing resource scheduling method and device | |
| CN104134173A (en) | Main and distribution network graph/model/data integrated system based on remote retrieval and information interaction | |
| CN106649646A (en) | Method and device for deleting duplicated data | |
| CN103914265A (en) | Cluster fine-grained memory management method | |
| CN104765749A (en) | Data storage method and device | |
| CN103970853A (en) | Method and device for optimizing search engine | |
| CN102404411A (en) | Data synchronization method of cloud storage system | |
| CN104298687A (en) | Hash partitioning management method and device | |
| CN108174235A (en) | A kind of video loading method and device | |
| CN103853838A (en) | Data processing method and device | |
| CN103607731A (en) | Method and device for processing measurement reports | |
| Sun et al. | A cost-efficient scheduling algorithm of on-demand broadcasts | |
| CN104978695A (en) | Electricity charge data processing method and apparatus | |
| CN104123329B (en) | Searching method and device | |
| CN102685222B (en) | A kind of cloud SRM device for electric power system | |
| CN103297542A (en) | Operating system bus and balancing method supporting online expansion and retraction of components | |
| CN104242993A (en) | Medium-low voltage power communication access network bandwidth predication method | |
| CN104679821A (en) | Data access processing method and data access processing device | |
| CN104992698A (en) | Song on-demand arrangement method and device for KTV song-on-demand system | |
| CN106612298B (en) | A kind of content distribution method and system based on large-scale network node | |
| CN103796226A (en) | Network optimizing method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 301-302, 3rd Floor, Tiancheng Information Building, No. 88 South Tiancheng Road, High Speed Rail New City, Xiangcheng District, Suzhou City, Jiangsu Province, 215133 Patentee after: SUZHOU MAXNET NETWORK SAFETY TECHNOLOGY Co.,Ltd. Address before: 215021 International Science and Technology Park Phase III 8B, No. 1355 Jinjihu Avenue, Industrial Park, Suzhou City, Jiangsu Province Patentee before: SUZHOU MAXNET NETWORK SAFETY TECHNOLOGY Co.,Ltd. |