CN103793530A - Method, device and system for cleaning up business data regularly - Google Patents
Method, device and system for cleaning up business data regularly Download PDFInfo
- Publication number
- CN103793530A CN103793530A CN201410066606.5A CN201410066606A CN103793530A CN 103793530 A CN103793530 A CN 103793530A CN 201410066606 A CN201410066606 A CN 201410066606A CN 103793530 A CN103793530 A CN 103793530A
- Authority
- CN
- China
- Prior art keywords
- cleaning
- business datum
- clean
- record
- task database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for cleaning up business data regularly. The method includes the steps that corresponding clean-up records are established in a clean-up task database for each piece of business data in advance; the clean-up task database is scanned regularly, and the clean-up records meeting the clean-up conditions and the corresponding business data are deleted. The invention further discloses a device and system for cleaning up business data regularly. With the method, device and system, pressure of the main business database can be reduced.
Description
Technical field
The present invention relates to technical field of data storage, particularly a kind of methods, devices and systems of periodic cleaning business datum.
Background technology
Under a lot of scenes, there is demand will regularly clear up business datum.As in e-commerce system, requiring to exceed the order that certain hour do not pay the bill will cancel.The business datum that the general full table scan of prior art need to be cleared up, but in the situation that data volume is large, full table scan business datum can cause under efficiency, affects the shortcomings such as Service Database performance.
Summary of the invention
The object of the present invention is to provide a kind of methods, devices and systems of periodic cleaning business datum, can alleviate the pressure in service data master storehouse.
For achieving the above object, the invention provides a kind of method of periodic cleaning business datum, the method comprises:
Be that each business datum is set up corresponding cleaning record in clean-up task database in advance;
Periodic scanning clean-up task database, deletes the cleaning record and the corresponding business datum that meet cleaning condition.
For achieving the above object, the present invention also provides a kind of device of periodic cleaning business datum, is applied to cleaning subsystem, and this device comprises:
Clean-up task database, is used to the corresponding cleaning of each business datum storage record;
Scanning element, for periodic scanning clean-up task database;
Processing unit, deletes for the cleaning record to meeting cleaning condition and corresponding business datum.
For achieving the above object, the present invention also provides a kind of system of periodic cleaning business datum, and this system comprises service sub-system and cleaning subsystem;
Described service sub-system, in Service Database storage service data;
Described cleaning subsystem, sets up corresponding cleaning record for being each business datum in advance at clean-up task database; Periodic scanning clean-up task database, deletes the cleaning record and the corresponding business datum that meet cleaning condition.
The method of the periodic cleaning business datum that in sum, the embodiment of the present invention provides: in advance for each business datum is set up corresponding cleaning record in clean-up task database; Periodic scanning clean-up task database, deletes the cleaning record and the corresponding business datum that meet cleaning condition.Like this, scanning be clean-up task database, rather than service data master of the prior art storehouse, is the cleaning record of the breviary relevant to business datum due to what set up in clean-up task database, scan efficiency higher, can not affect the performance of Service Database simultaneously.For the order that exceedes certain hour and do not pay the bill, both deleted the cleaning record in clean-up task database, delete again the business datum in Service Database, thereby realize the object of cleaning business datum.
Accompanying drawing explanation
Fig. 1 is the schematic flow sheet of embodiment of the present invention periodic cleaning business datum method.
Fig. 2 is the structural representation that is applied to the periodic cleaning business datum device of said method in the specific embodiment of the invention.
Embodiment
For making object of the present invention, technical scheme and advantage clearer, referring to the accompanying drawing embodiment that develops simultaneously, scheme of the present invention is described in further detail.
Core concept of the present invention is: on the basis of prior art service sub-system, increasing cleaning subsystem, is that each business datum is set up corresponding cleaning record in clean-up task database by cleaning subsystem in advance; Periodic scanning clean-up task database, deletes the cleaning record and the corresponding business datum that meet cleaning condition.Use independently clean-up task database, can reduce the pressure of data scrubbing to service data master storehouse, keep lightweight, promote data scrubbing efficiency.
The embodiment of the present invention provides a kind of method of periodic cleaning business datum, and as shown in Figure 1, the method comprises its schematic flow sheet:
Particularly, the notification message that the embodiment of the present invention writes by receiving business datum, for each business datum is set up corresponding cleaning record in clean-up task database.
Here, the frequency of periodic scanning clean-up task database can be determined according to concrete application.
Preferably, the method for above-mentioned deletion is specially: first delete corresponding business datum in Service Database, then delete the cleaning record in clean-up task database.This is because if first delete cleaning record, then delete business datum, suppose that business datum is because the reasons such as system are deleted failed words, this business datum that should delete will be present in Service Database always, first deletes business datum so be preferably, then deletes cleaning record.Wherein, delete business datum, can realize by modes such as calling interfaces.
Further, the method also comprises: according to the message of received business datum state variation, judge whether this business datum needs cleaning, if do not needed, delete corresponding cleaning record.
Based on same inventive concept, the present invention also proposes a kind of device of periodic cleaning business datum, and referring to Fig. 2, Fig. 2 is the structural representation that is applied to the device of the periodic cleaning business datum of said method in the specific embodiment of the invention, this application of installation, in cleaning subsystem, comprising:
Clean-up task database 201, is used to the corresponding cleaning of each business datum storage record;
Further, described processing unit 203, also, for according to the message of received business datum state variation, judges whether this business datum needs cleaning, if do not needed, deletes corresponding cleaning record.
Further, described device further comprises:
Creating unit 204, for the notification message writing by receiving business datum, for each business datum is set up corresponding cleaning record in clean-up task database.
Preferably, the embodiment of the present invention also provides a kind of system of periodic cleaning business datum, and this system comprises service sub-system and cleaning subsystem;
Described service sub-system, in Service Database storage service data;
Described cleaning subsystem, sets up corresponding cleaning record for being each business datum in advance at clean-up task database; Periodic scanning clean-up task database, deletes the cleaning record and the corresponding business datum that meet cleaning condition.
Described service sub-system is in the time that stored business datum state changes, notice cleaning subsystem, for clearing up subsystem according to the message of received business datum state variation, judges whether this business datum needs cleaning, if do not needed, delete corresponding cleaning record.
Described service sub-system is in the time of every business datum of storage, and notice cleaning subsystem, for clearing up subsystem for each business datum is at the corresponding cleaning of clean-up task database foundation record.
For clearly demonstrating the present invention, to enumerate concrete scene below method of the present invention and device are elaborated, whole operation flow is as follows:
Suppose that the business datum of storing in the Service Database of service sub-system is order, order contents is:
Order number date that places an order is bought name of product quantity purchase payment status
User is every writes an above-mentioned order to Service Database, send a piece of news to cleaning subsystem by Active MQ, this message can be carried order number, the information such as date and payment status places an order, cleaning subsystem receives after this message, in the clean-up task database of self, write one and clear up accordingly record with this order, this cleaning record is by the content after order breviary, fairly simple, for example cleaning record can get off the Message Record carrying in Active MQ, and cleaning record comprises order number, date and payment status place an order;
The scanning element of cleaning subsystem, can be every 30 seconds scanning clean-up task databases, if payment status corresponding to order number in cleaning record, still be " arrearage " placing an order after 24 hours dates, exceed the order of not paying the bill for 24 hours and need in Service Database, cancel like this, therefore, such cleaning record meets cleaning condition, so, the processing unit of cleaning subsystem, can delete in Service Database, record corresponding order with this cleaning, then delete this cleaning record.
In this process, user may determine a certain order to pay the bill after consideration, that is to say, if business datum state changes in the Service Database of service sub-system, the payment status of for example some orders becomes " received payment ", send a piece of news to cleaning subsystem by Active MQ, this message can be carried order number, order status changes the time, order becomes the information such as the state of " received payment " from " arrearage ", cleaning subsystem receives after this message, judge that this business datum becomes valid data, do not need to clear up this business datum, , only delete in clean-up task database and clear up accordingly record with this order.
The method of periodic cleaning business datum of the present invention, Apparatus and system, can bring following benefit:
One, in periodic scanning clean-up task database, need the data of cleaning, and do not scan Service Database, thereby avoided clean-up task to affect Service Database performance;
Two, for the data that meet cleaning condition, clear up the corresponding data in Service Database and clean-up task database simultaneously, keep the lightweight of task list, improving performance.
The above, be only preferred embodiment of the present invention, is not intended to limit protection scope of the present invention.Within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (10)
1. a method for periodic cleaning business datum, the method comprises:
Be that each business datum is set up corresponding cleaning record in clean-up task database in advance;
Periodic scanning clean-up task database, deletes the cleaning record and the corresponding business datum that meet cleaning condition.
2. the method for claim 1, is characterized in that, the method for described deletion is specially: first delete corresponding business datum in Service Database, then delete the cleaning record in clean-up task database.
3. the method for claim 1, is characterized in that, the method further comprises:
According to the message of received business datum state variation, judge whether this business datum needs cleaning, if do not needed, delete corresponding cleaning record.
4. the method for claim 1, is characterized in that, the notification message writing by receiving business datum, for each business datum is set up corresponding cleaning record in clean-up task database.
5. a device for periodic cleaning business datum, is applied to cleaning subsystem, and this device comprises:
Clean-up task database, is used to the corresponding cleaning of each business datum storage record;
Scanning element, for periodic scanning clean-up task database;
Processing unit, deletes for the cleaning record to meeting cleaning condition and corresponding business datum.
6. device as claimed in claim 5, is characterized in that,
Described processing unit, also, for according to the message of received business datum state variation, judges whether this business datum needs cleaning, if do not needed, deletes corresponding cleaning record.
7. device as claimed in claim 5, is characterized in that, described device further comprises:
Creating unit, for the notification message writing by receiving business datum, for each business datum is set up corresponding cleaning record in clean-up task database.
8. a system for periodic cleaning business datum, is characterized in that, this system comprises service sub-system and cleaning subsystem;
Described service sub-system, in Service Database storage service data;
Described cleaning subsystem, sets up corresponding cleaning record for being each business datum in advance at clean-up task database; Periodic scanning clean-up task database, deletes the cleaning record and the corresponding business datum that meet cleaning condition.
9. system as claimed in claim 8, it is characterized in that, described service sub-system is in the time that stored business datum state changes, notice cleaning subsystem, for clearing up subsystem according to the message of received business datum state variation, judge whether this business datum needs cleaning, if do not needed, delete corresponding cleaning record.
10. system as claimed in claim 8, is characterized in that, described service sub-system is in the time of every business datum of storage, and notice cleaning subsystem, for clearing up subsystem for each business datum is at the corresponding cleaning of clean-up task database foundation record.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410066606.5A CN103793530A (en) | 2014-02-26 | 2014-02-26 | Method, device and system for cleaning up business data regularly |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410066606.5A CN103793530A (en) | 2014-02-26 | 2014-02-26 | Method, device and system for cleaning up business data regularly |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103793530A true CN103793530A (en) | 2014-05-14 |
Family
ID=50669196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410066606.5A Pending CN103793530A (en) | 2014-02-26 | 2014-02-26 | Method, device and system for cleaning up business data regularly |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103793530A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104036001A (en) * | 2014-06-13 | 2014-09-10 | 上海新炬网络技术有限公司 | Dynamic hotlist priority scheduling based quick data cleaning method |
| CN105975600A (en) * | 2016-05-11 | 2016-09-28 | 乐视控股（北京）有限公司 | Big data processing task optimization method and device |
| CN105975577A (en) * | 2016-05-04 | 2016-09-28 | 乐视控股（北京）有限公司 | Data optimization method and device in big data processing |
| CN106528840A (en) * | 2016-11-11 | 2017-03-22 | 中国银行股份有限公司 | Service data clearing method and system based on banking system |
| CN107832230A (en) * | 2017-12-04 | 2018-03-23 | 中国工商银行股份有限公司 | Method of testing, equipment and system based on data tuning |
| CN110795243A (en) * | 2019-10-21 | 2020-02-14 | Oppo（重庆）智能科技有限公司 | Application processing method and related device |
| US10783125B2 (en) | 2016-11-08 | 2020-09-22 | International Business Machines Corporation | Automatic data purging in a database management system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101344893A (en) * | 2008-07-17 | 2009-01-14 | 中兴通讯股份有限公司 | History data access method and apparatus |
| CN102915382A (en) * | 2012-11-21 | 2013-02-06 | 亚信联创科技（中国）有限公司 | Method and device for carrying out data query on database based on indexes |
| CN103064833A (en) * | 2011-10-18 | 2013-04-24 | 阿里巴巴集团控股有限公司 | Method of cleaning database history data and system thereof |
-
2014
- 2014-02-26 CN CN201410066606.5A patent/CN103793530A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101344893A (en) * | 2008-07-17 | 2009-01-14 | 中兴通讯股份有限公司 | History data access method and apparatus |
| CN103064833A (en) * | 2011-10-18 | 2013-04-24 | 阿里巴巴集团控股有限公司 | Method of cleaning database history data and system thereof |
| CN102915382A (en) * | 2012-11-21 | 2013-02-06 | 亚信联创科技（中国）有限公司 | Method and device for carrying out data query on database based on indexes |
Non-Patent Citations (2)
| Title |
|---|
| 邹小三: "如何自动给数据库减肥", 《华南金融电脑》 * |
| 郭鹏: "《Cassandra实战》", 30 June 2011 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104036001A (en) * | 2014-06-13 | 2014-09-10 | 上海新炬网络技术有限公司 | Dynamic hotlist priority scheduling based quick data cleaning method |
| CN104036001B (en) * | 2014-06-13 | 2017-05-10 | 上海新炬网络信息技术有限公司 | Dynamic hotlist priority scheduling based quick data cleaning method |
| CN105975577A (en) * | 2016-05-04 | 2016-09-28 | 乐视控股（北京）有限公司 | Data optimization method and device in big data processing |
| WO2017190469A1 (en) * | 2016-05-04 | 2017-11-09 | 乐视控股（北京）有限公司 | Data optimisation method and apparatus in big data processing |
| CN105975600A (en) * | 2016-05-11 | 2016-09-28 | 乐视控股（北京）有限公司 | Big data processing task optimization method and device |
| US10783125B2 (en) | 2016-11-08 | 2020-09-22 | International Business Machines Corporation | Automatic data purging in a database management system |
| CN106528840A (en) * | 2016-11-11 | 2017-03-22 | 中国银行股份有限公司 | Service data clearing method and system based on banking system |
| CN107832230A (en) * | 2017-12-04 | 2018-03-23 | 中国工商银行股份有限公司 | Method of testing, equipment and system based on data tuning |
| CN110795243A (en) * | 2019-10-21 | 2020-02-14 | Oppo（重庆）智能科技有限公司 | Application processing method and related device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103793530A (en) | Method, device and system for cleaning up business data regularly | |
| CN102413156B (en) | Distributed data storage system and method | |
| CN111027984B (en) | Service order processing method, system, electronic equipment and computer storage medium | |
| US8078206B2 (en) | Mobile phone and method for managing short messages of the mobile phone | |
| CN101753936B (en) | Configuration method of storage space and video server for video monitoring system | |
| CN102486785A (en) | Data synchronization method, device and system | |
| CN114003439B (en) | Data backup method, device, equipment and storage medium | |
| CN111581017B (en) | Backup and recovery system and method for modern application | |
| CN106874145A (en) | A kind of asynchronous data backup method based on message queue | |
| CN101957849A (en) | Static statement generation method and device | |
| CN102299947A (en) | Touch digital newspaper-reading high-definition interactive advertising system and data updating method thereof | |
| CN101937378B (en) | Method for carrying out back-up protection on data of storage equipment and computer system | |
| CN110532066A (en) | A kind of realization method and system of transaction consistency | |
| CN101142573A (en) | System and method for global data synchronization | |
| CN102024010A (en) | Data processing system and data processing method | |
| CN114356654A (en) | Backup system, backup method, backup device, computer device, and storage medium | |
| CN109743179A (en) | A kind of electronics license signs and issues method on demand | |
| US9317502B2 (en) | System and method for monitoring items | |
| CN102419721A (en) | Method for scatter storing, centralized backing up and restoring metro card consumption data | |
| CN106302625A (en) | Data-updating method, device and related system | |
| CN102521078B (en) | Educational administration resource grade backup method based on third party cloud | |
| CN102195936A (en) | Method and system for storing multimedia file and method and system for reading multimedia file | |
| CN1996886A (en) | Cleaning method of redundant data of the IPTV system | |
| CN112818166B (en) | Video information query method and device, electronic equipment and storage medium | |
| CN111274255B (en) | Service data monitoring method and system, monitoring architecture, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140514 |
|
| RJ01 | Rejection of invention patent application after publication |