CN108573036A - Method for cleaning and its system, storage medium, the electronic equipment of historical data - Google Patents
Method for cleaning and its system, storage medium, the electronic equipment of historical data Download PDFInfo
- Publication number
- CN108573036A CN108573036A CN201810283195.3A CN201810283195A CN108573036A CN 108573036 A CN108573036 A CN 108573036A CN 201810283195 A CN201810283195 A CN 201810283195A CN 108573036 A CN108573036 A CN 108573036A
- Authority
- CN
- China
- Prior art keywords
- historical data
- clearance
- data
- thread
- cleaning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Present invention is disclosed the method for cleaning of historical data and its system, storage medium, electronic equipments.The method for cleaning of the historical data includes the following steps：Read data purge rule；By in data purge rule specified data library, obtaining historical data for clearance, wherein database includes at least one set of historical data queue for clearance, and each group of historical data queue for clearance includes multiple historical data units for clearance；According to data purge rule, the quantity of the thread needed for the historical data for clearance that cleaning is got is calculated；According to the required number of threads being calculated, first thread and the second thread be at least configured with to each group of historical data queue for clearance, first thread and the second thread parallel and meanwhile respectively by be located in one group of historical data queue for clearance head of the queue first historical data unit for clearance and positioned at tail of the queue the last one historical data unit for clearance successively opposite carry out data scrubbing.
Description
Technical field
The present invention relates to a kind of method for cleaning of historical data and its system, storage medium, electronic equipments.
Background technology
As the degree of automation is higher and higher, computer system scale is increasingly huge, and number of users is increasing, computer
System in the process of running, inevitably will produce a large amount of intermediate data.Wherein, many data persist in system
Database in.Increase with the run time of system, more and more data may influence whether system normal table
Operation, for example, the increase of data volume can produce tremendous influence to data effectiveness of retrieval.These data are according to different use
On the way, it may be necessary to be filed or be deleted.To reduce since data scale increases, caused system performance and stability
The harmful effect of decline.Therefore, it is necessary to certain cleaning is carried out to historical data.
Mostly use the cleaning that single thread carries out historical data greatly in the method for cleaning of existing historical data.Certainly it also deposits
In the method that some clear up historical data using dual-thread (using two threads).For example, referring to Fig. 1, show
Gone out in a kind of method for cleaning of historical data of the prior art using two one group of thread pair historical data queues for clearance into
The schematic diagram of row scale removal process.As shown in Figure 1, currently, during being cleared up for one group of historical data queue for clearance,
Since two thread L10 and L20 are by the historical data list for clearance positioned at group historical data queue head of the queue one end for clearance
Member starts, successively to the progress data scrubbing of tail of the queue direction, i.e. thread L10 clears up historical data cells D 10 for clearance；Line
Journey L20 clears up historical data cells D 20 for clearance.But it for data purge method shown in FIG. 1, is going through
During the cleaning (deletion) of history data, these historical datas also keep externally providing service, if two threads
(thread L10 and thread L20) all reads physically neighbouring historical data unit (thread i.e. as shown in Figure 1 for clearance
L10 clears up historical data cells D 10 for clearance, thread L20 clears up historical data cells D 20 for clearance) when,
The problem of will produce the probability that data are locked higher (especially page lock), in turn, causes the performance of data purge to decline.
Invention content
For the defects in the prior art, the object of the present invention is to provide a kind of method for cleaning of historical data and its it is
System, storage medium, electronic equipment can achieve the purpose that safely and efficiently to delete data, and reduce the number to system as far as possible
Influence of production is run according to library.
A kind of method for cleaning of historical data, the method for cleaning of the historical data are provided according to an aspect of the present invention
Include the following steps：S10：Read data purge rule；S20：By the data purge rule specified data library
In, obtain historical data for clearance, wherein database includes at least one set of historical data queue for clearance, described in each group
Historical data queue for clearance includes multiple historical data units for clearance；S30：It is regular according to the data purge,
Calculate the quantity of the thread needed for the historical data for clearance that cleaning is got, wherein the quantity of the thread is more than or equal to 2；
S40：According to the required number of threads being calculated, corresponding thread is configured to historical data queue for clearance described in each group
Carry out data scrubbing, wherein at least it is configured with first thread and the second thread, it is the first thread and the second thread parallel, same
When respectively by one group of historical data queue for clearance be located at head of the queue first historical data unit for clearance and be located at tail of the queue
The last one historical data unit for clearance start opposite to carry out data scrubbing successively.
Preferably, further include following steps after the completion of the step S40：S50：By going through after the completion of the step S40
History data scrubbing result is preserved to a storage element；Further include as follows before the step S20 after the completion of step S10
Step：S601：Detect the data scrubbing result of previous group historical data queue for clearance；S602：Judge whether cleaning succeeds, if
It is to then follow the steps S603, if it is not, thening follow the steps S604；S603：Preserve the cleaning of previous group historical data queue for clearance
Quantity, and be added in the summation of data purge quantity on the same day；S604：The reason of recording data scrubbing failure, and sentence
Whether the disconnected data purge rule currently read is reasonable.
Preferably, the judging result in step S602 is no, then further includes following steps before the step S604：
Suspend the cleaning of historical data.
Preferably, further include the estimated cleaning amount of a current database in the data purge rule, in the step
Further include following steps after the completion of rapid S40：Judge whether the cleaning amount of historical data is less than the estimated clear of the current database
Reason amount；If so, data purge of the pause to the database
Preferably, third thread and the 4th thread are also configured in the step S40, wherein the first thread, second
The cleaning amount of the historical data of thread, third thread and the 4th thread is equal；The first thread is by one group of history number for clearance
According to being located at the historical data unit for clearance of head of the queue first in queue to the last one historical data list for clearance positioned at tail of the queue
First direction is cleared up；The third thread is by apart from the described first N number of history number for clearance of historical data unit for clearance
It is cleared up to the last one described historical data cell orientation for clearance according to a historical data unit for clearance of unit, N
For natural number；Second thread is from the last one described historical data unit for clearance to described first history number for clearance
It is cleared up according to cell orientation；4th thread is N number of for clearance by the last one described historical data unit for clearance of distance
One historical data unit for clearance of historical data unit carries out clear to described first historical data cell orientation for clearance
Reason.
Preferably, the historical data for clearance described in a historical data control centre timer-triggered scheduler in the step S20.
Preferably, each data scrubbing amount of each thread is 50 to 100, and each thread completes one
Suspend 20 to 50 milliseconds after secondary data scrubbing.
According to another aspect of the present invention, a kind of cleaning system of historical data is also provided, the historical data it is clear
Reason system includes：Rule reading module, for reading data purge rule；Data acquisition module, for by the history
In data scrubbing rule specified data library, historical data for clearance is obtained, wherein database includes at least one set of for clearance
Historical data queue, historical data queue for clearance described in each group includes multiple historical data units for clearance；Calculate mould
Block, for according to data purge rule, calculating the number of the thread needed for the historical data for clearance that cleaning is got
Amount, wherein the quantity of the thread is more than or equal to 2；Data scrubbing module, for according to the required Thread Count being calculated
Amount configures corresponding thread to historical data queue for clearance described in each group and carries out data scrubbing, wherein is at least configured with the
One thread and the second thread, the first thread and the second thread parallel while respectively by one group of historical data queue for clearance
In be located at head of the queue first historical data unit for clearance and positioned at tail of the queue the last one historical data unit for clearance according to
Secondary beginning is opposite to carry out data scrubbing.
According to a further aspect of the invention, a kind of computer readable storage medium is also provided, computer is stored thereon with
The step of program, which realizes the method for cleaning of above-mentioned historical data when being executed by processor.
According to a further aspect of the invention, a kind of electronic equipment is also provided, including：Processor；And memory, it is used for
Store the executable instruction of the processor；Wherein, the processor is configured to execute via the executable instruction is executed
The step of method for cleaning of above-mentioned historical data.
Compared with the prior art, the method for cleaning of historical data provided in an embodiment of the present invention and system, storage medium,
In electronic equipment by reading data purge rule after, obtain historical data for clearance, calculate that cleaning gets wait for it is clear
The quantity of the thread needed for historical data is managed, in turn, corresponding thread is configured to historical data queue for clearance described in each group
Carry out data scrubbing, wherein be at least configured with first thread and the second thread, first thread and the second thread parallel while right
Data scrubbing is proceeded by by the historical data unit for clearance at both ends in one group of historical data queue for clearance successively.The history
The method for cleaning of data can deal with mass data cleaning, be not less than ten million or even hundred million for the maximum cleaning amount of each database
Grade, and while clearing up historical data, the influence to system is reduced, it avoids generating apparent shadow to the runnability of system
It rings.
Description of the drawings
Upon reading the detailed description of non-limiting embodiments with reference to the following drawings, other feature of the invention,
Objects and advantages will become more apparent upon：
Fig. 1 be the prior art a kind of historical data method for cleaning in use two one group of thread pair history numbers for clearance
The schematic diagram of scale removal process is carried out according to queue；
Fig. 2 is the flow chart of the method for cleaning of the historical data of one embodiment of the present of invention；
Fig. 3 is two threads pair, one group of history for clearance in the method for cleaning of the historical data of one embodiment of the present of invention
Data queue carries out the schematic diagram of scale removal process；
Two one group of threads pair are for clearance in method for cleaning of the Fig. 4 for the historical data of an alternative embodiment of the invention goes through
History data queue carries out the schematic diagram of scale removal process；
Fig. 5 is the module composition schematic diagram of the cleaning system of the historical data of one embodiment of the present of invention；
Fig. 6 is that the present invention discloses a kind of computer readable storage medium schematic diagram in exemplary embodiment；And
Fig. 7 is that the present invention discloses a kind of electronic equipment schematic diagram in exemplary embodiment.
Specific implementation mode
Example embodiment is described more fully with reference to the drawings.However, example embodiment can be with a variety of shapes
Formula is implemented, and is not understood as limited to embodiment set forth herein；On the contrary, thesing embodiments are provided so that the present invention will
Fully and completely, and by the design of example embodiment comprehensively it is communicated to those skilled in the art.It is identical attached in figure
Icon note indicates same or similar structure, thus will omit repetition thereof.
Described feature, structure or characteristic can be incorporated in one or more embodiments in any suitable manner
In.In the following description, many details are provided to fully understand embodiments of the present invention to provide.However,
One of ordinary skill in the art would recognize that without one or more in specific detail, or using other methods, constituent element, material
Material etc., can also put into practice technical scheme of the present invention.In some cases, be not shown in detail or describe known features, material or
Person operates to avoid the fuzzy present invention.
In addition, attached drawing is only the schematic illustrations of the disclosure, it is not necessarily drawn to scale.Identical attached drawing mark in figure
Note indicates same or similar part, thus will omit repetition thereof.Some block diagrams shown in attached drawing are work(
Energy entity, not necessarily must be corresponding with physically or logically independent entity.Software form may be used to realize these work(
Energy entity, or these functional entitys are realized in one or more hardware modules or integrated circuit, or at heterogeneous networks and/or place
These functional entitys are realized in reason device device and/or microcontroller device.
Purport design according to the present invention, a kind of method for cleaning of historical data of the invention include the following steps：It reads
Data purge rule；By in the data purge rule specified data library, obtaining historical data for clearance,
In, database includes at least one set of historical data queue for clearance, and historical data queue for clearance described in each group includes
Multiple historical data units for clearance；According to data purge rule, the history number for clearance that cleaning is got is calculated
According to the quantity of required thread, wherein the quantity of the thread is more than or equal to 2；According to the required number of threads being calculated,
Corresponding thread is configured to historical data queue for clearance described in each group and carries out data scrubbing, wherein is at least configured with first
Thread and the second thread, the first thread and the second thread parallel while respectively by one group of historical data queue for clearance
First historical data unit for clearance positioned at head of the queue and the last one historical data unit for clearance positioned at tail of the queue are successively
Start opposite progress data scrubbing.
The technology contents of the present invention are described further with reference to the accompanying drawings and examples.
Fig. 2 is referred to, it illustrates the flow charts of the method for cleaning of the historical data of one embodiment of the present of invention.Specifically
For, as shown in Fig. 2, in an embodiment of the present invention, the method for cleaning of the historical data includes the following steps：
Step S10：Read data purge rule.Specifically, in embodiments of the present invention, user will need to carry out
The type of the historical data of cleaning, database default in data purge rule.The data purge rule can protect
There are in an individual database, or can also be stored in can be in the arbitrary system of persistent storage.In the reality of the present invention
It applies in example, above-mentioned data purge rule is arranged in the mode for typically setting up a data purge rule list.It can be found in one
One data purge rule list of lower citing.
Table 1
In table 1 above, each field is defined as follows：
DatabaseName：Configuration needs the instance name for the database cleared up；
TableName：Configuration needs the title for the table being cleaned；
CleanUpGroup：Conclusion is grouped to the data cleared up of needs, group number is identical be placed in same batch by
Sequencing executes；
NumOfThread：The maximum sub-line journey quantity for needing to open is executed every time；
WhereClause：The deletion rule condition (SQL) for the tables of data being cleaned；
PrimaryKey：The major key for the tables of data being cleaned, system are only supported to delete according to major key；
MaxProcessLimit：Each dominant record quantity that executes batch (planned dispatching each time) and can delete；
ThreadDeleteLimit：Each independent sub-line journey is during deletion, while the maximum quantity deleted；
OrderByClause：Sort criteria.
It should be noted that in an embodiment of the present invention, being carried out primarily directed to the data in relevant database clear
Therefore reason, is advised to reduce the problem that the strap in database between the table and table of data comes in history design data scrubbing
Then during table, do not allow to establish external key.In addition, above-mentioned table 1 is only schematically illustrated, it is not limited thereto.
Step S20：By in the data purge rule specified data library, obtaining historical data for clearance.Wherein,
Database includes at least one set of historical data queue for clearance, and historical data queue for clearance described in each group includes multiple
Historical data unit for clearance.In an embodiment of the present invention, by a historical data control centre in the step S20
(JobAgent) historical data for clearance described in timer-triggered scheduler.Wherein, historical data control centre (JobAgent) is a meter
The scheduling engine for the task of drawing, main function is (such as can be according to set time or fixed according to the demand of user
Every modes such as times) execute scheduled historical data operation plan.
Further, it when obtaining historical data for clearance every time, needs according to user in data purge rule
The scheduling parameter of middle setting is obtained.For above-mentioned table 1, specific format is：[DatabaseName]_
[GroupName].This means that providing database instance title and the group name for currently needing to clear up.By reading historical data
Cleaning rule, you can acquisition currently needs historical data for clearance.
Step S30：According to data purge rule, calculate needed for the historical data for clearance that cleaning is got
The quantity of thread.In embodiments of the present invention, the quantity of the thread is more than or equal to 2.It should be noted that thread is in the present invention
Refer to that the program of data purge executes the minimum unit of stream in embodiment.
Step S40：According to the required number of threads being calculated, historical data queue for clearance described in each group is matched
It sets corresponding thread and carries out data scrubbing.Wherein, first thread and the second thread, the first thread and second are at least configured with
Thread parallel at the same respectively by one group of historical data queue for clearance be located at head of the queue first historical data unit for clearance
Start opposite progress data scrubbing successively with the last one historical data unit for clearance positioned at tail of the queue.In order to reduce history number
According to scale removal process to database generate performance on influence, in a preferred embodiment of the invention, each number of per thread
It it is 50 to 100 according to cleaning amount, and per thread completes 20 to 50 milliseconds of pause after a data scrubbing.
Further, in embodiments of the present invention, it is mainly optimized to being directed to dual-thread.Fig. 3 is referred to, is shown
In the method for cleaning of the historical data of one embodiment of the present of invention two one group of thread pair historical data queues for clearance into
The schematic diagram of row scale removal process.It is illustrated by taking one group 2000 historical data queues for clearance as an example in Fig. 3.Such as Fig. 3 institutes
Show, the historical data queue for clearance of the group 2000 includes 20 historical data units for clearance, each history for clearance
Data cell includes 100 historical datas for clearance.After getting group historical data queue for clearance, first thread L1 by
First historical data cells D 1 for clearance of group historical data queue for clearance is to the last one historical data list for clearance
First directions D2 are cleared up；Second thread L2 by group historical data queue for clearance the last one historical data list for clearance
First D2 is cleared up to first 1 direction of historical data cells D for clearance.Manner of cleaning up in the embodiment of the present invention compares Fig. 1
Shown in for manner of cleaning up, can be higher (especially page lock) to avoid the probability that data are locked is led to the problem of, improve history
The pick-up performance of data.
Further, Fig. 4 is referred to, it illustrates in the method for cleaning of the historical data of an alternative embodiment of the invention
The historical data queue for clearance of two one group of threads pair carries out the schematic diagram of scale removal process.Exist with difference shown in above-mentioned Fig. 3
In being also configured with third thread and the 4th thread in above-mentioned steps S40.Wherein, the first thread, the second thread, third
The cleaning amount of the historical data of thread and the 4th thread is equal.As shown in figure 4, first thread L1 is by group historical data for clearance
First historical data cells D 1 for clearance of queue is cleared up to the last one 2 direction of historical data cells D for clearance.The
Three thread L3 are by the historical data cells D 1 for clearance of distance first, the history number for clearance of N number of historical data unit for clearance
It is cleared up to the last one 2 direction of historical data cells D for clearance according to cells D 3, wherein N is natural number.Second thread L2
From the last one historical data cells D 2 for clearance of group historical data queue for clearance to first historical data for clearance
1 direction of cells D is cleared up.4th thread L4 is by apart from the last one historical data cells D 2 for clearance, N number of history for clearance
The historical data cells D 4 for clearance of data cell is cleared up to first 1 direction of historical data cells D for clearance.It needs
Bright, the cleaning quantity of the historical data unit for clearance of above-mentioned first thread L1 and the second thread L2 is N+1, third line
The quantity of journey L3 and the historical data unit for clearance of the 4th thread L4 cleanings is also N+1, in other words, historical data list for clearance
The quantity of historical data unit for clearance between first D3 and historical data cells D 4 for clearance is 2N.Aforementioned four thread carries out
The mode of data purge can more effectively accelerate the cleaning efficiency of historical data.
Further, in one embodiment of the invention, further include following steps after the completion of the step S40：
Step S50：Data purge result after the completion of the step S40 is preserved to a storage element.It is specific next
It says, after the completion of being executed each time to data purge, when the cleaning result of secondary historical data can be cached in a storage
In unit (Redis), the term of validity of storage can be the same day.
In turn, further include following steps before the step S20 after the completion of step S10：
Step S601：Detect the data scrubbing result of previous group historical data queue for clearance.Result after detection can be with
It is formed in following table 2.
Table 2
Step S602：Judge whether cleaning succeeds.If so, S603 is thened follow the steps, if it is not, thening follow the steps S604.
Step S603：The cleaning quantity of previous group historical data queue for clearance is preserved, and is added to the history number on the same day
In summation according to cleaning quantity.
Step S604：The reason of recording data scrubbing failure, and whether judge the data purge rule currently read
Rationally.
Further, the judging result in step S602 is no, then further includes walking as follows before the step S604
Suddenly：Suspend the cleaning of historical data.Specifically, if the data scrubbing failure of previous group historical data queue for clearance, temporarily
Stop the cleaning of historical data.
The reason of by log history data scrubbing success and failure, it may be convenient to the data purge on the same day
Entire implementation procedure is analyzed and is counted, and developer and database maintenance personnel is helped to find problem.For example, if encountering
The frequently situation of time-out, whether developer needs to analyze current deletion plan reasonable etc..
Further, further include the estimated cleaning amount of a current database in the data purge rule, described
Further include following steps after the completion of step S40：
Whether the cleaning amount of historical data is less than the estimated cleaning amount of the current database in judgment step S40.
If so, data purge of the pause to the database.
Specifically, when judge in the scale removal process of historical data historical data cleaning amount be less than current database it is pre-
When counting cleaning amount, for example, the estimated cleaning amount of current database is to delete 10000 datas still only to delete in upper step S40
In addition to 9000 datas；So, pause can wait until that the second talent is extensive to the data purge of the database on the day of system
It is multiple.
Further, in an embodiment of the present invention, the read and write abruption of database obtains history number for clearance from library is read
According to, by read library access, reduce the performance pressures to writing library.In reading process, major key is only obtained, reduces data transmission
Burden.
Further, the present invention also provides a kind of cleaning systems of historical data, for realizing above-mentioned historical data
Method for cleaning.Specifically, Fig. 3 is referred to, it illustrates the moulds of the cleaning system of the historical data of one embodiment of invention
Block constitutes schematic diagram.The cleaning system of the historical data includes at least rule reading module 1, data acquisition module 2, calculates mould
Block 3 and data scrubbing module 4.
Specifically, rule reading module 1 is for reading data purge rule.
Data acquisition module 2 is used for by the data purge rule specified data library, obtaining history for clearance
Data.Wherein, database includes at least one set of historical data queue for clearance, historical data queue for clearance described in each group
Include multiple historical data units for clearance.
Computing module 3 is used to, according to data purge rule, calculate the group history for clearance that cleaning is got
The quantity of thread needed for data queue.Wherein, the quantity of the thread is more than or equal to 2.
Data scrubbing module 4 is used for according to the required number of threads being calculated, to history for clearance described in each group
Data queue configures corresponding thread and carries out data scrubbing.Wherein, first thread and the second thread are at least configured with.Described first
It thread and the second thread parallel while for clearance being gone through by being located at first of head of the queue in one group of historical data queue for clearance respectively
History data cell and start opposite to carry out data scrubbing successively positioned at the last one historical data unit for clearance of tail of the queue.
In an exemplary embodiment of the disclosure, a kind of computer readable storage medium is additionally provided, meter is stored thereon with
The clear of historical data described in any one above-mentioned embodiment may be implemented in calculation machine program, the program when being executed by such as processor
Each step that control system and telephone traffic system execute in reason method.In some possible embodiments, of the invention each
Aspect is also implemented as a kind of form of program product comprising program code, when described program product on the terminal device
When operation, said program code is for making the terminal device execute the cleaning of the above-mentioned historical data shown in Fig. 2 of this specification
Described in method part according to various illustrative embodiments of the invention the step of.
Refering to what is shown in Fig. 4, describing the program product for realizing the above method according to the embodiment of the present invention
300, portable compact disc read only memory (CD-ROM) may be used and include program code, and can in terminal device,
Such as it is run on PC.However, the program product of the present invention is without being limited thereto, in this document, readable storage medium storing program for executing can be with
To be any include or the tangible medium of storage program, the program can be commanded execution system, device either device use or
It is in connection.
The arbitrary combination of one or more readable mediums may be used in described program product.Readable medium can be readable letter
Number medium or readable storage medium storing program for executing.Readable storage medium storing program for executing for example can be but be not limited to electricity, magnetic, optical, electromagnetic, infrared ray or
System, device or the device of semiconductor, or the arbitrary above combination.The more specific example of readable storage medium storing program for executing is (non exhaustive
List) include：It is electrical connection, portable disc, hard disk, random access memory (RAM) with one or more conducting wires, read-only
Memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc read only memory
(CD-ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.
The computer readable storage medium may include the data letter propagated in a base band or as a carrier wave part
Number, wherein carrying readable program code.Diversified forms, including but not limited to electromagnetism may be used in the data-signal of this propagation
Signal, optical signal or above-mentioned any appropriate combination.Readable storage medium storing program for executing can also be any other than readable storage medium storing program for executing
Readable medium, which can send, propagate either transmission for being used by instruction execution system, device or device or
Person's program in connection.The program code for including on readable storage medium storing program for executing can transmit with any suitable medium, packet
Include but be not limited to wireless, wired, optical cable, RF etc. or above-mentioned any appropriate combination.
It can be write with any combination of one or more programming languages for executing the program that operates of the present invention
Code, described program design language include object oriented program language-Java, C++ etc., further include conventional
Procedural programming language-such as " C " language or similar programming language.Program code can be fully in user
It executes on computing device, partly execute on a user device, being executed as an independent software package, partly in user's calculating
Upper side point is executed or is executed in remote computing device or server completely on a remote computing.It is being related to far
In the situation of journey computing device, remote computing device can pass through the network of any kind, including LAN (LAN) or wide area network
(WAN), it is connected to user calculating equipment, or, it may be connected to external computing device (such as utilize ISP
To be connected by internet).
In an exemplary embodiment of the disclosure, a kind of electronic equipment is also provided, which may include processor,
And the memory of the executable instruction for storing the processor.Wherein, the processor is configured to via described in execution
Executable instruction executes control system and traffic system in the method for cleaning of historical data described in any one above-mentioned embodiment
Each step that system executes.
Person of ordinary skill in the field it is understood that various aspects of the invention can be implemented as system, method or
Program product.Therefore, various aspects of the invention can be embodied in the following forms, i.e.,：It is complete hardware embodiment, complete
The embodiment combined in terms of full Software Implementation (including firmware, microcode etc.) or hardware and software, can unite here
Referred to as circuit, " module " or " system ".
The electronic equipment 600 of this embodiment according to the present invention is described referring to Fig. 5.The electronics that Fig. 5 is shown
Equipment 600 is only an example, should not bring any restrictions to the function and use scope of the embodiment of the present invention.
As shown in figure 5, electronic equipment 600 is showed in the form of universal computing device.The component of electronic equipment 600 can wrap
It includes but is not limited to：At least one processing unit 610, at least one storage unit 620, (including the storage of connection different system component
Unit 620 and processing unit 610) bus 630, display unit 640 etc..
Wherein, the storage unit has program stored therein code, and said program code can be held by the processing unit 610
Row so that the processing unit 610 execute the above-mentioned historical data of this specification method for cleaning part described according to this hair
The step of bright various illustrative embodiments.For example, the processing unit 610 can execute as shown in Figure 2 by control system
Each step that system and telephone traffic system are completed.
The storage unit 620 may include the readable medium of volatile memory cell form, such as random access memory
Unit (RAM) 6201 and/or cache memory unit 6202 can further include read-only memory unit (ROM) 6203.
The storage unit 620 can also include program/practicality work with one group of (at least one) program module 6205
Tool 6204, such program module 6205 include but not limited to：Operating system, one or more application program, other programs
Module and program data may include the realization of network environment in each or certain combination in these examples.
Bus 630 can be to indicate one or more in a few class bus structures, including storage unit bus or storage
Cell controller, peripheral bus, graphics acceleration port, processing unit use the arbitrary bus structures in a variety of bus structures
Local bus.
Electronic equipment 600 can also be with one or more external equipments 700 (such as keyboard, sensing equipment, bluetooth equipment
Deng) communication, can also be enabled a user to one or more equipment interact with the electronic equipment 600 communicate, and/or with make
Any equipment that the electronic equipment 600 can be communicated with one or more of the other computing device (such as router, modulation /demodulation
Device etc.) communication.This communication can be carried out by input/output (I/O) interface 650.Also, electronic equipment 600 can be with
By network adapter 660 and one or more network (such as LAN (LAN), wide area network (WAN) and/or public network,
Such as internet) communication.Network adapter 660 can be communicated by bus 630 with other modules of electronic equipment 600.It should
Understand, although not shown in the drawings, other hardware and/or software module can be used in conjunction with electronic equipment 600, including but it is unlimited
In：Microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive and number
According to backup storage system etc..
Through the above description of the embodiments, those skilled in the art is it can be readily appreciated that example described herein is implemented
Mode can also be realized by software realization in such a way that software is in conjunction with necessary hardware.Therefore, according to the disclosure
The technical solution of embodiment can be expressed in the form of software products, the software product can be stored in one it is non-volatile
Property storage medium (can be CD-ROM, USB flash disk, mobile hard disk etc.) in or network on, including some instructions are so that a calculating
Equipment (can be personal computer, server or network equipment etc.) executes the above-mentioned history according to disclosure embodiment
The method for cleaning of data.
In conclusion method for cleaning and system, storage medium, the electronics of historical data provided in an embodiment of the present invention are set
In standby by reading data purge rule after, obtain historical data for clearance, calculate the history for clearance that cleaning is got
It is clear to configure corresponding thread progress data in turn to historical data for clearance described in each group for the quantity of thread needed for data
Reason, wherein be at least configured with first thread and the second thread, first thread and the second thread parallel while clear to being waited for by one group
The historical data unit for clearance at both ends proceeds by data scrubbing successively in reason historical data.The method for cleaning of the historical data
Mass data cleaning can be dealt with, ten million or even hundred million grades are not less than for the maximum cleaning amount of each database, and clearing up
While historical data, the influence to system is reduced, avoids generating apparent influence to the runnability of system.
Although the present invention is disclosed as above with alternative embodiment, it is not limited to the present invention.Belonging to the present invention
Those skilled in the art, without departing from the spirit and scope of the present invention, when various change and modification can be made.Therefore,
Protection scope of the present invention is subject to the range defined depending on claims.
Claims (10)
1. a kind of method for cleaning of historical data, which is characterized in that the method for cleaning of the historical data includes the following steps：
S10：Read data purge rule；
S20：By in the data purge rule specified data library, obtaining historical data for clearance, wherein in database
Including at least one set of historical data queue for clearance, historical data queue for clearance described in each group includes multiple for clearance goes through
History data cell；
S30：According to data purge rule, the number of the thread needed for the historical data for clearance that cleaning is got is calculated
Amount, wherein the quantity of the thread is more than or equal to 2；
S40：According to the required number of threads being calculated, historical data queue for clearance described in each group is configured corresponding
Thread carries out data scrubbing, wherein is at least configured with first thread and the second thread, the first thread and the second thread are simultaneously
It goes while respectively by being located at first historical data unit for clearance of head of the queue in one group of historical data queue for clearance and being located at
The last one of tail of the queue historical data unit for clearance starts opposite progress data scrubbing successively.
2. the method for cleaning of historical data according to claim 1, which is characterized in that after the completion of the step S40 also
Include the following steps：
S50：Data purge result after the completion of the step S40 is preserved to a storage element；
Further include following steps before the step S20 after the completion of step S10：
S601：Detect the data scrubbing result of previous group historical data queue for clearance；
S602：Judge whether cleaning succeeds, if so, S603 is thened follow the steps, if it is not, thening follow the steps S604；
S603：The cleaning quantity of previous group historical data queue for clearance is preserved, and is added to the data purge number on the same day
In the summation of amount；
S604：The reason of recording data scrubbing failure, and judge whether the data purge rule currently read is reasonable.
3. the method for cleaning of historical data according to claim 2, which is characterized in that the judging result in step S602
It is no, then further includes following steps before the step S604：
Suspend the cleaning of historical data.
4. the method for cleaning of historical data according to claim 2, which is characterized in that in the data purge rule
The estimated cleaning amount for further including a current database further includes following steps after the completion of the step S40：
Judge whether the cleaning amount of historical data is less than the estimated cleaning amount of the current database；
If so, data purge of the pause to the database.
5. the method for cleaning of historical data according to claim 1, which is characterized in that be also configured in the step S40
Third thread and the 4th thread, wherein the first thread, the historical data of the second thread, third thread and the 4th thread
Cleaning amount is equal；
The first thread from one group of historical data queue for clearance be located at the historical data unit for clearance of head of the queue first to
The last one historical data cell orientation for clearance positioned at tail of the queue is cleared up；
The third thread is by apart from one of the described first N number of historical data unit for clearance of historical data unit for clearance
Historical data unit for clearance is cleared up to the last one described historical data cell orientation for clearance, and N is natural number；
Second thread is from the last one described historical data unit for clearance to described first historical data list for clearance
First direction is cleared up；
4th thread is by the one of distance the last one N number of historical data unit for clearance of historical data unit for clearance
A historical data unit for clearance is cleared up to described first historical data cell orientation for clearance.
6. the method for cleaning of historical data according to claim 1, which is characterized in that by a history in the step S20
Historical data for clearance described in the timer-triggered scheduler of data dispatch center.
7. the method for cleaning of historical data according to claim 1, which is characterized in that each data of each thread
Cleaning amount is 50 to 100, and each thread completes 20 to 50 milliseconds of pause after a data scrubbing.
8. a kind of cleaning system of historical data, which is characterized in that the cleaning system of the historical data includes：
Rule reading module, for reading data purge rule；
Data acquisition module is used for by the data purge rule specified data library, obtaining historical data for clearance,
Wherein, database includes at least one set of historical data queue for clearance, is wrapped in historical data queue for clearance described in each group
Include multiple historical data units for clearance；
Computing module, for according to data purge rule, calculating needed for the historical data for clearance that cleaning is got
Thread quantity, wherein the quantity of the thread be more than or equal to 2；
Data scrubbing module, for the required number of threads that basis is calculated, to historical data for clearance described in each group
Queue configures corresponding thread and carries out data scrubbing, wherein is at least configured with first thread and the second thread, the first thread
With the second thread parallel while respectively by first history number for clearance positioned at head of the queue in one group of historical data queue for clearance
Start opposite progress data scrubbing successively according to unit and positioned at the last one historical data unit for clearance of tail of the queue.
9. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is held by processor
The step of method for cleaning of the historical data described in any one of claim 1 to 7 is realized when row.
10. a kind of electronic equipment, which is characterized in that including：
Processor；And
Memory, the executable instruction for storing the processor；
Wherein, the processor is configured to come described in any one of perform claim requirement 1 to 7 via the execution executable instruction
Historical data method for cleaning the step of.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810283195.3A CN108573036A (en) | 2018-04-02 | 2018-04-02 | Method for cleaning and its system, storage medium, the electronic equipment of historical data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810283195.3A CN108573036A (en) | 2018-04-02 | 2018-04-02 | Method for cleaning and its system, storage medium, the electronic equipment of historical data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108573036A true CN108573036A (en) | 2018-09-25 |
Family
ID=63573933
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810283195.3A Pending CN108573036A (en) | 2018-04-02 | 2018-04-02 | Method for cleaning and its system, storage medium, the electronic equipment of historical data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108573036A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109885565A (en) * | 2019-02-14 | 2019-06-14 | 中国银行股份有限公司 | A kind of tables of data method for cleaning and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064833A (en) * | 2011-10-18 | 2013-04-24 | 阿里巴巴集团控股有限公司 | Method of cleaning database history data and system thereof |
| US8645176B2 (en) * | 2006-10-05 | 2014-02-04 | Trimble Navigation Limited | Utilizing historical data in an asset management environment |
| CN105739956A (en) * | 2014-12-09 | 2016-07-06 | 阿里巴巴集团控股有限公司 | Method and system for constructing intelligent rule model of computer system |
| CN105760487A (en) * | 2016-02-17 | 2016-07-13 | 中国工商银行股份有限公司 | Historical data cleaning method and device |
| CN106909641A (en) * | 2017-02-16 | 2017-06-30 | 青岛高校信息产业股份有限公司 | A kind of real-time data memory device |
-
2018
- 2018-04-02 CN CN201810283195.3A patent/CN108573036A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8645176B2 (en) * | 2006-10-05 | 2014-02-04 | Trimble Navigation Limited | Utilizing historical data in an asset management environment |
| CN103064833A (en) * | 2011-10-18 | 2013-04-24 | 阿里巴巴集团控股有限公司 | Method of cleaning database history data and system thereof |
| CN105739956A (en) * | 2014-12-09 | 2016-07-06 | 阿里巴巴集团控股有限公司 | Method and system for constructing intelligent rule model of computer system |
| CN105760487A (en) * | 2016-02-17 | 2016-07-13 | 中国工商银行股份有限公司 | Historical data cleaning method and device |
| CN106909641A (en) * | 2017-02-16 | 2017-06-30 | 青岛高校信息产业股份有限公司 | A kind of real-time data memory device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109885565A (en) * | 2019-02-14 | 2019-06-14 | 中国银行股份有限公司 | A kind of tables of data method for cleaning and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107330034B (en) | Log analysis method and device, computer equipment and storage medium | |
| CN103312763B (en) | FTP client FTP, management server, workspace environment setting method and workspace environment set program | |
| CN109614427A (en) | The access method and device of Various database, storage medium and electronic equipment | |
| CN109783472A (en) | Moving method, device, computer equipment and the storage medium of table data | |
| CN112052138A (en) | Service data quality detection method and device, computer equipment and storage medium | |
| CN109933565A (en) | A kind of rapid section method and system of multiple dimensioned mass remote sensing image | |
| CN107392801A (en) | The method and its device, storage medium, electronic equipment of order are upset in control | |
| CN109669931A (en) | Historical data exception analysis method, system, equipment and storage medium | |
| CN108763577A (en) | node processing method and device, storage medium and electronic equipment | |
| CN109582451A (en) | Method for scheduling task, system, equipment and readable medium | |
| CN107111649A (en) | User and system data are uploaded into destination locations from source position | |
| CN108573036A (en) | Method for cleaning and its system, storage medium, the electronic equipment of historical data | |
| CN107633386A (en) | Management method, system, equipment and the storage medium of network terminal information | |
| CN110515899A (en) | File positioning method and device | |
| CN110175070A (en) | Management method, device, system, medium and the electronic equipment of distributed data base | |
| CN110300011B (en) | Alarm root cause positioning method, device and computer readable storage medium | |
| CN113836237A (en) | Method and device for auditing data operation of database | |
| Priyanka et al. | Fundamentals of wireless sensor networks using machine learning approaches: Advancement in big data analysis using Hadoop for oil pipeline system with scheduling algorithm | |
| CN109614257A (en) | Processing method, device, computer storage medium and the electronic equipment of program exception | |
| Cao et al. | LogKV: Exploiting key-value stores for event log processing | |
| CN107818538A (en) | Processing method, device, equipment and the computer-readable recording medium of watermarking images | |
| CN109739724B (en) | Data monitoring method, system, electronic device and storage medium | |
| CN109597933A (en) | Method, system, equipment and the storage medium of medical keyword precise search | |
| CN109684321A (en) | Data quality management method, device, electronic equipment, storage medium | |
| CN114912628A (en) | Feature selection method and device, electronic equipment and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180925 |
|
| RJ01 | Rejection of invention patent application after publication |