CN103020304A - Data processing method and equipment - Google Patents
Data processing method and equipment Download PDFInfo
- Publication number
- CN103020304A CN103020304A CN2012105929994A CN201210592999A CN103020304A CN 103020304 A CN103020304 A CN 103020304A CN 2012105929994 A CN2012105929994 A CN 2012105929994A CN 201210592999 A CN201210592999 A CN 201210592999A CN 103020304 A CN103020304 A CN 103020304A
- Authority
- CN
- China
- Prior art keywords
- time
- processing procedure
- data
- processing
- register
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Retry When Errors Occur (AREA)
Abstract
The invention provides a data processing method and equipment. The method comprises the following steps: receiving non-real time update data to be processed and storing the data in a processing database table; determining a time range beginning value of a this-time processing process according to a time range finishing value of a last-time processing process and a predetermined rollback time; determining a time range finishing value of the this-time processing process; scanning the non-real time update data of which the time stamps in the processing database table are between the time range beginning value and the time range finishing value of the this-time processing process, and inquiring the non-real time update data not processed therein; and starting the this-time processing process to process the inquired non-real time update data. The method and the equipment solve the problem of leakage processing of data records in the time range clinical point of two processing processes, and can achieve a better data processing effect and improve the system reliability.
Description
Technical field
The present invention relates to the microcomputer data processing field, particularly, relate to a kind of data processing method and equipment.
Background technology
Data processing ageingly can be divided into two types of real-time update mode and non real-time update modes according to what data were processed, wherein, data corresponding to real-time update mode are the data of having relatively high expectations to ageing, after operating, the user in database, obtain at once embodying, the result that subsequent user operation must draw based on front single job, and relevant with the data manipulation sequencing; Data corresponding to non real-time update mode are to ageing less demanding data, after operating, the user do not need in tables of data, to embody at once, only need embody in certain time interval getting final product, follow-up each user's operation is not necessarily based on the result of front single job.
In the large scale business banking system, because the business processing amount is huge, various system datas can be fought for system resource when upgrading, and the renewal of data has exclusiveness, must record the reading that locks to data when being Data Update, bottleneck appears when having caused hotlist record real-time update, for example, banking amount statistics needs to be updated 1,000,000 times in 8 hours of peak traffic phase every day, be to be updated average p.s. 34 times, if this data also adopt the mode of real-time update to upgrade, because the renewal temperature of data is high, and Data Update has exclusiveness, can cause the data processing stand-by period longer, the long expense that affects user's operation and the operation of increase system of system operation time.Therefore at present for upgrading temperature height and the ageing less demanding data of Data Update (hereinafter to be referred as non real-time more new data), usually process according to non real-time update mode, namely this part data is detached out from all pending data and carry out asynchronous operation, to reach the purpose that improves in-line processing overall operation speed.
Aspect the processing of non real-time more new data, there is following dual mode at present:
A kind of mode is to treat the deal with data table to carry out full table scan, therefrom find out treatment state and be untreated record and it is upgraded, but because each processing procedure all needs the pending tables of data of full table scan, therefore the treatment effeciency of this mode is unusually low.
Another mode is to determine first the time range of this processing procedure, and the record that is untreated in the scope is processed between treating in the deal with data table at this moment, determine successively again that afterwards follow-up time range processes, than first kind of way, although this mode has improved treatment effeciency, but so that some that is in the critical point of adjacent twice processing procedure time range records the possibility that exists by drain process.The reason that produces drain process mainly is to have the regular hour poor because the timestamp of data recording and data recording really appear between the time point on the tables of data, although that is: system has registered a timestamp according to the current time for it in the generated data record, but owing to also this record has been carried out just really being inserted in the tables of data after other operation subsequently, therefore it is poor that the timestamp of data recording and data recording really appear between the time point on the tables of data life period, for example shown in Figure 1, the timestamp of a pending a record generation is 10:00, newly-increased rear owing to carried out again other inquiry, newly-increased, the operations such as deletion SQL, until just making submitting to operation unification before, processes 10:01, being about to a record really is written in the tables of data, although this mistiming is very short, but concerning computing machine, if the time range of this processing procedure is by the end of 10:00, then owing to not yet appearing in the tables of data, a record therefore is not scanned, the zero-time of the corresponding time range of processing procedure next time is " greater than 10:00 ", and therefore to be 10:00 do not meet the next time time requirement of scanning of the timestamp of this record cause a record by drain process; Same situation may occur in some data recording that are in the critical point of adjacent twice processing procedure time range, causes the data treatment effect very undesirable.
Summary of the invention
The fundamental purpose of the embodiment of the invention is to provide a kind of data processing method and equipment, is used for solving in the asynchronous process process situation that pending record is omitted occurring.
To achieve these goals, the embodiment of the invention provides a kind of data processing method, comprising:
Receive pending non real-time more new data and be stored to pending database table;
According to time range end value and the predefined rollback time of processing procedure last time, determine the time range starting value of this processing procedure;
Determine the time range end value of this processing procedure;
Timestamp in the described pending database table is scanned at the time range starting value of described this processing procedure and the non real-time more new data between the time range end value, and inquire about wherein not processed non real-time more new data;
Start this processing procedure, the described non real-time more new data that inquires is processed.
Preferably, in the described data processing method, described time range end value and predefined rollback time according to processing procedure last time, determine specifically to comprise the time range starting value of this processing procedure:
The time range end value of processing procedure deducted the described rollback time with described last time, obtained the time range starting value of this processing procedure.
Preferably, in the described data processing method, before this processing procedure of described startup, also comprise:
According to concluding time and predefined processing interval time of processing procedure last time, determine the start time of this processing procedure;
Start this processing procedure in the start time of described this processing procedure.
Preferably, in the described data processing method, described according to last time processing procedure concluding time and process interval time, determine the start time of this processing procedure, specifically comprise:
The concluding time of processing procedure added the above and processed interval time with described last time, obtained the start time of this processing procedure.
Preferably, described data processing method also comprises:
In this processing procedure, utilize every system time when successfully processing a non real-time more new data to upgrade the operation monitoring time;
Start the monitoring processing procedure, obtain the current operation monitoring time, and judge that whether described current operation monitoring time and the result after described processing addition interval time are greater than current system time, if then definite this processing procedure is normally carried out, if not, then determine this processing procedure abnormal, send warning, and when receiving the restart indication of user's input, restart this processing procedure.
Preferably, described data processing method also comprises:
Instruction according to user's input arranges the described rollback time, processes interval time.
A kind of data processing equipment comprises: pending data-carrier store, the first register, the second register, time range processor, data processor and new processor more; Wherein,
Described pending data-carrier store is used for receiving pending non real-time more new data and being stored to pending database table;
Described the first register is used for the time range end value of storage processing procedure last time;
Described the second register is used for the storage rollback time;
Described time range processor connects described the first register and the second register, is used for time range end value and described rollback time according to described last time of processing procedure, determines the time range starting value of this processing procedure; And the time range end value of definite this processing procedure;
Described data processor connects respectively described pending data-carrier store and described time range processor, be used for described pending database table timestamp is scanned at the time range starting value of described this processing procedure and the non real-time more new data between the time range end value, and inquire about wherein not processed non real-time more new data; Start this processing procedure, the described non real-time more new data that inquires is processed;
Described more new processor connects described the first register, is used for utilizing the time range end value of this processing procedure to upgrade described the first register after this processing procedure finishes.
Preferably, described data processing equipment also comprises: the 3rd register, the 4th register, self-starting processor;
Described the 3rd register is used for the concluding time of storage processing procedure last time;
Described the 4th register is used for stores processor interval time;
Described self-starting processor connects described the 3rd register, the 4th register and data processor, is used for concluding time and predefined processing interval time according to processing procedure last time, determines the start time of this processing procedure; And start described data processor in the start time of described this processing procedure;
Described more new processor also connects described the 3rd register, is used for utilizing the concluding time of this processing procedure to upgrade described the 3rd register after this processing procedure finishes.
Preferably, described data processing equipment also comprises: the 5th register, monitoring processor, warning and reboot process device;
Described the 5th register is used for the storage running monitoring period;
Described more new processor also connects described the 5th register, is used in this processing procedure, utilizes every system time when successfully processing a non real-time more new data to upgrade the operation monitoring time;
Described monitoring processor connects described the 5th register, be used for starting the monitoring processing procedure, obtain the current operation monitoring time, and judge that whether described current operation monitoring time and the result after described processing addition interval time are greater than current system time, if then determine described data processor normal operation, if not, then determine described data processor operation exception, send the monitoring unexpected message to described warning and reboot process device;
Described warning and reboot process device connect described monitoring processor, are used for sending when receiving described monitoring unexpected message warning, and when receiving the restart indication of user's input, restart described data processor.
Preferably, described data processing equipment also comprises: also comprise: parameter arranges interface;
Described parameter arranges interface and connects described the second register, the 4th register, is used for the described rollback time being set and processing interval time according to the instruction of user's input.
By means of technique scheme, the present invention is by arranging the rollback time, so that the time range that adjacent twice processing procedure covers occurs is overlapping, thereby avoided being in data recording in the time range critical point of twice processing procedure by the situation of drain process; By non real-time more new data and real-time update data are processed respectively, so that in a large number ageing less demanding data can non real-timely be upgraded, solved the bottleneck problem of hotlist record real-time update; Timely reminding user when abnormal conditions appear in processing has improved the security that data are processed; In each processing procedure, search pending record by time range, avoided full dose to scan pending record sheet, the problem that search efficiency is low, system overhead is large.Than prior art, the embodiment of the invention had both solved the bottleneck problem of hotlist record real-time update, can obtain again preferably data treatment effect, had improved the reliability of system.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art, the accompanying drawing of required use was done to introduce simply during the below will describe embodiment, apparently, accompanying drawing in the following describes only is some embodiments of the present invention, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain according to these accompanying drawings other accompanying drawing.
Fig. 1 is that the prior art that provides of background technology of the present invention is to the synoptic diagram of non real-time renewal data processing time scope;
Fig. 2 is the schematic flow sheet of the data processing method that provides of the embodiment of the invention one;
Fig. 3 is the structured flowchart of the data processing equipment that provides of the embodiment of the invention two;
Fig. 4 is the structured flowchart of the data processing equipment that provides of the embodiment of the invention three;
Fig. 5 is the structured flowchart of the data storage device that provides of the embodiment of the invention three;
Fig. 6 is the structured flowchart of the asynchronous process device that provides of the embodiment of the invention three;
Fig. 7 is the structured flowchart of the asynchronous process starter gear that provides of the embodiment of the invention three;
Fig. 8 is the synoptic diagram to non real-time renewal data processing time scope that the embodiment of the invention three provides;
Fig. 9 is the schematic flow sheet that non real-time more new data is processed that the embodiment of the invention three provides.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the invention, the technical scheme in the embodiment of the invention is clearly and completely described, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, those of ordinary skills belong to the scope of protection of the invention not making the every other embodiment that obtains under the creative work prerequisite.
Embodiment one
Present embodiment provides a kind of data processing method, as shown in Figure 2, comprising:
Step S201 receives pending non real-time more new data and is stored to pending database table;
Step S202, according to last time processing procedure the time range end value and and the predefined rollback time, determine the time range starting value of this processing procedure;
Step S203 determines the time range end value of this processing procedure;
Step S204 scans at the time range starting value of described this processing procedure and the non real-time more new data between the time range end value timestamp in the described pending database table, and inquires about wherein not processed non real-time more new data;
Step S205 starts this processing procedure, and the described non real-time more new data that inquires is processed.
Concrete, in the present embodiment, the processing procedure of described processing procedure last time for carrying out before this processing procedure; The described rollback time is the predefined time of user, can rule of thumb set the concrete size of rollback time in the implementation, and present embodiment is not construed as limiting the occurrence of rollback time;
In the present embodiment, processing procedure has corresponding time range each time, used condition of scanning when this time range is the pending database table of computer scanning treats namely that timestamp scans at the starting value of time range and the non real-time more new data records between the end value in the process database table; Wherein, the time range starting value of this processing procedure was determined according to time range end value and the rollback time of processing procedure last time, and the time range end value of this processing procedure can be set according to actual conditions, as long as greater than the time range starting value of this processing procedure and less than or equal to current system time, present embodiment is not construed as limiting the time range end value of this processing procedure, better, can select current system time as the time range end value of this processing procedure;
The data processing method that present embodiment provides is by arranging the rollback time, so that the time range that adjacent twice processing procedure covers appearance is overlapping, thereby avoided being in data recording in the time range critical point of twice processing procedure by the situation of drain process, than prior art, the embodiment of the invention had both solved the bottleneck problem of hotlist record real-time update, can obtain again preferably data treatment effect.
Preferably, among the step S202 of present embodiment, described according to last time processing procedure the time range end value and and the predefined rollback time, determine specifically to comprise the time range starting value of this processing procedure:
The time range end value of processing procedure deducted the described rollback time with described last time, obtained the time range starting value of this processing procedure.
Concrete, in the present embodiment, except the time range end value of utilizing described last time of processing procedure deducts the time range starting value of result as this processing procedure of described rollback time, can also have other time range end value of utilizing processing procedure last time and and the rollback time determine the situation of the time range starting value of this processing procedure, for example, utilizing the time range end value of described last time of processing procedure to deduct the N(N of described rollback time and be natural number) doubly result is as the time range starting value of this processing procedure, present embodiment to the time range end value of utilizing processing procedure last time and and the rollback time determine that the process of the time range starting value of this processing procedure do not do concrete restriction, can determine according to actual conditions.
Preferably, among the step S205 of present embodiment, before this processing procedure of described startup, also comprise:
According to concluding time and predefined processing interval time of processing procedure last time, determine the start time of this processing procedure;
Start this processing procedure in the start time of described this processing procedure.
Concrete, in the present embodiment, have certain time interval between the adjacent processing procedure of carrying out for twice, i.e. the predefined processing time interval of user after each processing procedure finishes, just begins next time processing procedure interval time through described processing again.
Preferably, described according to last time processing procedure concluding time and process interval time, determine the start time of this processing procedure, specifically comprise:
The concluding time of processing procedure added the above and processed interval time with described last time, obtained the start time of this processing procedure.
Concrete, except the concluding time of utilizing described last time of processing procedure adds the above result who processes interval time the start time as this processing procedure, other concluding times of utilizing processing procedure last time can also be arranged in the present embodiment and process the situation of determining the start time of this processing procedure interval time, for example, utilizing concluding time of processing procedure last time to add the above N(N that processes interval time and be natural number) doubly result is as start time of this processing procedure, present embodiment is not done concrete restriction to the concluding time of utilizing processing procedure last time and the process of processing the start time of determining this processing procedure interval time, can determine according to actual conditions.
Preferably, the data processing method that provides of present embodiment also comprises:
In this processing procedure, utilize every system time when successfully processing a non real-time more new data to upgrade the operation monitoring time;
Start the monitoring processing procedure, obtain the current operation monitoring time, and judge that whether described current operation monitoring time and the result after described processing addition interval time are greater than current system time, if then definite this processing procedure is normally carried out, if not, then determine this processing procedure abnormal, send warning, and when receiving the restart indication of user's input, restart this processing procedure.
Concrete, whether normally carry out in order to monitor this processing procedure, utilize the monitoring processing procedure to monitor in the present embodiment, its specifically processing means are: in this processing procedure, whenever, successfully process a non real-time more new data, go to upgrade the operation monitoring time with regard to utilizing current system time; When starting the monitoring processing procedure, result after current operation monitoring time and processing addition interval time and current system time are compared, since adjacent twice processing procedure be spaced apart described processing interval time, therefore, if the interval in current system time and the last successful processing time that non real-time more new data is finished is greater than described processing interval time (be the operation monitoring time process result after addition interval time less than current system time with described), this processing procedure abnormal then is described, otherwise, just illustrate that this processing procedure is in the state that normally carries out.
Preferably, the data processing method that provides of present embodiment also comprises:
Instruction according to user's input arranges the described rollback time, processes interval time.
Concrete, this enforcement is not construed as limiting described rollback time and the occurrence of processing interval time, can set according to actual conditions.
Embodiment two
Present embodiment provides a kind of data processing equipment, and as shown in Figure 3, this equipment comprises: the first register 301, the second register 302, pending data-carrier store 303, time range processor 304, data processor 305 and new processor 306 more; Wherein,
Described pending data-carrier store 303 is used for receiving pending non real-time more new data and being stored to pending database table;
Described the first register 301 is used for the time range end value of storage processing procedure last time;
Described the second register 302 is used for the storage rollback time;
Described time range processor 304 connects described the first register 301 and the second registers 302, is used for time range end value and described rollback time according to described last time of processing procedure, determines the time range starting value of this processing procedure; And the time range end value of definite this processing procedure;
Described data processor 305 connects respectively described pending data-carrier store 303 and described time range processor 304, be used for described pending database table timestamp is scanned at the time range starting value of described this processing procedure and the non real-time more new data between the time range end value, and inquire about wherein not processed non real-time more new data; Start this processing procedure, the described non real-time more new data that inquires is processed;
Described more new processor 306 connects described the first register 301, is used for utilizing the time range end value of this processing procedure to upgrade described the first register 301 after this processing procedure finishes.
Preferably, the data processing equipment that provides of present embodiment also comprises: the 3rd register, the 4th register, self-starting processor;
Described the 3rd register is used for the concluding time of storage processing procedure last time;
Described the 4th register is used for stores processor interval time;
Described self-starting processor connects described the 3rd register, the 4th register and data processor 305, is used for concluding time and predefined processing interval time according to processing procedure last time, determines the start time of this processing procedure; And start described data processor 305 in the start time of described this processing procedure;
Described more new processor 306 also connects described the 3rd register, is used for utilizing the concluding time of this processing procedure to upgrade described the 3rd register after this processing procedure finishes.
Preferably, the data processing equipment that provides of present embodiment also comprises: the 5th register, monitoring processor, warning and reboot process device;
Described the 5th register is used for the storage running monitoring period;
Described more new processor 306 also connects described the 5th register, is used in this processing procedure, utilizes every system time when successfully processing a non real-time more new data to upgrade the operation monitoring time;
Described monitoring processor connects described the 5th register, be used for starting the monitoring processing procedure, obtain the current operation monitoring time, and judge that whether described current operation monitoring time and the result after described processing addition interval time are greater than current system time, if then determine described data processor 305 normal operations, if not, then determine described data processor 305 operation exceptions, send the monitoring unexpected message to described warning and reboot process device;
Described warning and reboot process device connect described monitoring processor, are used for sending when receiving described monitoring unexpected message warning, and when receiving the restart indication of user's input, restart described data processor 305.
Preferably, the data processing equipment that provides of present embodiment also comprises: parameter arranges interface;
Described parameter arranges interface and connects described the second register 302, the 4th register, is used for the described rollback time being set and processing interval time according to the instruction of user's input.
Concrete, the course of work of each function element can referring to the embodiment of embodiment one, repeat no more herein in the data processing equipment that present embodiment provides.
Embodiment three
Present embodiment provides a concrete data processing equipment, and as shown in Figure 4, data screening device 1, data storage device 2, asynchronous process device 3, parameter arrange interface 4 and asynchronous process starter gear 5.Wherein, data screening device 1, asynchronous process device 3, parameter arrange interface 4 and are connected in data storage device 2, and asynchronous process starter gear 5 is connected in asynchronous process device 3.Data screening device 1 externally is connected in external system 6.
The function of data screening device 1 is to receive external system 6 need to do the data of upgrading operation, the row filter of going forward side by side, if being judged as the real-time update data then directly upgrades operation and withdraws from processing, if be judged as non real-time more new data, then store in the data storage device 2, as pending data recording;
The function of data storage device 2 is temporary transient each time value that generates in storage pending data recording, system operational parameters and the operational process;
The function of asynchronous process device 3 is to confirm the time range of this issued transaction by the time rollback, the pending record that falls into time range is upgraded processing, then, calculate next time start-up time, automatically start asynchronous process device 3 and again carry out pending recording processing.
The function that general parameter arranges interface 4 is to receive user instruction, and the system operational parameters in the data memory storage 2 is arranged and adjusts.
Whether the function of asynchronous process starter gear 5 is monitoring asynchronous process device 3 normally operations, if the instruction of restarting asynchronous process device 3 is not then carried out in normal operation; If misoperation, prompting user then, and rerun asynchronous process device 3.
As shown in Figure 5, data storage device 2 comprises parameter storage unit 21, time temporary storage location 22, pending data-carrier store 23 and new processor 24 more;
As shown in Figure 6, asynchronous process device 3 comprises: time range processor 31, data processor 32 and self-starting processor 33;
As shown in Figure 7, asynchronous process starter gear 5 comprises: monitoring processor 51 and warning and reboot process device 52;
Table 1
| Parameter name | Parameter value |
| The rollback time | t |
| Process interval time | w |
Time temporary storage location 22 is made of three registers, is respectively applied to store field: last time processing procedure time range end value, last time processing procedure concluding time and operation monitoring time (as shown in table 2); Wherein, the time range end value of processing procedure " last time " be used for calculating the next time starting value of the time range of issued transaction; The concluding time of processing procedure " last time " preserves the concluding time of last data processor 32 operations, is used for calculating the time that next time starts asynchronous process device 3; Whether " operation monitoring time " preserves the system time of at every turn successfully processing a data record, be used for judging asynchronous process device 3 normally operation;
Table 2
| Field | Field value |
| Last time processing procedure the time range end value | T |
| Last time processing procedure concluding time | W |
| The operation monitoring time | M |
Pending data-carrier store 23 is used for storing non real-time renewal database table, and as shown in table 3, every record in this database table comprises data message, treatment state and timestamp three partial contents at least; Wherein, " data message " preserves the concrete data message that need to upgrade operation; " treatment state " is used for recording the treatment state of pending record, and record is defaulted as 0-when newly-increased and is untreated, and is updated to 1-and processes after being processed by asynchronous process device 3; Timestamp when " timestamp " records this data message and generate, asynchronous process device 3 by the time range that calculates issued transaction after, judge whether " timestamp " falls into this time range, determines whether these data to be processed;
Table 3
Three registers in 24 tie-time of the new processor temporary storage location 22 more are used for upgrading the field time range end value of processing procedure " last time ", the concluding time of processing procedure " last time " and " operation monitoring time ";
The function of time range processor 31 is to calculate the time range of this issued transaction, in order to need to select data message to be processed according to this time range; At first, the value t of " rollback time " parameter in the parameter storage unit 21 in the reading out data memory storage 2; Time temporary storage location 22 obtains the time range end value of processing procedure " last time " in the reading out data memory storage 2, is assumed to be T0; Then, deduct " rollback time " t with the time range end value of processing procedure " last time " T0, obtain time value T1 as this time range starting value, obtaining the current time in system is that T2 is as this time range end value again; At last, draw time range T1 to T2 as the time range of the pending record of screening, time range as shown in Figure 8;
The function of data processor 32 is the time ranges that calculate according to time range processor 31, and the pending record that obtains in the corresponding time range is processed; At first, the pending data-carrier store 23 of data processor 32 access, the value of choosing " timestamp " falls into all records of time range " T1 to T2 ", and judges whether its " treatment state " is that 0-is untreated; Then, qualified pending record is carried out data process, and more " treatment state " of new record processed for 1-; Whenever handling after notes records more, new processor 24 is updated to current system time with " operation monitoring time " field value in the time temporary storage location 22; All recording processing of this processing procedure finish, more new processor 24 just the time range end value of processing procedure " last time " field value in the time temporary storage location 22 is updated to T2, and general's concluding time of processing procedure " last time " field value be updated to current system time.
The function of self-starting processor 33 is according to certain interval time, automatically starts 3 pairs of pending records of asynchronous process device and processes; At first, 33 access time of self-starting processor temporary storage location 22 obtains the concluding time of processing procedure " last time " field value, is assumed to be W0, and access parameter storage unit 21 is obtained " processing interval time " parameter value w; Then, the concluding time of processing procedure " last time " W0 is added " processing interval time " w, obtain the start time W1 of next processing procedure; Then, when system time arrives W1, again start asynchronous process device 3;
The function of monitoring processor 51 is that operation monitoring time and current system time are compared, and whether monitoring asynchronous process device 3 works; Concrete processing procedure is: at first, 51 access time of monitoring processor temporary storage location 22 obtains current " operation monitoring time " field value M, and access parameter storage unit 21 is obtained " processing interval time " w; Then compare: whether M+w is less than the current system time; If M+w is less than the current system time, then explanation " operation monitoring time " M is not above upgrading normal time, i.e. asynchronous process device 3 operations occur unusual, and monitoring processor 51 sends monitors unexpected message to reporting to the police and reboot process device 52; If M+w, then illustrates asynchronous process device 3 normal operations greater than the current system time;
The function of warning and reboot process device 52 is according to monitoring processor 51 judged results, and whether control carries out the instruction that the user starts asynchronous process device 3; If receive the monitoring unexpected message that monitoring processor 51 sends, then reporting to the police occurs unusually with 3 operations of prompting user asynchronous process device, and restarts asynchronous process device 3 when receiving user instruction; If do not receive the monitoring unexpected message that monitoring processor 51 sends, then when receiving user instruction, do not allow to restart asynchronous process, namely do not do any operation.
The workflow of the data processing equipment that provides below in conjunction with 9 pairs of present embodiments of accompanying drawing is elaborated:
Suppose in this data processing equipment that the parameter that parameter storage unit 21 is preserved is as shown in table 4, the data of time temporary storage location 22 storages are as shown in table 5:
Table 4
| Parameter name | Parameter value (minute) |
| The |
3 |
| |
1 |
Table 5
| Field | Field value |
| Last time processing procedure the time range end value | 09:03:00 |
| Last time processing procedure concluding time | 09:03:05 |
| The operation monitoring time | 09:03:05 |
Step S901: data screening device 1 screens for the data that external system 6 is imported into, is judged as non real-time more new data, and then the pending record of registration in pending data-carrier store 23 is as shown in table 6;
Table 6
| Data message | Treatment state | Timestamp |
| XXXXX | 0 | 10:00:00 |
| XXXXX | 0 | 10:01:00 |
| XXXXX | 0 | 10:02:00 |
| XXXXX | 0 | 10:03:00 |
| XXXXX | 0 | 10:04:00 |
| XXXXX | 0 | 10:05:00 |
| XXXXX | 0 | 10:06:00 |
| XXXXX | 0 | 10:07:00 |
| XXXXX | 0 | 10:08:00 |
| XXXXX | 0 | 10:09:00 |
Step S902: time range processor 31 at first obtains " rollback time " from parameter storage unit 21 parameter value is 3 minutes, and the field value that obtains the time range end value of processing procedure " last time " from time temporary storage location 22 is 09:03:00; Then use the time range end value of processing procedure " last time " to deduct the time range starting value " 09:00:00 " that " rollback time " obtains this processing procedure; Obtain the current system time " 10:11:00 " as the time range end value of this processing procedure; Draw at last time range " 09:00:00-10:11:00 " as the time range of this processing procedure;
Step S903: the non real-time renewal database table of storage in the pending data-carrier store 23 of data processor 32 scannings, selecting " timestamp " numerical value falls within the time range " 09:00:00-10:11:00 " and " treatment state " is untreated for 0() pending record, 10 records in the above-mentioned pending data-carrier store 23 all within process range, carry out data to these 10 records and process; Then, upgrade " treatment state " of pending record and process for 1(), whenever handle a record, more new processor 24 is updated to current system time with the field value of " operation monitoring time "; All recording processing of this processing procedure finish, more new processor 24 is updated to " 10:11:00 " with the time range end value of processing procedure " last time " field value in the time temporary storage location 22, and the concluding time of processing procedure " last time " field value is updated to the current system time;
Step S904: monitoring processor 51 at first obtains current system time Time, and the parameter value that obtains " processing interval time " from parameter storage unit 21 is 1 minute, and obtaining current " operation monitoring time " from time temporary storage location 22 is 10:11:05; Then, add " processing interval time " with " operation monitoring time ", obtain time 10:13:05; Then, whether compare the time " 10:13:05 " greater than current system time T ime, if greater than, asynchronous process device 3 normal operations then are described, forward step S905 to and continue to carry out; Otherwise, asynchronous process device 3 operation exceptions are described, forward step S906 to and continue to carry out;
Step S905: the field value that 33 access time of self-starting processor temporary storage location 22 obtains the concluding time of processing procedure " last time " is 10:11:05, and the parameter value that access parameter storage unit 21 is obtained " processing interval time " is 1 minute; Then, will add the concluding time of processing procedure " last time " " processing interval time ", the start time that obtains next processing procedure is 10:12:05; When system time arrives 10:12:05, again start asynchronous process device 3, forward step S902 to and continue to carry out;
Step S906: warning and reboot process device 52 send warning according to monitoring processor 51 judged results, and 3 operations of prompting user asynchronous process device occur unusual;
Step S907: warning and reboot process device 52 judge whether to receive user instruction and restart asynchronous process device 3, if so, then restart asynchronous process device 3, forward step S902 to; Otherwise, process ends.
In sum, data processing method provided by the invention and device have following beneficial effect:
(1) by the rollback time is set, so that the time range that adjacent twice processing procedure covers appearance is overlapping, thereby avoided being in data recording in the time range critical point of twice processing procedure by the situation of drain process, had preferably data treatment effect, improved the reliability of system;
(2) by non real-time more new data and real-time update data are processed respectively, so that in a large number ageing less demanding data can non real-timely be upgraded, solved the bottleneck problem of hotlist record real-time update;
(3) timely reminding user when abnormal conditions appear in processing has improved the security that data are processed;
(4) in each processing procedure, search pending record by time range, avoided full dose to scan pending record sheet, the problem that search efficiency is low, system overhead is large.
Above-described specific embodiment; purpose of the present invention, technical scheme and beneficial effect are further described; institute is understood that; the above only is specific embodiments of the invention; the protection domain that is not intended to limit the present invention; within the spirit and principles in the present invention all, any modification of making, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a data processing method is characterized in that, comprising:
Receive pending non real-time more new data and be stored to pending database table;
According to time range end value and the predefined rollback time of processing procedure last time, determine the time range starting value of this processing procedure;
Determine the time range end value of this processing procedure;
Timestamp in the described pending database table is scanned at the time range starting value of described this processing procedure and the non real-time more new data between the time range end value, and inquire about wherein not processed non real-time more new data;
Start this processing procedure, the described non real-time more new data that inquires is processed.
2. data processing method according to claim 1 is characterized in that, described time range end value and predefined rollback time according to processing procedure last time, determines specifically to comprise the time range starting value of this processing procedure:
The time range end value of processing procedure deducted the described rollback time with described last time, obtained the time range starting value of this processing procedure.
3. data processing method according to claim 1 is characterized in that, before this processing procedure of described startup, also comprises:
According to concluding time and predefined processing interval time of processing procedure last time, determine the start time of this processing procedure;
Start this processing procedure in the start time of described this processing procedure.
4. data processing method according to claim 3 is characterized in that, described according to last time processing procedure concluding time and process interval time, determine the start time of this processing procedure, specifically comprise:
The concluding time of processing procedure added the above and processed interval time with described last time, obtained the start time of this processing procedure.
5. data processing method according to claim 4 is characterized in that, also comprises:
In this processing procedure, utilize every system time when successfully processing a non real-time more new data to upgrade the operation monitoring time;
Start the monitoring processing procedure, obtain the current operation monitoring time, and judge that whether described current operation monitoring time and the result after described processing addition interval time are greater than current system time, if then definite this processing procedure is normally carried out, if not, then determine this processing procedure abnormal, send warning, and when receiving the restart indication of user's input, restart this processing procedure.
6. data processing method according to claim 5 is characterized in that, also comprises:
Instruction according to user's input arranges the described rollback time, processes interval time.
7. a data processing equipment is characterized in that, comprising: pending data-carrier store, the first register, the second register, time range processor, data processor and new processor more; Wherein,
Described pending data-carrier store is used for receiving pending non real-time more new data and being stored to pending database table;
Described the first register is used for the time range end value of storage processing procedure last time;
Described the second register is used for the storage rollback time;
Described time range processor connects described the first register and the second register, is used for time range end value and described rollback time according to described last time of processing procedure, determines the time range starting value of this processing procedure; And the time range end value of definite this processing procedure;
Described data processor connects respectively described pending data-carrier store and described time range processor, be used for described pending database table timestamp is scanned at the time range starting value of described this processing procedure and the non real-time more new data between the time range end value, and inquire about wherein not processed non real-time more new data; Start this processing procedure, the described non real-time more new data that inquires is processed;
Described more new processor connects described the first register, is used for utilizing the time range end value of this processing procedure to upgrade described the first register after this processing procedure finishes.
8. data processing equipment according to claim 7 is characterized in that, also comprises: the 3rd register, the 4th register, self-starting processor;
Described the 3rd register is used for the concluding time of storage processing procedure last time;
Described the 4th register is used for stores processor interval time;
Described self-starting processor connects described the 3rd register, the 4th register and data processor, is used for concluding time and predefined processing interval time according to processing procedure last time, determines the start time of this processing procedure; And start described data processor in the start time of described this processing procedure;
Described more new processor also connects described the 3rd register, is used for utilizing the concluding time of this processing procedure to upgrade described the 3rd register after this processing procedure finishes.
9. data processing equipment according to claim 7 is characterized in that, also comprises: the 5th register, monitoring processor, warning and reboot process device;
Described the 5th register is used for the storage running monitoring period;
Described more new processor also connects described the 5th register, is used in this processing procedure, utilizes every system time when successfully processing a non real-time more new data to upgrade the operation monitoring time;
Described monitoring processor connects described the 5th register, be used for starting the monitoring processing procedure, obtain the current operation monitoring time, and judge that whether described current operation monitoring time and the result after described processing addition interval time are greater than current system time, if then determine described data processor normal operation, if not, then determine described data processor operation exception, send the monitoring unexpected message to described warning and reboot process device;
Described warning and reboot process device connect described monitoring processor, are used for sending when receiving described monitoring unexpected message warning, and when receiving the restart indication of user's input, restart described data processor.
10. data processing equipment according to claim 9 is characterized in that, also comprises: parameter arranges interface;
Described parameter arranges interface and connects described the second register, the 4th register, is used for the described rollback time being set and processing interval time according to the instruction of user's input.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012105929994A CN103020304A (en) | 2012-12-31 | 2012-12-31 | Data processing method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012105929994A CN103020304A (en) | 2012-12-31 | 2012-12-31 | Data processing method and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103020304A true CN103020304A (en) | 2013-04-03 |
Family
ID=47968907
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012105929994A Pending CN103020304A (en) | 2012-12-31 | 2012-12-31 | Data processing method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103020304A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636401A (en) * | 2013-11-15 | 2015-05-20 | 中国石油天然气股份有限公司 | Method and device for data rollback of SCADA (supervisory control and data acquisition) system |
| CN105450737A (en) * | 2015-11-13 | 2016-03-30 | 网易（杭州）网络有限公司 | Data processing method, data processing device and data processing system |
| CN105468699A (en) * | 2015-11-18 | 2016-04-06 | 珠海多玩信息技术有限公司 | Duplicate removal data statistics method and equipment |
| CN105589797A (en) * | 2015-09-17 | 2016-05-18 | ***股份有限公司 | Method for synchronous data time delay detection between master database and slave database |
| CN107026744A (en) * | 2015-06-25 | 2017-08-08 | 中兴通讯股份有限公司 | A kind of acquisition terminal device system start time and the method and apparatus of end time |
| CN108334633A (en) * | 2018-02-26 | 2018-07-27 | 平安普惠企业管理有限公司 | Data-updating method, device, computer equipment and storage medium |
| CN109741153A (en) * | 2019-01-18 | 2019-05-10 | 网联清算有限公司 | Overtime transaction processing method, device, computer equipment and storage medium |
| CN109815291A (en) * | 2018-12-28 | 2019-05-28 | 北京城市网邻信息技术有限公司 | Method of data synchronization, device, electronic equipment and storage medium |
| CN110069525A (en) * | 2018-08-20 | 2019-07-30 | 赛凡信息科技（厦门）有限公司 | A kind of scheme of acceleration search database information |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100036895A1 (en) * | 2008-08-06 | 2010-02-11 | International Business Machines Corporation | Representation of system clock changes in time based file systems |
| CN102591993A (en) * | 2012-02-16 | 2012-07-18 | 中国工商银行股份有限公司 | Asynchronous processing method and device for hotlist data update |
| CN202600778U (en) * | 2012-02-16 | 2012-12-12 | 中国工商银行股份有限公司 | Asynchronous processing system for updating hotlist data |
-
2012
- 2012-12-31 CN CN2012105929994A patent/CN103020304A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100036895A1 (en) * | 2008-08-06 | 2010-02-11 | International Business Machines Corporation | Representation of system clock changes in time based file systems |
| CN102591993A (en) * | 2012-02-16 | 2012-07-18 | 中国工商银行股份有限公司 | Asynchronous processing method and device for hotlist data update |
| CN202600778U (en) * | 2012-02-16 | 2012-12-12 | 中国工商银行股份有限公司 | Asynchronous processing system for updating hotlist data |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636401A (en) * | 2013-11-15 | 2015-05-20 | 中国石油天然气股份有限公司 | Method and device for data rollback of SCADA (supervisory control and data acquisition) system |
| CN104636401B (en) * | 2013-11-15 | 2018-04-06 | 中国石油天然气股份有限公司 | Method and device for data rollback of SCADA (supervisory control and data acquisition) system |
| CN107026744B (en) * | 2015-06-25 | 2020-04-14 | 中兴通讯股份有限公司 | Method and device for acquiring system start time and end time of terminal device |
| CN107026744A (en) * | 2015-06-25 | 2017-08-08 | 中兴通讯股份有限公司 | A kind of acquisition terminal device system start time and the method and apparatus of end time |
| CN105589797B (en) * | 2015-09-17 | 2018-06-22 | ***股份有限公司 | The method that data synchronizing relay detects between master-slave database |
| CN105589797A (en) * | 2015-09-17 | 2016-05-18 | ***股份有限公司 | Method for synchronous data time delay detection between master database and slave database |
| CN105450737B (en) * | 2015-11-13 | 2019-02-12 | 网易（杭州）网络有限公司 | A kind of data processing method, device and system |
| CN105450737A (en) * | 2015-11-13 | 2016-03-30 | 网易（杭州）网络有限公司 | Data processing method, data processing device and data processing system |
| CN105468699A (en) * | 2015-11-18 | 2016-04-06 | 珠海多玩信息技术有限公司 | Duplicate removal data statistics method and equipment |
| CN105468699B (en) * | 2015-11-18 | 2019-06-18 | 珠海多玩信息技术有限公司 | Duplicate removal data statistical approach and equipment |
| CN108334633A (en) * | 2018-02-26 | 2018-07-27 | 平安普惠企业管理有限公司 | Data-updating method, device, computer equipment and storage medium |
| CN108334633B (en) * | 2018-02-26 | 2020-09-15 | 平安普惠企业管理有限公司 | Data updating method and device, computer equipment and storage medium |
| CN110069525A (en) * | 2018-08-20 | 2019-07-30 | 赛凡信息科技（厦门）有限公司 | A kind of scheme of acceleration search database information |
| CN109815291A (en) * | 2018-12-28 | 2019-05-28 | 北京城市网邻信息技术有限公司 | Method of data synchronization, device, electronic equipment and storage medium |
| CN109815291B (en) * | 2018-12-28 | 2021-08-20 | 北京城市网邻信息技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN109741153A (en) * | 2019-01-18 | 2019-05-10 | 网联清算有限公司 | Overtime transaction processing method, device, computer equipment and storage medium |
| CN109741153B (en) * | 2019-01-18 | 2021-02-12 | 网联清算有限公司 | Overtime transaction processing method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103020304A (en) | Data processing method and equipment | |
| CN108683539B (en) | Block chain network management method, block chain network management device, block chain network management medium and electronic equipment | |
| US10310900B2 (en) | Operating programs on a computer cluster | |
| CN103092903A (en) | Database Log Parallelization | |
| US20110218971A1 (en) | System, Method And Computer Program Product For Managing Caches | |
| CN102043859A (en) | Data updating method and device | |
| CN104021043A (en) | Interruption reentry method and system of batch applications | |
| CN103544153A (en) | Data updating method and system based on database | |
| CN105719112A (en) | Determination method and device for distribution abnormal state, and server | |
| CN113672263B (en) | Firmware parameter updating control method and device and electronic equipment | |
| CN102591993A (en) | Asynchronous processing method and device for hotlist data update | |
| CN107315657A (en) | A kind of method and device of data backup | |
| CN103235811A (en) | Data storage method and device | |
| JP2013045356A (en) | Transaction simultaneous execution control system, transaction simultaneous execution control method, and program | |
| CN202600778U (en) | Asynchronous processing system for updating hotlist data | |
| CN103838762A (en) | Uninterrupted data processing device and method based on data table rotation | |
| CN111125156A (en) | Data query method and device and electronic equipment | |
| CN115168384A (en) | Data consistency processing method, device, server and storage medium | |
| US11321120B2 (en) | Data backup method, electronic device and computer program product | |
| CN115578180A (en) | Bank outlet cash resource management method and device | |
| CN115391703A (en) | Interface information processing method and device | |
| CN113849161A (en) | Application control method and device, storage medium and electronic equipment | |
| CN107967275B (en) | Data processing method and device in relational database | |
| CN111950770A (en) | Method and device for managing resource return auxiliary strategy and electronic equipment | |
| CN116521400B (en) | Article information processing method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20130403 |