CN111078714A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN111078714A CN111078714A CN201911165309.5A CN201911165309A CN111078714A CN 111078714 A CN111078714 A CN 111078714A CN 201911165309 A CN201911165309 A CN 201911165309A CN 111078714 A CN111078714 A CN 111078714A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- processing
- date
- storage date
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 136
- 238000013500 data storage Methods 0.000 claims abstract description 91
- 238000000034 method Methods 0.000 claims abstract description 39
- 230000004044 response Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 5
- 230000003203 everyday effect Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Finance (AREA)
- Accounting & Taxation (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data processing method and a device, wherein the method comprises the following steps: acquiring a data file of data to be processed, wherein the data file comprises a data storage date and a transaction date; judging whether the data to be processed jumps or not according to the data storage date and the transaction date; if the data to be processed does not jump, judging whether newly added data exists in the data to be processed according to the data storage date; and if the data to be processed contains new data, processing the data based on the new data. The embodiment of the invention carries out data processing based on the newly added data, thereby avoiding the processing of a large amount of repeated data, solving the problems of long time consumption, slow service response and large consumption of system resources in the existing data processing, and meeting the requirements of various application scenes.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a data processing method and device.
Background
With the development of economy, people are more and more exposed to data, and data management is concerned by people. For example, pension management generally uses a wind control system to acquire data from a managed bank and perform corresponding processing. For subsequent data analysis based on the processed data.
The data processing method of the wind control system is as follows: all the combined tasks of shifting the date are executed by timing. For example: setting the timing to be eight night points every day, and setting the offset date to be 30 days, the wind control system needs to process all combined task data within 30 days at eight night points every day.
However, the data processing method is time-consuming, resulting in slow response speed of the service, and needs to consume a large amount of system resources, which is not suitable for practical application.
Disclosure of Invention
The embodiment of the invention provides a data processing method and a data processing device, which are used for solving the problems that the existing data processing method is long in time consumption, low in service response speed, large in system resource consumption and not suitable for practical application.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
acquiring a data file of data to be processed, wherein the data file comprises a data storage date and a transaction date;
judging whether the data to be processed jumps or not according to the data storage date and the transaction date;
if the data to be processed does not jump, judging whether newly added data exists in the data to be processed according to the data storage date;
and if the data to be processed contains new data, processing the data based on the new data.
Optionally, the method further comprises:
acquiring a processing data track table, wherein the processing data track table comprises storage dates of processed data;
the judging whether the data to be processed has newly added data according to the data storage date comprises the following steps:
and judging whether the data to be processed has new data or not according to the data storage date and the date of the processed data.
Optionally, the determining whether the data to be processed jumps according to the data storage date and the transaction date includes:
judging whether the data storage date is consistent with the transaction date;
if the data storage date is consistent with the transaction date, judging that the data to be processed does not jump;
and if the data storage date is not consistent with the transaction date, judging that the data to be processed jumps.
Optionally, after the acquiring the data file of the data to be processed, the method further includes:
detecting whether the format of the data file of the data to be processed is correct or not according to the format of a preset data file;
if the format of the data file of the data to be processed is correct, detecting whether the data file of the data to be processed is complete according to preset keywords;
and if the data file of the data to be processed is complete, executing the step of judging whether the data to be processed jumps according to the data storage date and the transaction date.
Optionally, before the acquiring the data file of the data to be processed, the method further includes: acquiring data to be processed;
the acquiring of the data to be processed comprises:
setting an upper limit of data processing capacity;
acquiring the data to be processed from a preset database according to a data processing priority identifier of the data in the preset database and the data processing amount upper limit;
alternatively, the first and second electrodes may be,
and acquiring the data to be processed from the preset database according to the data storage date of the data in the preset database and the upper limit of the data processing amount.
Optionally, the performing data processing based on the new data includes:
acquiring a newly added landing estimation table according to the newly added data;
acquiring position holding information according to the newly added landing estimation table;
calculating a combined asset according to the position taken information;
and performing index landing processing on the combined assets.
Optionally, after the data processing based on the newly added data, the method further includes:
and updating the processing data track table according to the processing result.
In a second aspect, an embodiment of the present invention provides an apparatus for data processing, including:
the system comprises a first acquisition module, a second acquisition module and a processing module, wherein the first acquisition module is used for acquiring a data file of data to be processed, and the data file comprises a data storage date and a transaction date;
the judging module is used for judging whether the data to be processed jumps or not according to the data storage date and the transaction date;
the data processing device is also used for judging whether the data to be processed has newly added data or not according to the data storage date if the data to be processed does not jump;
and the first processing module is used for processing data based on the newly added data if the data to be processed contains the newly added data.
Optionally, the method further comprises: a second obtaining module for obtaining the first data,
the second acquisition module is used for acquiring a processing data track table, and the processing data track table comprises the storage date of the processed data;
the judging module judges whether the data to be processed has new data according to the data storage date, and the judging module comprises the following steps:
and judging whether the data to be processed has new data or not according to the data storage date and the storage date of the processed data.
Optionally, the determining module determines whether the data to be processed jumps according to the data storage date and the transaction date, including:
judging whether the data storage date is consistent with the transaction date;
if the data storage date is consistent with the transaction date, judging that the data to be processed does not jump;
and if the data storage date is not consistent with the transaction date, judging that the data to be processed jumps.
Optionally, the method further comprises: a second processing module for performing a second processing operation,
after the data file of the data to be processed is obtained, detecting whether the format of the data file of the data to be processed is correct or not according to the format of a preset data file;
if the format of the data file of the data to be processed is correct, detecting whether the data file of the data to be processed is complete according to preset keywords;
and if the data file of the data to be processed is complete, the judging module executes the step of judging whether the data to be processed jumps according to the data storage date and the transaction date.
Optionally, the method further comprises: a third obtaining module for obtaining the data of the first group,
the data file is used for acquiring the data to be processed before the data file acquiring the data to be processed;
the third obtaining module obtains data to be processed, and the third obtaining module comprises:
setting an upper limit of data processing capacity;
acquiring the data to be processed from a preset database according to a data processing priority identifier of the data in the preset database and the data processing amount upper limit;
alternatively, the first and second electrodes may be,
and acquiring the data to be processed from the preset database according to the data storage date of the data in the preset database and the upper limit of the data processing amount.
Optionally, the first processing module is specifically configured to:
acquiring a newly added landing estimation table according to the newly added data;
acquiring position holding information according to the newly added landing estimation table;
calculating a combined asset according to the position taken information;
and performing index landing processing on the combined assets.
Optionally, the method further comprises: the updating module is used for updating the data of the data storage module,
and the updating module is used for updating the processing data track table according to a processing result after the first processing module performs data processing based on the newly added data.
In a third aspect, an embodiment of the present invention provides an apparatus for data processing, including: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of data processing as set forth in the first aspect above and in various possible designs of the first aspect.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored, and when a processor executes the computer-executable instructions, the method for processing data according to the first aspect and various possible designs of the first aspect is implemented.
The method comprises the steps of obtaining a data file of data to be processed, wherein the data file comprises a data storage date and a transaction date, judging whether the data to be processed jumps according to the data storage date and the transaction date, and executing subsequent steps if the data to be processed does not jump so as to process error data in time, so that consumption of a large amount of hardware resources is saved, risks of problem data can be effectively prevented, and whether newly added data exist in the data to be processed is judged according to the data storage date; if the data to be processed has the newly added data, the data processing is carried out based on the newly added data, so that the processing of a large amount of repeated data is avoided, the problems of long time consumption, slow service response and large consumption of system resources in the conventional data processing are solved, and the requirements of various application scenes are met.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a block diagram of a system for data processing according to an embodiment of the present invention;
fig. 2 is a first flowchart illustrating a data processing method according to an embodiment of the present invention;
fig. 3 is a second flowchart illustrating a data processing method according to an embodiment of the present invention;
FIG. 4 is a first schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 6 is a schematic hardware structure diagram of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data processing method of the wind control system is as follows: all the combined tasks of shifting the date are executed by timing. For example: setting the timing to be eight night points every day, and setting the offset date to be 30 days, the wind control system needs to process all combined task data within 30 days at eight night points every day.
However, the data processing method is time-consuming, resulting in slow response speed of the service, and needs to consume a large amount of system resources, which is not suitable for practical application.
Therefore, in view of the above problems, the present embodiment provides a data processing method, which includes obtaining a data file of data to be processed, where the data file includes a data storage date and a transaction date, determining whether the data to be processed jumps according to the data storage date and the transaction date, and if the data to be processed does not jump, performing subsequent steps to process error data in time, so as to not only save consumption of a large amount of hardware resources, but also effectively prevent risks of problem data, and determining whether there is new data in the data to be processed according to the data storage date; if the data to be processed has the newly added data, the data processing is carried out based on the newly added data, so that the processing of a large amount of repeated data is avoided, the problems of long time consumption, slow service response, consumption of a large amount of system resources and the like of the existing data processing are solved, and the requirements of various application scenes are met.
The present embodiment provides a method for data processing, which may be applied to the schematic architecture diagram of the data processing system shown in fig. 1, where as shown in fig. 1, the system provided in the present embodiment includes a terminal 101. The terminal 101 may obtain a data file of the data to be processed, for example, a data file of the data to be processed may be obtained from a custodian, where the data file includes a data storage date and a transaction date; the terminal 101 can judge whether the data to be processed jumps according to the data storage date and the transaction date; whether newly added data exist in the data to be processed can be judged according to the data storage date of the data file; if the data to be processed has new data, the new data can be processed. The terminal 101 may be a computer, a mobile phone, or the like. The present embodiment does not particularly limit the implementation manner of the terminal 101 as long as the terminal 101 can perform the above-described data processing.
The following describes the technical solution of the present invention and how to solve the above technical problems with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 2 is a first flowchart illustrating a data processing method according to an embodiment of the present invention, where an execution main body of the embodiment may be the terminal 101 in the embodiment shown in fig. 1, and the embodiment is not limited herein. As shown in fig. 2, the method may include:
s201: and acquiring a data file of the data to be processed, wherein the data file comprises a data storage date and a transaction date.
The above manner of obtaining the data file of the data to be processed may be: and decompressing and analyzing the data to be processed to obtain a corresponding data file with the state of waiting for processing.
The data to be processed is data that needs to be processed, for example: data from the managed row that needs to be processed.
The data file includes a data storage date and a transaction date, and optionally, the data file may further include a data combination name and the like.
S202: and judging whether the data to be processed jumps or not according to the data storage date and the transaction date.
Optionally, the determining whether the data to be processed jumps according to the data storage date and the transaction date may be implemented by, but not limited to:
judging whether the data storage date is consistent with the transaction date;
if the data storage date is consistent with the transaction date, judging that the data to be processed does not jump;
and if the data storage date is not consistent with the transaction date, judging that the data to be processed jumps.
Illustratively, the transaction dates are 7 month 1, 7 month 5 and 7 month 10 of this year, if the data storage dates are also 7 month 1, 7 month 5 and 7 month 10 of this year, it is determined that the data to be processed do not jump, otherwise, it is determined that the data to be processed jump.
If the data to be processed does not jump, step S203 is executed.
And if the data to be processed jumps, namely when the data to be processed is discontinuous, the subsequent steps are not executed for the moment. When the data to be processed jumps, the data is incorrect, the subsequent steps are executed, and the hardware resources are wasted because the re-calculation is required to be conducted again.
By judging whether the data to be processed jumps or not, executing the subsequent steps when the data to be processed does not jump, and temporarily not executing the subsequent steps when the data to be processed jumps, the error data can be processed in time, so that the consumption of a large amount of hardware resources is saved, and the risk of problem data can be effectively prevented.
S203: and if the data to be processed does not jump, judging whether newly added data exists in the data to be processed according to the data storage date.
Optionally, before determining whether there is new data in the data to be processed according to the data storage date, a processed data track table is obtained, where the processed data track table includes a storage date of processed data.
Illustratively, the stored hosting line file is traversed, and a processing data track table corresponding to the storage date of the recorded processed data is obtained.
And judging whether the data to be processed has new data or not through the acquired processing data track table, and ensuring the reliability of judging the new data.
According to the data storage date, whether new data are added in the data to be processed can be judged by, but not limited to, the following modes:
and judging whether the data to be processed has new data or not according to the data storage date and the storage date of the processed data.
For example, the above-mentioned determining whether there is new data in the data to be processed according to the data storage date and the storage date of the processed data may be implemented by, but not limited to, the following manners:
whether there is a date in the data storage date that the storage date of the processed data does not exist,
and if the date which does not exist in the storage date of the processed data exists in the data storage dates, judging that the data to be processed contains newly added data.
And when the data storage dates of all the data in the data to be processed are contained in the storage date of the processed data, namely the data storage dates of all the data in the data to be processed are stored in the processed data track table, judging that no new data is added in the data to be processed.
And when the data storage date in the data to be processed is not included in the storage date of the processed data, namely the data storage date of the data in the data to be processed is not in the processed data track table, judging that the data to be processed has new data, and judging that the data of which the data storage date is not in the processed data track table is the new data.
Whether newly added data exist in the data to be processed can be accurately judged according to the data storage date and the date of the processed data.
S204: and if the data to be processed contains new data, processing the data based on the new data.
Optionally, performing data processing based on the new data includes:
acquiring a newly added landing estimation table according to the newly added data;
acquiring position holding information according to the newly added landing estimation table;
calculating a combined asset according to the position taken information;
and performing index landing processing on the combined assets.
According to the data processing method provided by the embodiment of the invention, the data file of the data to be processed is obtained, wherein the data file comprises the data storage date and the transaction date, whether the data to be processed jumps or not is judged according to the data storage date and the transaction date, the subsequent steps are executed when the data to be processed does not jump, and when the data to be processed jumps, the subsequent steps are not executed temporarily, so that the error data can be processed in time, the consumption of a large amount of hardware resources is saved, and meanwhile, the risk of problem data can be effectively prevented; judging whether newly added data exist in the data to be processed or not according to the data storage date; if the data to be processed has newly-added data, the data processing is carried out based on the newly-added data, so that the processing of a large amount of repeated data is avoided, and the problems of long time consumption, slow service response and large consumption of system resources are solved.
Fig. 3 is a second flowchart illustrating a data processing method according to an embodiment of the present invention, where an execution main body of the embodiment may be the terminal 101 in the embodiment shown in fig. 1, and the embodiment is not limited herein. As shown in fig. 3, the method includes:
s301: and acquiring data to be processed.
The acquiring of the data to be processed comprises:
setting an upper limit of data processing capacity;
acquiring the data to be processed from a preset database according to a data processing priority identifier of the data in the preset database and the data processing amount upper limit;
alternatively, the first and second electrodes may be,
and acquiring the data to be processed from the preset database according to the data storage date of the data in the preset database and the upper limit of the data processing amount.
Wherein, the upper limit of the data processing amount can be set according to the processing capacity.
Illustratively, according to the data processing priority level identifier, the data with the higher priority level can be processed preferentially, and according to the upper limit of the processing amount, the data to be processed is obtained according to the priority order.
Or the data with the earlier storage date can be processed preferentially according to the storage date of the data, and the data to be processed can be acquired according to the order from the morning to the evening of the storage date according to the online processing amount.
S302: and acquiring a data file of the data to be processed, wherein the data file comprises a data storage date and a transaction date.
S303: and detecting whether the format of the data file of the data to be processed is correct or not according to the format of a preset data file.
S304: if the format of the data file of the data to be processed is correct, detecting whether the data file of the data to be processed is complete according to preset keywords;
if the data file of the data to be processed is complete, S305 is executed.
If the format of the data file of the data to be processed is incorrect or incomplete, the data file is invalid, and the subsequent operation under the condition that the data file is invalid can be avoided by detecting the format and the integrity of the data file of the data to be processed.
S305: and judging whether the data to be processed jumps or not according to the data storage date and the transaction date.
The steps S302 and S305 are the same as the implementation of the steps S201 to S202, and are not described herein again.
S306: and if the data to be processed does not jump, acquiring a processed data track table, wherein the processed data track table comprises the storage date of the processed data.
S307: and judging whether the data to be processed has new data or not according to the data storage date and the storage date of the processed data.
S308: and if the data to be processed contains new data, processing the data based on the new data.
The step S308 is the same as the step S204, and is not described herein again.
S309: and updating the processing data track table according to the processing result.
Illustratively, the data processed in step S308 is added to the processing data trace table, and the processing data trace table is updated.
According to the data processing method provided by the embodiment of the invention, the data to be processed can be obtained according to different application scenes according to the data processing priority identification or the data storage date and the data processing amount upper limit; acquiring a data file of data to be processed, wherein the data file comprises a data storage date and a transaction date, judging the correctness and the completeness of the data file format of the data to be processed, if the data file format of the data to be processed is incorrect or incomplete, the data file is invalid, and detecting the format and the completeness of the data file of the data to be processed, so that the subsequent operation under the condition that the data file is invalid can be avoided; judging whether the data to be processed jumps according to the data storage date and the transaction date, executing the subsequent steps when the data to be processed does not jump, and temporarily not executing the subsequent steps when the data to be processed jumps, so that error data can be processed in time, consumption of a large amount of hardware resources is saved, and risks of problem data can be effectively prevented; if not, judging whether the data to be processed has new data or not according to the data storage date; if the newly added data exist, the data processing is carried out based on the newly added data, so that the processing of a large amount of repeated data is avoided, and the problems of long time consumption, slow service response and large consumption of system resources are solved; by updating the processing data track table according to the processing result and processing the data track table to obtain the date of the updated processed data, whether new data are added in the data to be processed can be judged more accurately, and repeated processing of the data is avoided.
Corresponding to the method of data processing in the foregoing embodiments, fig. 4 is a schematic structural diagram of a data processing apparatus provided for the embodiment of the present invention. As shown in fig. 4, the data processing apparatus 40 includes: a first obtaining module 401, a judging module 402 and a first processing module 403.
The first obtaining module 401 is configured to obtain a data file of data to be processed, where the data file includes a data storage date and a transaction date;
the judging module 402 is used for judging whether the data to be processed jumps or not according to the data storage date and the transaction date;
the data processing device is also used for judging whether the data to be processed has newly added data or not according to the data storage date if the data to be processed does not jump;
a first processing module 403, configured to perform data processing based on the newly added data if the to-be-processed data includes the newly added data.
The apparatus provided in the embodiment of the present invention may be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, which are not described herein again.
Fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention. As shown in fig. 5, the data processing apparatus 50 further includes, on the basis of fig. 4: a second obtaining module 404 and an updating module 405 and a second processing module 406 and a third obtaining module 407.
Optionally, the second obtaining module 404 is configured to obtain a processing data track table, where the processing data track table includes a storage date of the processed data;
the determining module 402 is specifically configured to:
and judging whether the data to be processed has new data or not according to the data storage date and the storage date of the processed data.
Optionally, the determining module 402 determines whether the data to be processed jumps according to the data storage date and the transaction date, including:
judging whether the data storage date is consistent with the transaction date;
if the data storage date is consistent with the transaction date, judging that the data to be processed does not jump;
and if the data storage date is not consistent with the transaction date, judging that the data to be processed jumps.
Optionally, the second processing module 406 is configured to, after the data file of the to-be-processed data is obtained, detect whether the format of the data file of the to-be-processed data is correct according to a format of a preset data file;
if the format of the data file of the data to be processed is correct, detecting whether the data file of the data to be processed is complete according to preset keywords;
if the data file of the data to be processed is complete, the determining module 402 executes the step of determining whether the data to be processed jumps according to the data storage date and the transaction date.
Optionally, the third obtaining module 407 is configured to obtain the data to be processed before obtaining the data file of the data to be processed;
the third obtaining module 407 obtains data to be processed, including:
setting an upper limit of data processing capacity;
acquiring the data to be processed from a preset database according to a data processing priority identifier of the data in the preset database and the data processing amount upper limit;
alternatively, the first and second electrodes may be,
and acquiring the data to be processed from the preset database according to the data storage date of the data in the preset database and the upper limit of the data processing amount.
Optionally, the first processing module 403 is specifically configured to:
acquiring a newly added landing estimation table according to the newly added data;
acquiring position holding information according to the newly added landing estimation table;
calculating a combined asset according to the position taken information;
and performing index landing processing on the combined assets.
Optionally, the updating module 405 is configured to update the processing data track table according to a processing result after the first processing module 403 performs data processing based on the newly added data.
The apparatus provided in the embodiment of the present invention may be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, which are not described herein again.
Fig. 6 is a schematic hardware structure diagram of a data processing apparatus according to an embodiment of the present invention. As shown in fig. 6, the data processing apparatus 60 of the present embodiment includes: a processor 601 and a memory 602; wherein
A memory 602 for storing computer-executable instructions;
the processor 601 is configured to execute the computer executable instructions stored in the memory to perform the steps of the data processing method in the foregoing embodiments. Reference may be made in particular to the description relating to the method embodiments described above.
Alternatively, the memory 602 may be separate or integrated with the processor 601.
When the memory 602 is provided separately, the data processing apparatus further includes a bus 603 for connecting the memory 602 and the processor 601.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer executing instruction is stored in the computer-readable storage medium, and when a processor executes the computer executing instruction, the method for processing data as described above is implemented.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical division, and other divisions may be realized in practice, for example, a plurality of modules may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to implement the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each module may exist alone physically, or two or more modules are integrated into one unit. The unit formed by the modules can be realized in a hardware form, and can also be realized in a form of hardware and a software functional unit.
The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute some steps of the methods according to the embodiments of the present invention.
It should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method for processing data disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of hardware and software modules within a processor.
The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile storage NVM, such as at least one disk memory, and may also be a usb disk, a removable hard disk, a read-only memory, a magnetic or optical disk, etc.
The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present invention are not limited to only one bus or one type of bus.
The storage medium may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium may reside as discrete components in an electronic device or host device.
Those of ordinary skill in the art will understand that: all or part of the steps of the method embodiments for implementing the data processing described above may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When the program is executed, the steps of the method embodiment comprising the data processing are executed; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method of data processing, comprising:
acquiring a data file of data to be processed, wherein the data file comprises a data storage date and a transaction date;
judging whether the data to be processed jumps or not according to the data storage date and the transaction date;
if the data to be processed does not jump, judging whether newly added data exists in the data to be processed according to the data storage date;
and if the data to be processed contains new data, processing the data based on the new data.
2. The method of claim 1, further comprising:
acquiring a processing data track table, wherein the processing data track table comprises storage dates of processed data;
the judging whether the data to be processed has newly added data according to the data storage date comprises the following steps:
and judging whether the data to be processed has new data or not according to the data storage date and the storage date of the processed data.
3. The method of claim 1, wherein the determining whether the data to be processed jumps according to the data storage date and the transaction date comprises:
judging whether the data storage date is consistent with the transaction date;
if the data storage date is consistent with the transaction date, judging that the data to be processed does not jump;
and if the data storage date is not consistent with the transaction date, judging that the data to be processed jumps.
4. The method according to claim 1, further comprising, after said obtaining a data file of data to be processed:
detecting whether the format of the data file of the data to be processed is correct or not according to the format of a preset data file;
if the format of the data file of the data to be processed is correct, detecting whether the data file of the data to be processed is complete according to preset keywords;
and if the data file of the data to be processed is complete, executing the step of judging whether the data to be processed jumps according to the data storage date and the transaction date.
5. The method according to claim 1, further comprising, before said obtaining a data file of data to be processed: acquiring data to be processed;
the acquiring of the data to be processed comprises:
setting an upper limit of data processing capacity;
acquiring the data to be processed from a preset database according to a data processing priority identifier of the data in the preset database and the data processing amount upper limit;
alternatively, the first and second electrodes may be,
and acquiring the data to be processed from the preset database according to the data storage date of the data in the preset database and the upper limit of the data processing amount.
6. The method of claim 1, wherein the processing data based on the new data comprises:
acquiring a newly added landing estimation table according to the newly added data;
acquiring position holding information according to the newly added landing estimation table;
calculating a combined asset according to the position taken information;
and performing index landing processing on the combined assets.
7. The method of claim 2, further comprising, after the processing data based on the new data:
and updating the processing data track table according to the processing result.
8. An apparatus for data processing, comprising:
the system comprises a first acquisition module, a second acquisition module and a processing module, wherein the first acquisition module is used for acquiring a data file of data to be processed, and the data file comprises a data storage date and a transaction date;
the judging module is used for judging whether the data to be processed jumps or not according to the data storage date and the transaction date;
the data processing device is also used for judging whether the data to be processed has newly added data or not according to the data storage date if the data to be processed does not jump;
and the first processing module is used for processing data based on the newly added data if the data to be processed contains the newly added data.
9. An apparatus for data processing, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of data processing according to any of claims 1 to 7.
10. A computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, implement a method of data processing according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911165309.5A CN111078714B (en) | 2019-11-25 | 2019-11-25 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911165309.5A CN111078714B (en) | 2019-11-25 | 2019-11-25 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111078714A true CN111078714A (en) | 2020-04-28 |
| CN111078714B CN111078714B (en) | 2023-08-15 |
Family
ID=70311505
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911165309.5A Active CN111078714B (en) | 2019-11-25 | 2019-11-25 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111078714B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8059635B1 (en) * | 2006-05-05 | 2011-11-15 | Workday, Inc. | Non-destructive data storage |
| CN105183669A (en) * | 2015-09-22 | 2015-12-23 | 珠海格力电器股份有限公司 | Data storage method and device |
| US20170323280A1 (en) * | 2016-05-05 | 2017-11-09 | Mastercard International Incorporated | Method and system for facilitating installments in an electronic transaction |
| CN107833053A (en) * | 2017-10-18 | 2018-03-23 | 中国银行股份有限公司 | The Information Authentication method and device of core banking system |
| CN108509581A (en) * | 2018-03-29 | 2018-09-07 | 四川斐讯信息技术有限公司 | A kind of method and system improving log-file information accuracy |
| CN109285080A (en) * | 2018-09-04 | 2019-01-29 | 中国平安财产保险股份有限公司 | Policy information scroll processing method, device, computer equipment and storage medium |
| CN109636553A (en) * | 2018-11-13 | 2019-04-16 | 平安科技（深圳）有限公司 | Credential management method, apparatus, computer equipment and storage medium |
| CN110457401A (en) * | 2019-07-08 | 2019-11-15 | 南京苏宁软件技术有限公司 | Date storage method, device, computer equipment and storage medium |
-
2019
- 2019-11-25 CN CN201911165309.5A patent/CN111078714B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8059635B1 (en) * | 2006-05-05 | 2011-11-15 | Workday, Inc. | Non-destructive data storage |
| CN105183669A (en) * | 2015-09-22 | 2015-12-23 | 珠海格力电器股份有限公司 | Data storage method and device |
| US20170323280A1 (en) * | 2016-05-05 | 2017-11-09 | Mastercard International Incorporated | Method and system for facilitating installments in an electronic transaction |
| CN107833053A (en) * | 2017-10-18 | 2018-03-23 | 中国银行股份有限公司 | The Information Authentication method and device of core banking system |
| CN108509581A (en) * | 2018-03-29 | 2018-09-07 | 四川斐讯信息技术有限公司 | A kind of method and system improving log-file information accuracy |
| CN109285080A (en) * | 2018-09-04 | 2019-01-29 | 中国平安财产保险股份有限公司 | Policy information scroll processing method, device, computer equipment and storage medium |
| CN109636553A (en) * | 2018-11-13 | 2019-04-16 | 平安科技（深圳）有限公司 | Credential management method, apparatus, computer equipment and storage medium |
| CN110457401A (en) * | 2019-07-08 | 2019-11-15 | 南京苏宁软件技术有限公司 | Date storage method, device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| SZWX855: "Log4j记录日志问题", pages 1 - 2 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111078714B (en) | 2023-08-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109445690B (en) | RAID card performance optimization method, device, terminal and storage medium | |
| CN110955661B (en) | Data fusion method and device, readable storage medium and electronic equipment | |
| CN102073514A (en) | Method for updating basic input/output system | |
| CN111475494A (en) | Mass data processing method, system, terminal and storage medium | |
| CN112860706A (en) | Service processing method, device, equipment and storage medium | |
| CN110675249A (en) | Matching method, device, server and storage medium for network lending | |
| CN112148337A (en) | Firmware upgrading method and device | |
| CN115129257A (en) | Data reading and writing method, electronic equipment and computer readable storage medium | |
| CN114780019A (en) | Electronic device management method and device, electronic device and storage medium | |
| CN106897224B (en) | Method and device for determining software testing range | |
| CN110188297B (en) | Resource information display method, computing device and computer storage medium | |
| CN113272785B (en) | Method for mounting file system, terminal equipment and storage medium | |
| CN111026613A (en) | Log processing method and device | |
| CN111209283A (en) | Data processing method and device | |
| CN111078714A (en) | Data processing method and device | |
| CN113064895B (en) | Incremental updating method, device and system for map | |
| CN113220573B (en) | Test method and device for micro-service architecture and electronic equipment | |
| CN111339105B (en) | Data storage method, device and server | |
| CN110703988B (en) | Storage pool creating method, system, terminal and storage medium for distributed storage | |
| CN114637672A (en) | Automatic data testing method and device, computer equipment and storage medium | |
| CN114461531A (en) | Platform adaptability test method, device, equipment and storage medium of test case | |
| CN109840213B (en) | Test data creating method, device, terminal and storage medium for GUI test | |
| CN110609988A (en) | Form verification method and equipment | |
| CN113254352A (en) | Test method, device, equipment and storage medium for test case | |
| CN113761179A (en) | Comment management and control method, server and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |